rrdcached no performance data

[wneville]

I'm certainly making this harder than it should be, huh? Third times the charm - the profile.zip should be in your inbox

[ssax]

Received. Thank you!

It looks like you might be missing your /usr/local/nagios/etc/pnp directory.

What is the output of this command?

Code: Select all

ls -l /usr/local/nagios/etc/

What XI version are you running? You can find it on the bottom left hand corner after logging into the web interface (it's missing from your profile).

[wneville]

I am running NagiosXI 5.8.7. The output of

Code: Select all

ls -l /usr/local/nagios/etc/

is below:

Code: Select all

total 232
drwxrwxr-x 2 apache nagios     6 Dec 14 15:40 cfgprep
-rw-rw-r-- 1 apache nagios  1813 Dec 21 07:46 cgi.cfg
-rw-rw-r-- 1 apache nagios 26959 Dec 21 12:02 commands.cfg
-rw-rw-r-- 1 apache nagios  1084 Dec 21 12:02 contactgroups.cfg
-rw-rw-r-- 1 apache nagios 11285 Dec 21 12:02 contacts.cfg
-rw-rw-r-- 1 apache nagios  1665 Dec 21 12:02 contacttemplates.cfg
-rw-rw-r-- 1 apache nagios   796 Dec 21 12:02 hostdependencies.cfg
-rw-rw-r-- 1 apache nagios   794 Dec 21 12:02 hostescalations.cfg
-rw-rw-r-- 1 apache nagios   786 Dec 21 12:02 hostextinfo.cfg
-rw-rw-r-- 1 apache nagios  2290 Dec 21 12:02 hostgroups.cfg
drwxrwxr-x 2 apache nagios 24576 Dec 21 11:39 hosts
-rw-rw-r-- 1 apache nagios 17095 Dec 21 12:02 hosttemplates.cfg
drwxrwxr-x 2 apache nagios     6 Dec 20 16:05 import
-rw-rw-r-- 1 apache nagios  5867 Dec 21 11:39 nagios.cfg
-rw-rw-r-- 1 apache nagios   684 Nov 22 13:45 ndo.cfg
-rw-rw-r-- 1 apache nagios 12838 Nov 22 13:46 nrpe.cfg
-rw-rw-r-- 1 apache nagios  5345 Nov 22 13:46 nsca.cfg
drwxrwxr-x 4 apache nagios   246 Dec 21 11:39 pnp
-rw-rw-r-- 1 apache nagios     0 Nov 22 13:52 recurringdowntime.cfg
-rw-rw-r-- 1 apache nagios   210 Nov 22 13:44 resource.cfg
-rw-rw-r-- 1 apache nagios  1627 Nov 22 13:46 send_nsca.cfg
-rw-rw-r-- 1 apache nagios   802 Dec 21 12:02 servicedependencies.cfg
-rw-rw-r-- 1 apache nagios   800 Dec 21 12:02 serviceescalations.cfg
-rw-rw-r-- 1 apache nagios   792 Dec 21 12:02 serviceextinfo.cfg
-rw-rw-r-- 1 apache nagios  3443 Dec 21 12:02 servicegroups.cfg
drwxrwxr-x 2 apache nagios  4096 Dec 21 11:39 services
-rw-rw-r-- 1 apache nagios 25803 Dec 21 12:02 servicetemplates.cfg
drwxrwxr-x 2 apache nagios    68 Dec 21 11:39 static
-rw-rw-r-- 1 apache nagios 11406 Dec 21 12:02 timeperiods.cfg

within the pnp folder is the following

Code: Select all

total 44
-rw-rw-r-- 1 apache nagios 6952 Nov 22 13:45 background.pdf
drwxrwxr-x 2 apache nagios   37 Dec 21 11:39 check_commands
-rw-rw-r-- 1 apache nagios 3370 Nov 22 13:45 config.php
-rw-rw-r-- 1 apache nagios 3090 Dec 17 12:15 npcd.cfg
-rw-rw-r-- 1 apache nagios 4127 Nov 22 13:45 npcd.cfg-sample
drwxrwxr-x 2 apache nagios   36 Dec 21 11:39 pages
-rw-rw-r-- 1 apache nagios   63 Nov 22 13:45 pnp4nagios_release
-rw-rw-r-- 1 apache nagios  855 Dec 17 12:16 process_perfdata.cfg
-rw-rw-r-- 1 apache nagios  793 Nov 22 13:45 process_perfdata.cfg-sample
-rw-rw-r-- 1 apache nagios  524 Nov 22 13:45 rra.cfg
-rw-rw-r-- 1 apache nagios  524 Nov 22 13:45 rra.cfg-sample

[ssax]

Hmm, it must not have extracted properly because I see it in the profile now.

I think this is an EL8 issue, the /etc/sysconfig/rrdcached file options aren't processed . I'm looking into this (likely needs a unit file update) and will post an update shortly.

I would also edit this file:

Code: Select all

/usr/local/nagios/etc/pnp/npcd.cfg

Change this:

Code: Select all

load_threshold = 10.0

To this:

Code: Select all

load_threshold = 60.0

Edit this file:

Code: Select all

/usr/local/nagios/etc/pnp/process_perfdata.cfg

Change this:

Code: Select all

TIMEOUT = 5

To this:

Code: Select all

TIMEOUT = 20

Then restart npcd:

Code: Select all

systemctl restart npcd

[ssax]

Edit this file:

Code: Select all

/usr/lib/systemd/system/rrdcached.service

Change the file to this:

Code: Select all

[Unit]
Description=Data caching daemon for rrdtool
Documentation=man:rrdcached(1)

[Service]
Type=forking
EnvironmentFile=/etc/sysconfig/rrdcached
ExecStart=/usr/bin/rrdcached $OPTIONS

[Install]
WantedBy=multi-user.target

Then edit this file:

Code: Select all

/etc/sysconfig/rrdcached

Change this:

Code: Select all

    OPTIONS="-s nagios -m 0660 -l unix:/var/rrdtool/rrdcached/rrdcached.sock -F -w 900 -z 90 -j /var/rrdtool/rrdcached -b /var/rrdtool/rrdcached"
    RRDC_USER=nagios

To this:

Code: Select all

    OPTIONS="-U nagios -s nagios -m 0660 -l unix:/var/rrdtool/rrdcached/rrdcached.sock -F -w 900 -z 90 -j /var/rrdtool/rrdcached -b /var/rrdtool/rrdcached"
    RRDC_USER=nagios

Then edit this file:

Code: Select all

/usr/local/nagios/etc/pnp/process_perfdata.cfg

Change this:

Code: Select all

#RRD_DAEMON_OPTS = unix:/var/rrdtool/rrdcached/rrdcached.sock

To this:

Code: Select all

RRD_DAEMON_OPTS = unix:/var/rrdtool/rrdcached/rrdcached.sock

Then run this command, wait 15 minutes, and let us know if any graphs start showing up:

Code: Select all

systemctl restart rrdcached npcd

[wneville]

Thanks so much for your thorough response. I have followed those steps and will continue to monitor the perfdata.

I tailed /usr/local/nagios/var/perfdata.log just to see if anything was happening and see this:

2021-12-22 12:54:44 [706458] [0] *** TIMEOUT: Timeout after 20 secs. ***
2021-12-22 12:54:44 [706458] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2021-12-22 12:54:44 [706458] [0] *** TIMEOUT: Please check your npcd.cfg
2021-12-22 12:54:44 [706458] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1640195644.perfdata.service-PID-706458 deleted

FYI - we are seeing high IOwait % and are evaluating our disk options to see if that will help UI speed / perfdata timeouts etc

[ssax]

I would probably increase the TIMEOUT to 40 following the steps I sent before.

If you have issues again, PM me a fresh copy of your profile.zip.

[wneville]

After following your most recent instructions and making some changes to address performance issues (opened a ticket with Nagios support) - we have not lost performance data in two days. I can't thank you enough for your help!

[ssax]

Thanks for the update, glad to help! Let us know when we're okay to lock this one up and mark it as resolved.

[wneville]

Good with me, thanks again!