Graphs not working after upgrade from 5.5.8 to 5.8.1

[lonchara]

Good morning,

After doing an in-place upgrade from RHEL 6 to 7 and Nagios XI 5.5.8 to 5.8.1, everything works except the graphing. Notifications go out as expected, but checking any of the performance graphs within the last 24 hours has "no data to display." The directory /var/nagiosramdisk/spool/perfdata/ contains nothing. The perfdata.log hasn't had an update since January 20th (where it worked briefly, then stopped again). Increased the timeout of processing from 5 to 25 and restarted npcd. Everything in the GUI is showing as green. Even tried a recompile of MRTG only to see if that was the issue, and that's when it worked briefly but failed again.

Any suggestions of what to tweak or why it might be failing?

Attached the system profile.

[jdunitz]

Found a couple problems to start with, including the nagiosIM component that should be removed, and a Postgres setting you should check:

### NAGIOSIM detected: rm -rf /usr/local/nagiosxi/html/includes/components/nagiosim

#### Postgres detected: https://support.nagios.com/kb/article/n ... r-754.html

Let's start there!

--Jeffrey

[lonchara]

Good morning,

Thanks for those. Removed the directory for Nagiosim as requested, and turned off standard_conforming_strings in Postgres. Waited 30 minutes, and the graphs are still blank.

One thing I noticed is that while npcd is running, it has the following line in systemctl status npcd.service:

Jan 27 09:47:52 host npcd[87532]: NPCD started.
Jan 27 09:47:52 host systemd[1]: Failed to parse PID from file /usr/local/nagiosxi/var/subsys/npcd.pid: Invalid argument
Jan 27 09:47:52 host systemd[1]: Started LSB: Nagios NPCD Initscript.

So it's starting, but it's saying invalid argument regarding the .pid file. Which does exist and has a matching value to one of the processes shown in ps faux | grep ncpd.

[jdunitz]

Is the pid file readable by everyone?
Mine looks like this:

Code: Select all

[root@jpd-nagiosxi-one ~]#   ls -l /usr/local/nagiosxi/var/subsys/npcd.pid
-rw-r--r-- 1 root root 4 Jan 19 17:50 /usr/local/nagiosxi/var/subsys/npcd.pid
[root@jpd-nagiosxi-one ~]#

Also, I noticed in your profile that your ramdisk is 100% full, which certainly will cause problems.

What all have you got in there?

Code: Select all

du /var/nagiosramdisk | sort -n | tail -22

--Jeffrey

[lonchara]

Good morning,

My /usr/local/nagiosxi/var/subsys/npcd.pid matches with yours.
-rw-r--r-- 1 root root 5 Jan 27 09:47 /usr/local/nagiosxi/var/subsys/npcd.pid

I missed the ramdisk being full, when I do the du /var/nagiosramdisk | sort -n | tail -22, I get:

0 /var/nagiosramdisk/spool
0 /var/nagiosramdisk/spool/checkresults
0 /var/nagiosramdisk/spool/perfdata
0 /var/nagiosramdisk/spool/xidpe
0 /var/nagiosramdisk/tmp
307200 /var/nagiosramdisk

ls -laSh in /var/nagiosramdisk shows:
total 301M
-rw-r--r-- 1 nagios nagios 248M Jan 29 08:07 service-perfdata
-rw-r--r-- 1 nagios nagios 45M Jan 29 08:07 host-perfdata
-rw-r--r-- 1 nagios nagios 5.4M Jan 29 08:07 status.dat
-rw-r--r-- 1 nagios nagios 3.3M Jan 25 16:30 objects.cache
drwxr-xr-x. 25 root root 4.0K Jan 13 14:56 ..
drwxrwxrwt 4 nagios nagios 160 Jan 29 08:07 .
drwxrwxr-x 5 nagios nagios 100 Jan 14 10:58 spool
drwxrwxr-x 2 nagios nagios 40 Jan 28 08:34 tmp

So it seems like something isn't processing properly from this point.

Did try a restart of npcd, and bumped the timeout from 5 to 25 seconds in /usr/local/nagios/etc/pnp/process_perfdata.cfg. I did not change it in /usr/local/nrdp/plugins/Generic/process_perfdata.pl, which is still showing 5 seconds as timeout.

[ssax]

Edit this file:

Code: Select all

/usr/local/nagios/etc/pnp/npcd.cfg

Change this:

Code: Select all

perfdata_spool_dir = /usr/local/nagios/var/spool/perfdata/

To this:

Code: Select all

perfdata_spool_dir = /var/nagiosramdisk/spool/perfdata/

Then restart npcd:

Code: Select all

systemctl restart npcd

Then follow the "Update Performance Data Commands" section on page 9 of this doc:

https://assets.nagios.com/downloads/nag ... giosXI.pdf

Then apply configuration after, that should fix it.

If it doesn't, please send a FRESH copy of your profile.

[lonchara]

Still no go:

When I restart NPCD, I see the following:

[root~]# systemctl status npcd.service
● npcd.service - LSB: Nagios NPCD Initscript
Loaded: loaded (/etc/rc.d/init.d/npcd; bad; vendor preset: disabled)
Active: active (running) since Tue 2021-02-02 08:23:32 EST; 26s ago
Docs: man:systemd-sysv-generator(8)
Process: 127331 ExecStop=/etc/rc.d/init.d/npcd stop (code=exited, status=0/SUCCESS)
Process: 127397 ExecStart=/etc/rc.d/init.d/npcd start (code=exited, status=0/SUCCESS)
Main PID: 127400 (npcd)
CGroup: /system.slice/npcd.service
└─127400 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg

Feb 02 08:23:32 systemd[1]: Starting LSB: Nagios NPCD Initscript...
Feb 02 08:23:32 npcd[127397]: NPCD started.
Feb 02 08:23:32 systemd[1]: Failed to parse PID from file /usr/local/nagiosxi/var/subsys/npcd.pid: Invalid argument
Feb 02 08:23:32 systemd[1]: Started LSB: Nagios NPCD Initscript.

I've also attached the fresh profile.

[ssax]

Please do this (even though the top two commands look like they do the same thing, still run them as systemd sometimes has issues with init files):

Code: Select all

systemctl stop npcd
service npcd stop
systemctl start npcd
systemctl status npcd

EDIT: You still need to do this (it's not changed in your latest profile):

Then follow the "Update Performance Data Commands" section on page 9 of this doc:

https://assets.nagios.com/downloads/nag ... giosXI.pdf

Then apply configuration after, that should fix it.

[lonchara]

Good morning,

I did make the changes before, so I've started going through the Utilizing Ram Disk directions in detail. The /usr/local/nrdp/server/config.inc.php line was cfg["check_results_dir"]="/usr/local/nagios/var/spool/checkresults";, so I've updated that to match the documentation. I've also double checked the profile commands done via the UI, and see where I messed up there (updated the second half but not the first).

Attached is my most recent profile, with the changes made. Still getting: Feb 11 08:43:07 systemd[1]: Failed to parse PID from file /usr/local/nagiosxi/var/subsys/npcd.pid: Invalid argumentFeb 11 08:43:07 systemd[1]: Started LSB: Nagios NPCD Initscript.

Also still no difference in graph behavior after the changes made.

[ssax]

The profile looks proper in regards to the RAM Disk setup now.

The ramdisk is full again though:

Code: Select all

tmpfs                          700M  700M     0 100% /var/nagiosramdisk

Which file/directory is filling up the ramdisk? Get the output of these commands as root first:

Code: Select all

ls -lh /var/nagiosramdisk
ls -lh /var/nagiosramdisk/spool
du -sh /var/nagiosramdisk/spool/*

Send the output of this command as well:

Code: Select all

grep nag /etc/group

Then run these commands:

Code: Select all

systemctl stop nagios
systemctl restart ramdisk
systemctl restart npcd
systemctl start nagios

Then check the graphs in 10 minutes.

If that doesn't resolve it, please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:

https://support.nagios.com/tickets/