Page 1 of 2
Graphs not working after upgrade from 5.5.8 to 5.8.1
Posted: Wed Jan 27, 2021 9:45 am
by lonchara
Good morning,
After doing an in-place upgrade from RHEL 6 to 7 and Nagios XI 5.5.8 to 5.8.1, everything works except the graphing. Notifications go out as expected, but checking any of the performance graphs within the last 24 hours has "no data to display." The directory /var/nagiosramdisk/spool/perfdata/ contains nothing. The perfdata.log hasn't had an update since January 20th (where it worked briefly, then stopped again). Increased the timeout of processing from 5 to 25 and restarted npcd. Everything in the GUI is showing as green. Even tried a recompile of MRTG only to see if that was the issue, and that's when it worked briefly but failed again.
Any suggestions of what to tweak or why it might be failing?
Attached the system profile.
Re: Graphs not working after upgrade from 5.5.8 to 5.8.1
Posted: Wed Jan 27, 2021 6:30 pm
by jdunitz
Found a couple problems to start with, including the nagiosIM component that should be removed, and a Postgres setting you should check:
### NAGIOSIM detected: rm -rf /usr/local/nagiosxi/html/includes/components/nagiosim
#### Postgres detected:
https://support.nagios.com/kb/article/n ... r-754.html
Let's start there!
--Jeffrey
Re: Graphs not working after upgrade from 5.5.8 to 5.8.1
Posted: Thu Jan 28, 2021 9:05 am
by lonchara
Good morning,
Thanks for those. Removed the directory for Nagiosim as requested, and turned off standard_conforming_strings in Postgres. Waited 30 minutes, and the graphs are still blank.
One thing I noticed is that while npcd is running, it has the following line in systemctl status npcd.service:
Jan 27 09:47:52 host npcd[87532]: NPCD started.
Jan 27 09:47:52 host systemd[1]: Failed to parse PID from file /usr/local/nagiosxi/var/subsys/npcd.pid: Invalid argument
Jan 27 09:47:52 host systemd[1]: Started LSB: Nagios NPCD Initscript.
So it's starting, but it's saying invalid argument regarding the .pid file. Which does exist and has a matching value to one of the processes shown in ps faux | grep ncpd.
Re: Graphs not working after upgrade from 5.5.8 to 5.8.1
Posted: Thu Jan 28, 2021 6:28 pm
by jdunitz
Is the pid file readable by everyone?
Mine looks like this:
Code: Select all
[root@jpd-nagiosxi-one ~]# ls -l /usr/local/nagiosxi/var/subsys/npcd.pid
-rw-r--r-- 1 root root 4 Jan 19 17:50 /usr/local/nagiosxi/var/subsys/npcd.pid
[root@jpd-nagiosxi-one ~]#
Also, I noticed in your profile that your ramdisk is 100% full, which certainly will cause problems.
What all have you got in there?
Code: Select all
du /var/nagiosramdisk | sort -n | tail -22
--Jeffrey
Re: Graphs not working after upgrade from 5.5.8 to 5.8.1
Posted: Fri Jan 29, 2021 9:07 am
by lonchara
Good morning,
My /usr/local/nagiosxi/var/subsys/npcd.pid matches with yours.
-rw-r--r-- 1 root root 5 Jan 27 09:47 /usr/local/nagiosxi/var/subsys/npcd.pid
I missed the ramdisk being full, when I do the du /var/nagiosramdisk | sort -n | tail -22, I get:
0 /var/nagiosramdisk/spool
0 /var/nagiosramdisk/spool/checkresults
0 /var/nagiosramdisk/spool/perfdata
0 /var/nagiosramdisk/spool/xidpe
0 /var/nagiosramdisk/tmp
307200 /var/nagiosramdisk
ls -laSh in /var/nagiosramdisk shows:
total 301M
-rw-r--r-- 1 nagios nagios 248M Jan 29 08:07 service-perfdata
-rw-r--r-- 1 nagios nagios 45M Jan 29 08:07 host-perfdata
-rw-r--r-- 1 nagios nagios 5.4M Jan 29 08:07 status.dat
-rw-r--r-- 1 nagios nagios 3.3M Jan 25 16:30 objects.cache
drwxr-xr-x. 25 root root 4.0K Jan 13 14:56 ..
drwxrwxrwt 4 nagios nagios 160 Jan 29 08:07 .
drwxrwxr-x 5 nagios nagios 100 Jan 14 10:58 spool
drwxrwxr-x 2 nagios nagios 40 Jan 28 08:34 tmp
So it seems like something isn't processing properly from this point.
Did try a restart of npcd, and bumped the timeout from 5 to 25 seconds in /usr/local/nagios/etc/pnp/process_perfdata.cfg. I did not change it in /usr/local/nrdp/plugins/Generic/process_perfdata.pl, which is still showing 5 seconds as timeout.
Re: Graphs not working after upgrade from 5.5.8 to 5.8.1
Posted: Mon Feb 01, 2021 5:25 pm
by ssax
Edit this file:
Code: Select all
/usr/local/nagios/etc/pnp/npcd.cfg
Change this:
Code: Select all
perfdata_spool_dir = /usr/local/nagios/var/spool/perfdata/
To this:
Code: Select all
perfdata_spool_dir = /var/nagiosramdisk/spool/perfdata/
Then restart npcd:
Then follow the "
Update Performance Data Commands" section on page 9 of this doc:
https://assets.nagios.com/downloads/nag ... giosXI.pdf
Then apply configuration after, that should fix it.
If it doesn't, please send a FRESH copy of your profile.
Re: Graphs not working after upgrade from 5.5.8 to 5.8.1
Posted: Tue Feb 02, 2021 10:09 am
by lonchara
Still no go:
When I restart NPCD, I see the following:
[root~]# systemctl status npcd.service
● npcd.service - LSB: Nagios NPCD Initscript
Loaded: loaded (/etc/rc.d/init.d/npcd; bad; vendor preset: disabled)
Active: active (running) since Tue 2021-02-02 08:23:32 EST; 26s ago
Docs: man:systemd-sysv-generator(8)
Process: 127331 ExecStop=/etc/rc.d/init.d/npcd stop (code=exited, status=0/SUCCESS)
Process: 127397 ExecStart=/etc/rc.d/init.d/npcd start (code=exited, status=0/SUCCESS)
Main PID: 127400 (npcd)
CGroup: /system.slice/npcd.service
└─127400 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
Feb 02 08:23:32 systemd[1]: Starting LSB: Nagios NPCD Initscript...
Feb 02 08:23:32 npcd[127397]: NPCD started.
Feb 02 08:23:32 systemd[1]: Failed to parse PID from file /usr/local/nagiosxi/var/subsys/npcd.pid: Invalid argument
Feb 02 08:23:32 systemd[1]: Started LSB: Nagios NPCD Initscript.
I've also attached the fresh profile.
Re: Graphs not working after upgrade from 5.5.8 to 5.8.1
Posted: Tue Feb 02, 2021 6:25 pm
by ssax
Please do this (even though the top two commands look like they do the same thing, still run them as systemd sometimes has issues with init files):
Code: Select all
systemctl stop npcd
service npcd stop
systemctl start npcd
systemctl status npcd
EDIT:
You still need to do this (it's not changed in your latest profile):
Then follow the "Update Performance Data Commands" section on page 9 of this doc:
https://assets.nagios.com/downloads/nag ... giosXI.pdf
Then apply configuration after, that should fix it.
Re: Graphs not working after upgrade from 5.5.8 to 5.8.1
Posted: Thu Feb 11, 2021 9:14 am
by lonchara
Good morning,
I did make the changes before, so I've started going through the Utilizing Ram Disk directions in detail. The /usr/local/nrdp/server/config.inc.php line was cfg["check_results_dir"]="/usr/local/nagios/var/spool/checkresults";, so I've updated that to match the documentation. I've also double checked the profile commands done via the UI, and see where I messed up there (updated the second half but not the first).
Attached is my most recent profile, with the changes made. Still getting: Feb 11 08:43:07 systemd[1]: Failed to parse PID from file /usr/local/nagiosxi/var/subsys/npcd.pid: Invalid argumentFeb 11 08:43:07 systemd[1]: Started LSB: Nagios NPCD Initscript.
Also still no difference in graph behavior after the changes made.
Re: Graphs not working after upgrade from 5.5.8 to 5.8.1
Posted: Fri Feb 12, 2021 7:07 pm
by ssax
The profile looks proper in regards to the RAM Disk setup now.
The ramdisk is full again though:
Code: Select all
tmpfs 700M 700M 0 100% /var/nagiosramdisk
Which file/directory is filling up the ramdisk? Get the output of these commands as root first:
Code: Select all
ls -lh /var/nagiosramdisk
ls -lh /var/nagiosramdisk/spool
du -sh /var/nagiosramdisk/spool/*
Send the output of this command as well:
Then run these commands:
Code: Select all
systemctl stop nagios
systemctl restart ramdisk
systemctl restart npcd
systemctl start nagios
Then check the graphs in 10 minutes.
If that doesn't resolve it, please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:
https://support.nagios.com/tickets/