Page 1 of 2
Blank notifications page, npcd running wild, load average hi
Posted: Wed Nov 02, 2011 10:45 am
by jon.bilderback
Under the home tab -> incident management -> Notifications, the page does not come up, just stays blank.
In the /var/log/httpd/error_log file, i get the following:
PHP Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 19887545 bytes) in /usr/local/nagiosxi/html/includes/components/nagioscore/coreuiproxy.inc.php on line 154, referer:
http://nagios.lamar.edu/nagiosxi/
I have edited the /etc/php.ini file to change memory_limit from 16M to 128M, but still have the same issue.
The npcd service is using somewhere in the neighborhood of 80 %CPU when I look at it in "top", but I do not know if that is related.
My load average is 4.52, 4.11, 4.19.
VMWare install of server, updated to 2011r1.8
Re: Blank notifications page, npcd running wild, load averag
Posted: Wed Nov 02, 2011 11:38 am
by agriffin
There are a few things about this that seem strange to me. For example, npcd normally turns itself off when the load is too high. But I think it will be easier to take things one at a time. Try disabling npcd:
Afterwards, let me know if that fixes anything or if there are still problems. Note that performance graphs will stop updating; this is to be expected when npcd is not running. If everything else works we'll move on to re-enabling npcd and trying to reduce the load it's using.
Re: Blank notifications page, npcd running wild, load averag
Posted: Wed Nov 02, 2011 11:44 am
by jon.bilderback
I stopped the npcd service, and within a few minutes : load average: 1.32, 2.54, 3.50
Re: Blank notifications page, npcd running wild, load averag
Posted: Wed Nov 02, 2011 12:05 pm
by mguthrie
If your environment is large I would recommend increasing your php memory limit as high as 512.
npcd can eat up a fair amount of CPU with a high check volume, or if the process has been stopped for a while. I would check the /usr/local/nagios/var/npcd.log and make sure it's processing files ok and that there aren't permissions problems. If there are permissions issues with a large number of hosts and services it's possible that npcd would eat up a fair amount of CPU.
http://assets.nagios.com/downloads/nagi ... p#fixperms
How large is your environment (how many hosts + services?)
Re: Blank notifications page, npcd running wild, load averag
Posted: Thu Nov 03, 2011 10:04 am
by jon.bilderback
I changed the memory_limit in /etc/php.ini to 512M, but I'm still getting the same results:
PHP Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 20111998 bytes) in /usr/local/nagiosxi/html/includes/components/nagioscore/coreuiproxy.inc.php on line 154, referer:
http://nagios.lamar.edu/nagiosxi/index.php?
316 Hosts, 1853 services.
I ran the permission script, and restarted npcd, but still see the immediate spike in cpu load.
Re: Blank notifications page, npcd running wild, load averag
Posted: Thu Nov 03, 2011 10:39 am
by mguthrie
Ok, I think I have an idea as to what's going on for the memory error. Can you access your nagios service with the http://<youraddress>/nagios and pull up the notifications page. About how many notifications would you say you have on that page? (rough ballpark)
For PNP, check your logging settings in the following files.
/usr/local/nagios/etc/pnp/npcd.cfg
/usr/local/nagios/etc/pnp/process_perfdata.cfg
and try setting the logging to 0 if it's not already. Increased logging can increase CPU usage.
If that doesn't resolve the issue, you can turn logging back on and then watch the following log files for any clues:
/usr/local/nagios/var/perfdata.log
/usr/local/nagios/var/npcd.log
Re: Blank notifications page, npcd running wild, load averag
Posted: Thu Nov 03, 2011 12:52 pm
by jon.bilderback
from npcd.log:
[11-03-2011 12:37:20] NPCD: ERROR: Executed command exits with return code '7'
[11-03-2011 12:37:20] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//service-perfdata.1320341816'
I have set the logging to "0", and it looks like the cpu load is staying pretty low, and the performance data is showing up again in graphs.
For some reason, my password is not working on the http://<youraddress>/nagios page. Is this the same as the core config nagiosadmin password? If so, it isn't working. If not, how do I change that password?
Thanks
Re: Blank notifications page, npcd running wild, load averag
Posted: Thu Nov 03, 2011 2:00 pm
by lmiltchev
Your login pasword (Core & XI) and Nagios Core Config Manager password don't have to be the same.
If you are having login issues, you may try resetting your password by running in terminal:
Code: Select all
# cd /usr/local/nagiosxi/scripts
# ./reset_nagiosadmin_password.php --password=<newpassword>
where <newpassword> is your new password.
Then you can try to log in again:
In XI: http://<your_IP_address>/nagiosxi
In Core: http://<your_IP_address>/nagios
Hope this helps.
Re: Blank notifications page, npcd running wild, load averag
Posted: Thu Nov 10, 2011 11:19 am
by jon.bilderback
For some reason, even after resetting the nagiosadmin password in the terminal, I am still not able to get to that url: http://<youraddress>/nagios
Is there another place or way to change that password?
Re: Blank notifications page, npcd running wild, load averag
Posted: Thu Nov 10, 2011 12:22 pm
by mguthrie
Yeah, the /nagios URL requires an apache authentication. You can set this through the Admin->Reset Security Credentials and manually set if for the nagiosadmin user. If you want to add additional users you can do so with the following command:
Code: Select all
cd /usr/local/nagiosxi/etc
htpasswd htpasswd.users <username>
You'll be prompted to update the password.