Graphs not working after increasing RAM

[stebbo]

Hi All,

I recently upgraded to Nagios 2012 R2.9. I am running Centos 6.5 on a Hyper-V virtual machine. The upgrade was running well until I decided to increase the amount of RAM available from 2048MB to 4096MB and also the number of processors from 2 to 4.

After rebooting with the extra memory, the graphing components stopped working. None of the performance graphs would work. The last 4 hours were completely blank, and if I expanded to more data, the data would show, but would abruptly cease to show any data at 6pm last nite (the time when I increased the ram).

Not being sure whether it was the extra processors or RAM that caused the problem, I put the RAM back to 2048MB and rebooted. The graphs re-appeared, including all the data that had been collected overnight. So it seems to be a graphing presentation problem rather than a data collection/storage problem.

Any ideas?

Thanks,
Chris.

Nagios XI Installation Profile

Download Profile
System:

Nagios XI Version : 2012R2.9
nagios.xxx 2.6.32-431.5.1.el6.x86_64 x86_64
CentOS release 6.5 (Final)
Gnome is not installed
Apache Information

PHP Version: 5.3.3
Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/6.1.3 Safari/537.75.14
Server Name: nagios.xxx
Server Address: 10.174.64.188
Server Port: 443
Date/Time

PHP Timezone: Australia/Melbourne
PHP Time: Wed, 16 Apr 2014 09:25:53 +1000
System Time: Wed, 16 Apr 2014 09:25:53 +1000
Nagios XI Data

License ends in: TNPTSN

nagios (pid 1770) is running...
NPCD running (pid 1721).
ndo2db (pid 1789) is running...
CPU Load 15: 0.63
Total Hosts: 29
Total Services: 219
Function 'get_base_uri' returns: https://nagios.xxx/nagiosxi/
Function 'get_base_url' returns: https://nagios.xxx/nagiosxi/
Function 'get_backend_url(internal_call=false)' returns: https://nagios.xxx/nagiosxi/includes/co ... rofile.php
Function 'get_backend_url(internal_call=true)' returns: https://localhost/nagiosxi/backend/
Ping Test localhost

Running:
/bin/ping -c 3 localhost 2>&1
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.042 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.036 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.047 ms

--- localhost ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2012ms
rtt min/avg/max/mdev = 0.036/0.041/0.047/0.008 ms
Test wget To locahost

WGET From URL: http://localhost/nagiosql/index.php
Running:
/usr/bin/wget http://localhost/nagiosql/index.php
--2014-04-16 09:25:55-- http://localhost/nagiosql/index.php
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5259 (5.1K) [text/html]
Saving to: `/usr/local/nagiosxi/tmp/nagiosql_index.tmp'

0K ..... 100% 283M=0s

2014-04-16 09:25:55 (283 MB/s) - `/usr/local/nagiosxi/tmp/nagiosql_index.tmp' saved [5259/5259]

[slansing]

What is the output of:

Code: Select all

service npcd status

Code: Select all

tail -25 /usr/local/nagios/var/npcd.log

Is performance graphing turned on in the web interface?

[stebbo]

Hi Slansing,

thanks for your response. The npcd service is running. The tail of the log is pasted below.

The performance grapher is working. This installation has been going for almost 2 years with very few changes apart from adding and removing hosts / services. The system was graphing before and after I had increased the RAM, and the data that was collected during the time that I had increased the RAM was still there - just the graphs weren't displaying.

I'm wondering if it was just co-incidence? Should I try increasing the RAM again and see what happens?

The RAM was increased on 04-15-2014 at around 18:00. I notice from the log that there was an exit at 19:18 that is probably the culprit but there is no indication of why the exit. I was wondering whether the 4GB RAM was a problem or not?

Cheers,
Chris.

[04-11-2014 12:02:24] NPCD: Please have a look at 'npcd -V' to get license information
[04-11-2014 12:02:24] NPCD: HINT: load_threshold is enabled - ('10.000000')
[04-11-2014 12:09:55] NPCD: Caught Termination Signal - Hasta la vista... baby
[04-11-2014 12:40:36] NPCD: npcd Daemon (0.4.14) started with PID=1663
[04-11-2014 12:40:36] NPCD: Please have a look at 'npcd -V' to get license information
[04-11-2014 12:40:36] NPCD: HINT: load_threshold is enabled - ('10.000000')
[04-12-2014 19:33:59] NPCD: ERROR: Executed command exits with return code '7'
[04-12-2014 19:33:59] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1397295231.perfdata.service'
[04-12-2014 19:33:59] NPCD: ERROR: Executed command exits with return code '7'
[04-12-2014 19:33:59] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1397295216.perfdata.service'
[04-14-2014 19:32:44] NPCD: ERROR: Executed command exits with return code '7'
[04-14-2014 19:32:44] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1397467941.perfdata.service'
[04-14-2014 19:32:44] NPCD: ERROR: Executed command exits with return code '7'
[04-14-2014 19:32:44] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1397467941.perfdata.host'

[04-15-2014 19:18:21] NPCD: Caught Termination Signal - Hasta la vista… baby

[04-16-2014 09:08:55] NPCD: npcd Daemon (0.4.14) started with PID=1700
[04-16-2014 09:08:55] NPCD: Please have a look at 'npcd -V' to get license information
[04-16-2014 09:08:55] NPCD: HINT: load_threshold is enabled - ('10.000000')
[04-16-2014 09:19:29] NPCD: Caught Termination Signal - Hasta la vista... baby
[04-16-2014 09:20:13] NPCD: npcd Daemon (0.4.14) started with PID=1721
[04-16-2014 09:20:13] NPCD: Please have a look at 'npcd -V' to get license information
[04-16-2014 09:20:13] NPCD: HINT: load_threshold is enabled - ('10.000000')
[04-16-2014 10:19:53] NPCD: npcd Daemon (0.4.14) started with PID=1729
[04-16-2014 10:19:53] NPCD: Please have a look at 'npcd -V' to get license information
[04-16-2014 10:19:53] NPCD: HINT: load_threshold is enabled - ('10.000000')

[lmiltchev]

I'm wondering if it was just co-incidence? Should I try increased RAM again and see what happens?

It is very possible this was just a co-incidence. Graphs not working because of increased amount of RAM doesn't make much sense. Try increasing the RAM again and see what happens. If the issue reappears, post the tail of both, the ncpd.log and perfdata.log:

Code: Select all

tail /usr/local/nagios/var/npcd.log
tail /usr/local/nagios/var/perfdata.log

Also, show the permissions of the items in the "perfdata" directory:

Code: Select all

ll /usr/local/nagios/share/perfdata

[Box293]

I've noticed a similar problem occur in the past howerver I could never pinpoint what was causing the issue but I think it's CPU related.

What I experienced was:

• Shutdown XI Host
  Added extra CPU cores
  Powered XI Host back on
  Noticed that performance graphs were not updating
  Rebooted XI Host and everything started working OK again

I vagely remember that when it occurred last time, when the performance graphs were not being updated, one of the spool folders was filling up and not being processed. After the reboot it took a while for all the data in the spool folder to be processed but eventually all the graphs were updated.

So it would appear that we have both experienced the same problem.

FYI This was on a VM running on ESXi so it probably isn't related to Hyper V.

[stebbo]

Thanks for the responses folks. I shall try increasing the RAM again and see what happens.

[sreinhardt]

Great, let us know how it works for you!