Page 1 of 1

Graphs not working after increasing RAM

Posted: Tue Apr 15, 2014 6:25 pm
by stebbo
Hi All,

I recently upgraded to Nagios 2012 R2.9. I am running Centos 6.5 on a Hyper-V virtual machine. The upgrade was running well until I decided to increase the amount of RAM available from 2048MB to 4096MB and also the number of processors from 2 to 4.

After rebooting with the extra memory, the graphing components stopped working. None of the performance graphs would work. The last 4 hours were completely blank, and if I expanded to more data, the data would show, but would abruptly cease to show any data at 6pm last nite (the time when I increased the ram).

Not being sure whether it was the extra processors or RAM that caused the problem, I put the RAM back to 2048MB and rebooted. The graphs re-appeared, including all the data that had been collected overnight. So it seems to be a graphing presentation problem rather than a data collection/storage problem.

Any ideas?

Thanks,
Chris.

Nagios XI Installation Profile

Download Profile
System:

Nagios XI Version : 2012R2.9
nagios.xxx 2.6.32-431.5.1.el6.x86_64 x86_64
CentOS release 6.5 (Final)
Gnome is not installed
Apache Information

PHP Version: 5.3.3
Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/6.1.3 Safari/537.75.14
Server Name: nagios.xxx
Server Address: 10.174.64.188
Server Port: 443
Date/Time

PHP Timezone: Australia/Melbourne
PHP Time: Wed, 16 Apr 2014 09:25:53 +1000
System Time: Wed, 16 Apr 2014 09:25:53 +1000
Nagios XI Data

License ends in: TNPTSN

nagios (pid 1770) is running...
NPCD running (pid 1721).
ndo2db (pid 1789) is running...
CPU Load 15: 0.63
Total Hosts: 29
Total Services: 219
Function 'get_base_uri' returns: https://nagios.xxx/nagiosxi/
Function 'get_base_url' returns: https://nagios.xxx/nagiosxi/
Function 'get_backend_url(internal_call=false)' returns: https://nagios.xxx/nagiosxi/includes/co ... rofile.php
Function 'get_backend_url(internal_call=true)' returns: https://localhost/nagiosxi/backend/
Ping Test localhost

Running:
/bin/ping -c 3 localhost 2>&1
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.042 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.036 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.047 ms

--- localhost ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2012ms
rtt min/avg/max/mdev = 0.036/0.041/0.047/0.008 ms
Test wget To locahost

WGET From URL: http://localhost/nagiosql/index.php
Running:
/usr/bin/wget http://localhost/nagiosql/index.php
--2014-04-16 09:25:55-- http://localhost/nagiosql/index.php
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5259 (5.1K) [text/html]
Saving to: `/usr/local/nagiosxi/tmp/nagiosql_index.tmp'

0K ..... 100% 283M=0s

2014-04-16 09:25:55 (283 MB/s) - `/usr/local/nagiosxi/tmp/nagiosql_index.tmp' saved [5259/5259]

Re: Graphs not working after increasing RAM

Posted: Wed Apr 16, 2014 9:41 am
by slansing
What is the output of:

Code: Select all

service npcd status

Code: Select all

tail -25 /usr/local/nagios/var/npcd.log
Is performance graphing turned on in the web interface?

Re: Graphs not working after increasing RAM

Posted: Wed Apr 16, 2014 8:25 pm
by stebbo
Hi Slansing,

thanks for your response. The npcd service is running. The tail of the log is pasted below.

The performance grapher is working. This installation has been going for almost 2 years with very few changes apart from adding and removing hosts / services. The system was graphing before and after I had increased the RAM, and the data that was collected during the time that I had increased the RAM was still there - just the graphs weren't displaying.

I'm wondering if it was just co-incidence? Should I try increasing the RAM again and see what happens?

The RAM was increased on 04-15-2014 at around 18:00. I notice from the log that there was an exit at 19:18 that is probably the culprit but there is no indication of why the exit. I was wondering whether the 4GB RAM was a problem or not?

Cheers,
Chris.

[04-11-2014 12:02:24] NPCD: Please have a look at 'npcd -V' to get license information
[04-11-2014 12:02:24] NPCD: HINT: load_threshold is enabled - ('10.000000')
[04-11-2014 12:09:55] NPCD: Caught Termination Signal - Hasta la vista... baby
[04-11-2014 12:40:36] NPCD: npcd Daemon (0.4.14) started with PID=1663
[04-11-2014 12:40:36] NPCD: Please have a look at 'npcd -V' to get license information
[04-11-2014 12:40:36] NPCD: HINT: load_threshold is enabled - ('10.000000')
[04-12-2014 19:33:59] NPCD: ERROR: Executed command exits with return code '7'
[04-12-2014 19:33:59] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1397295231.perfdata.service'
[04-12-2014 19:33:59] NPCD: ERROR: Executed command exits with return code '7'
[04-12-2014 19:33:59] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1397295216.perfdata.service'
[04-14-2014 19:32:44] NPCD: ERROR: Executed command exits with return code '7'
[04-14-2014 19:32:44] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1397467941.perfdata.service'
[04-14-2014 19:32:44] NPCD: ERROR: Executed command exits with return code '7'
[04-14-2014 19:32:44] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1397467941.perfdata.host'

[04-15-2014 19:18:21] NPCD: Caught Termination Signal - Hasta la vista… baby

[04-16-2014 09:08:55] NPCD: npcd Daemon (0.4.14) started with PID=1700
[04-16-2014 09:08:55] NPCD: Please have a look at 'npcd -V' to get license information
[04-16-2014 09:08:55] NPCD: HINT: load_threshold is enabled - ('10.000000')
[04-16-2014 09:19:29] NPCD: Caught Termination Signal - Hasta la vista... baby
[04-16-2014 09:20:13] NPCD: npcd Daemon (0.4.14) started with PID=1721
[04-16-2014 09:20:13] NPCD: Please have a look at 'npcd -V' to get license information
[04-16-2014 09:20:13] NPCD: HINT: load_threshold is enabled - ('10.000000')
[04-16-2014 10:19:53] NPCD: npcd Daemon (0.4.14) started with PID=1729
[04-16-2014 10:19:53] NPCD: Please have a look at 'npcd -V' to get license information
[04-16-2014 10:19:53] NPCD: HINT: load_threshold is enabled - ('10.000000')

Re: Graphs not working after increasing RAM

Posted: Thu Apr 17, 2014 10:01 am
by lmiltchev
I'm wondering if it was just co-incidence? Should I try increased RAM again and see what happens?
It is very possible this was just a co-incidence. Graphs not working because of increased amount of RAM doesn't make much sense. Try increasing the RAM again and see what happens. If the issue reappears, post the tail of both, the ncpd.log and perfdata.log:

Code: Select all

tail /usr/local/nagios/var/npcd.log
tail /usr/local/nagios/var/perfdata.log
Also, show the permissions of the items in the "perfdata" directory:

Code: Select all

ll /usr/local/nagios/share/perfdata

Re: Graphs not working after increasing RAM

Posted: Thu Apr 17, 2014 7:50 pm
by Box293
I've noticed a similar problem occur in the past howerver I could never pinpoint what was causing the issue but I think it's CPU related.

What I experienced was:
  • Shutdown XI Host
    Added extra CPU cores
    Powered XI Host back on
    Noticed that performance graphs were not updating
    Rebooted XI Host and everything started working OK again
I vagely remember that when it occurred last time, when the performance graphs were not being updated, one of the spool folders was filling up and not being processed. After the reboot it took a while for all the data in the spool folder to be processed but eventually all the graphs were updated.

So it would appear that we have both experienced the same problem.

FYI This was on a VM running on ESXi so it probably isn't related to Hyper V.

Re: Graphs not working after increasing RAM

Posted: Wed Apr 23, 2014 4:47 am
by stebbo
Thanks for the responses folks. I shall try increasing the RAM again and see what happens.

Re: Graphs not working after increasing RAM

Posted: Wed Apr 23, 2014 9:53 am
by sreinhardt
Great, let us know how it works for you!