Nagios XI Ramdisk filling up

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Nagios XI Ramdisk filling up

Post by JakeHatMacys »

I've had this happen on one box consistently and the other I ran the install on it cleans itself up fine. It looks like it's storing a ton of perf data in:

/var/nagiosramdisk/spool/xidpe

Code: Select all

1453428858.perfdata.service  1453730733.perfdata.host     1454032606.perfdata.service  1454334495.perfdata.service
1453428873.perfdata.host     1453730733.perfdata.service  1454032621.perfdata.host     1454334496.perfdata.host
1453428873.perfdata.service  1453730748.perfdata.host     1454032621.perfdata.service  1454334510.perfdata.service
1453428888.perfdata.host     1453730748.perfdata.service  1454032636.perfdata.host     1454334511.perfdata.host
1453428888.perfdata.service  1453730763.perfdata.host     1454032636.perfdata.service  1454334525.perfdata.service
1453428903.perfdata.host     1453730763.perfdata.service  1454032651.perfdata.host     1454334526.perfdata.host
1453428903.perfdata.service  1453730778.perfdata.host     1454032651.perfdata.service  1454334540.perfdata.service
1453428918.perfdata.host     1453730778.perfdata.service  1454032666.perfdata.host     1454334541.perfdata.host
1453428918.perfdata.service  1453730793.perfdata.host     1454032666.perfdata.service  1454334555.perfdata.service
1453428933.perfdata.host     1453730793.perfdata.service  1454032681.perfdata.service  1454334556.perfdata.host
1453428933.perfdata.service  1453730808.perfdata.host     1454032682.perfdata.host     1454334570.perfdata.service
1453428948.perfdata.host     1453730808.perfdata.service  1454032696.perfdata.host     1454334571.perfdata.host
1453428948.perfdata.service  1453730823.perfdata.host     1454032696.perfdata.service  1454334585.perfdata.service
1453428963.perfdata.host     1453730823.perfdata.service  1454032711.perfdata.host     1454334586.perfdata.host
1453428963.perfdata.service  1453730838.perfdata.host     1454032711.perfdata.service  1454334600.perfdata.service
1453428978.perfdata.host     1453730838.perfdata.service  1454032726.perfdata.host     1454334601.perfdata.host
1453428978.perfdata.service  1453730853.perfdata.host     1454032726.perfdata.service  1454334615.perfdata.host
1453428993.perfdata.host     1453730853.perfdata.service  1454032741.perfdata.host     1454334616.perfdata.service
1453428993.perfdata.service  1453730868.perfdata.host     1454032742.perfdata.service  1454334630.perfdata.host
1453429008.perfdata.host     1453730868.perfdata.service  1454032756.perfdata.host     1454334631.perfdata.service
1453429008.perfdata.service  1453730883.perfdata.service  1454032756.perfdata.service  1454334645.perfdata.host
1453429023.perfdata.host     1453730884.perfdata.host     1454032771.perfdata.host     1454334646.perfdata.service
1453429023.perfdata.service  1453730898.perfdata.host     1454032771.perfdata.service  1454334660.perfdata.host
1453429038.perfdata.host     1453730898.perfdata.service  1454032786.perfdata.host     1454334661.perfdata.service
1453429038.perfdata.service  1453730913.perfdata.host     1454032786.perfdata.service  1454334675.perfdata.service
1453429053.perfdata.host     1453730913.perfdata.service  1454032801.perfdata.host     1454334676.perfdata.host
1453429053.perfdata.service  1453730928.perfdata.host     1454032802.perfdata.service  1454334690.perfdata.service
1453429068.perfdata.host     1453730928.perfdata.service  1454032816.perfdata.host     1454334691.perfdata.host
1453429068.perfdata.service  1453730943.perfdata.host     1454032816.perfdata.service  1454334705.perfdata.service
1453429083.perfdata.host     1453730944.perfdata.service  1454032831.perfdata.host     1454334706.perfdata.host
1453429083.perfdata.service  1453730958.perfdata.host     1454032831.perfdata.service  1454334720.perfdata.service
1453429098.perfdata.host     1453730958.perfdata.service  1454032846.perfdata.host     1454334721.perfdata.host
1453429098.perfdata.service  1453730973.perfdata.host     1454032846.perfdata.service
1453429113.perfdata.host     1453730973.perfdata.service  1454032861.perfdata.host
This keeps creeping up to like 2 gigs, I thought I walked through the steps again before manually to set permissions but kind of at a loss. I can try it again, just curious if you guys have seen this before. When it was originally set up it was done with the script documented and appeared to work fine.

Ever seen this before?
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Nagios XI Ramdisk filling up

Post by tgriep »

The buildup of files in the xidpe folder is usually caused by the perfdataproc cron not running.
Can you login as root to the Xi server, run the following and post the output?

Code: Select all

service crond status
ps -aef | grep perfdataproc
Could you post this file so we can review it?

Code: Select all

/usr/local/nagiosxi/html/config.inc.php
Be sure to check out our Knowledgebase for helpful articles and solutions!
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI Ramdisk filling up

Post by JakeHatMacys »

Code: Select all

[root@****** libexec]# service crond status
crond (pid  2506) is running...
You have new mail in /var/spool/mail/root
[root@******* libexec]# ps -aef | grep perfdataproc
nagios   20796 20781  0 12:56 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios   20802 20796  0 12:56 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
root     31256 29899  0 12:56 pts/97   00:00:00 grep perfdataproc
[root@******** libexec]#
You do not have the required permissions to view the files attached to this post.
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Nagios XI Ramdisk filling up

Post by lmiltchev »

Can you run the following commands and show the output in code wraps?

Code: Select all

grep ramdisk /usr/local/nagios/etc/nagios.cfg
ll -d /var/nagiosramdisk
ll /var/nagiosramdisk
ll /var/nagiosramdisk/spool
chage -l nagios
tail -50 /var/log/cron
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Nagios XI Ramdisk filling up

Post by lmiltchev »

Change this line in the confing.inc.php:

Code: Select all

$cfg['perfdata_spool'] = 'var/nagiosramdisk/spool/perfdata/';
to:

Code: Select all

$cfg['perfdata_spool'] = '/var/nagiosramdisk/spool/perfdata/';
and restart httpd.

Code: Select all

service httpd restart
Be sure to check out our Knowledgebase for helpful articles and solutions!
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI Ramdisk filling up

Post by JakeHatMacys »

lmiltchev wrote:Change this line in the confing.inc.php:

Code: Select all

$cfg['perfdata_spool'] = 'var/nagiosramdisk/spool/perfdata/';
to:

Code: Select all

$cfg['perfdata_spool'] = '/var/nagiosramdisk/spool/perfdata/';
and restart httpd.

Code: Select all

service httpd restart

Done.
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI Ramdisk filling up

Post by JakeHatMacys »

lmiltchev wrote:Can you run the following commands and show the output in code wraps?

Code: Select all

grep ramdisk /usr/local/nagios/etc/nagios.cfg
ll -d /var/nagiosramdisk
ll /var/nagiosramdisk
ll /var/nagiosramdisk/spool
chage -l nagios
tail -50 /var/log/cron

Code: Select all

grep ramdisk /usr/local/nagios/etc/nagios.cfg
service_perfdata_file=/var/nagiosramdisk/service-perfdata
host_perfdata_file=/var/nagiosramdisk/host-perfdata
check_result_path=/var/nagiosramdisk/spool/checkresults
object_cache_file=/var/nagiosramdisk/objects.cache
status_file=/var/nagiosramdisk/status.dat
temp_path=/var/nagiosramdisk/tmp
ll -d /var/nagiosramdisk
drwxrwxrwt 5 nagios nagios 180 Feb 2 09:56 /var/nagiosramdisk

Code: Select all

ll /var/nagiosramdisk
total 5200
drwxr-xr-x 2 nagios nagios      40 Dec 16 08:48 checkresults
-rw-r--r-- 1 nagios nagios    5387 Feb  2 09:56 host-perfdata
-rw-r--r-- 1 nagios nagios 2001577 Jan 29 11:21 objects.cache
-rw-r--r-- 1 nagios nagios   14519 Feb  2 09:56 service-perfdata
drwxrwxr-x 5 nagios nagios     100 Dec 16 08:49 spool
-rw-r--r-- 1 nagios nagios 3296116 Feb  2 09:56 status.dat
drwxrwxr-x 2 nagios nagios      40 Feb  1 12:19 tmp

Code: Select all

ll /var/nagiosramdisk/spool
total 0
drwxrwxr-x 2 nagios nagios     40 Feb  1 12:19 checkresults
drwxrwxr-x 2 nagios nagios 164960 Feb  2 09:56 perfdata
drwxrwxr-x 2 nagios nagios     40 Feb  2 09:56 xidpe

Code: Select all

chage -l nagios
Last password change                                    : Sep 23, 2015
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7

Code: Select all

tail -50 /var/log/cron
Feb  2 09:50:01 esu2v236 CROND[28782]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1)
Feb  2 09:50:01 esu2v236 CROND[28764]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1)
Feb  2 09:50:01 esu2v236 CROND[28775]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Feb  2 09:50:01 esu2v236 CROND[28784]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Feb  2 09:50:01 esu2v236 CROND[28772]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Feb  2 09:50:01 esu2v236 CROND[28794]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Feb  2 09:50:01 esu2v236 CROND[28795]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Feb  2 09:51:01 esu2v236 CROND[7782]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Feb  2 09:51:01 esu2v236 CROND[7783]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Feb  2 09:51:01 esu2v236 CROND[7785]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Feb  2 09:51:01 esu2v236 CROND[7788]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Feb  2 09:51:01 esu2v236 CROND[7790]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Feb  2 09:51:01 esu2v236 CROND[7793]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Feb  2 09:51:01 esu2v236 CROND[7786]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Feb  2 09:51:01 esu2v236 CROND[7795]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Feb  2 09:52:01 esu2v236 CROND[30423]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Feb  2 09:52:01 esu2v236 CROND[30429]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Feb  2 09:52:01 esu2v236 CROND[30428]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Feb  2 09:52:01 esu2v236 CROND[30430]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Feb  2 09:52:01 esu2v236 CROND[30427]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Feb  2 09:52:01 esu2v236 CROND[30431]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Feb  2 09:52:01 esu2v236 CROND[30426]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Feb  2 09:52:01 esu2v236 CROND[30425]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Feb  2 09:53:01 esu2v236 CROND[3914]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Feb  2 09:53:01 esu2v236 CROND[3917]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Feb  2 09:53:01 esu2v236 CROND[3915]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Feb  2 09:53:01 esu2v236 CROND[3921]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Feb  2 09:53:01 esu2v236 CROND[3922]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Feb  2 09:53:01 esu2v236 CROND[3919]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Feb  2 09:53:01 esu2v236 CROND[3930]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Feb  2 09:53:01 esu2v236 CROND[3927]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Feb  2 09:54:01 esu2v236 CROND[9014]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Feb  2 09:54:01 esu2v236 CROND[9015]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Feb  2 09:54:01 esu2v236 CROND[9017]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Feb  2 09:54:01 esu2v236 CROND[9022]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Feb  2 09:54:01 esu2v236 CROND[9024]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Feb  2 09:54:01 esu2v236 CROND[9018]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Feb  2 09:54:01 esu2v236 CROND[9023]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Feb  2 09:54:01 esu2v236 CROND[9020]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Feb  2 09:55:01 esu2v236 CROND[15814]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Feb  2 09:55:01 esu2v236 CROND[15817]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Feb  2 09:55:01 esu2v236 CROND[15816]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Feb  2 09:55:01 esu2v236 CROND[15818]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Feb  2 09:55:01 esu2v236 CROND[15819]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1)
Feb  2 09:55:01 esu2v236 CROND[15823]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Feb  2 09:55:01 esu2v236 CROND[15826]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Feb  2 09:55:01 esu2v236 CROND[15829]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1)
Feb  2 09:55:01 esu2v236 CROND[15825]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Feb  2 09:55:01 esu2v236 CROND[15828]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Feb  2 09:55:01 esu2v236 CROND[15827]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Nagios XI Ramdisk filling up

Post by tgriep »

Can you run the following and post the output?

Code: Select all

grep perfdata_spool_dir /usr/local/nagios/etc/pnp/npcd.cfg
ps -ef |grep npcd
ls /var/nagiosramdisk/spool/xidpe | wc -l
ls /var/nagiosramdisk/spool/perfdata/ | wc -l
ls /var/nagiosramdisk/spool/checkresults/ | wc -l
I just want to check a few more settings, etc...
Thanks
Be sure to check out our Knowledgebase for helpful articles and solutions!
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI Ramdisk filling up

Post by JakeHatMacys »

tgriep wrote:Can you run the following and post the output?

Code: Select all

grep perfdata_spool_dir /usr/local/nagios/etc/pnp/npcd.cfg
ps -ef |grep npcd
ls /var/nagiosramdisk/spool/xidpe | wc -l
ls /var/nagiosramdisk/spool/perfdata/ | wc -l
ls /var/nagiosramdisk/spool/checkresults/ | wc -l
I just want to check a few more settings, etc...
Thanks
Sure:

Code: Select all

grep perfdata_spool_dir /usr/local/nagios/etc/pnp/npcd.cfg
# perfdata_spool_dir - where we can find the
# perfdata_spool_dir = </path/to/directory/>
perfdata_spool_dir = /var/nagiosramdisk/spool/perfdata/
# in 'perfdata_spool_dir'

Code: Select all

ps -ef |grep npcd
nagios   16440     1  0 Jan11 ?        00:00:27 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
root     26610 25535  0 13:28 pts/94   00:00:00 grep npcd
ls /var/nagiosramdisk/spool/xidpe | wc -l
0

ls /var/nagiosramdisk/spool/perfdata/ | wc -l
0

ls /var/nagiosramdisk/spool/checkresults/ | wc -l
0
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Nagios XI Ramdisk filling up

Post by tgriep »

Thanks, It looks like the files are being processed now.
If everything is working for you, shall we close this post?
Be sure to check out our Knowledgebase for helpful articles and solutions!
Locked