Page 1 of 2
Nagios XI Ramdisk filling up
Posted: Mon Feb 01, 2016 8:59 am
by JakeHatMacys
I've had this happen on one box consistently and the other I ran the install on it cleans itself up fine. It looks like it's storing a ton of perf data in:
/var/nagiosramdisk/spool/xidpe
Code: Select all
1453428858.perfdata.service 1453730733.perfdata.host 1454032606.perfdata.service 1454334495.perfdata.service
1453428873.perfdata.host 1453730733.perfdata.service 1454032621.perfdata.host 1454334496.perfdata.host
1453428873.perfdata.service 1453730748.perfdata.host 1454032621.perfdata.service 1454334510.perfdata.service
1453428888.perfdata.host 1453730748.perfdata.service 1454032636.perfdata.host 1454334511.perfdata.host
1453428888.perfdata.service 1453730763.perfdata.host 1454032636.perfdata.service 1454334525.perfdata.service
1453428903.perfdata.host 1453730763.perfdata.service 1454032651.perfdata.host 1454334526.perfdata.host
1453428903.perfdata.service 1453730778.perfdata.host 1454032651.perfdata.service 1454334540.perfdata.service
1453428918.perfdata.host 1453730778.perfdata.service 1454032666.perfdata.host 1454334541.perfdata.host
1453428918.perfdata.service 1453730793.perfdata.host 1454032666.perfdata.service 1454334555.perfdata.service
1453428933.perfdata.host 1453730793.perfdata.service 1454032681.perfdata.service 1454334556.perfdata.host
1453428933.perfdata.service 1453730808.perfdata.host 1454032682.perfdata.host 1454334570.perfdata.service
1453428948.perfdata.host 1453730808.perfdata.service 1454032696.perfdata.host 1454334571.perfdata.host
1453428948.perfdata.service 1453730823.perfdata.host 1454032696.perfdata.service 1454334585.perfdata.service
1453428963.perfdata.host 1453730823.perfdata.service 1454032711.perfdata.host 1454334586.perfdata.host
1453428963.perfdata.service 1453730838.perfdata.host 1454032711.perfdata.service 1454334600.perfdata.service
1453428978.perfdata.host 1453730838.perfdata.service 1454032726.perfdata.host 1454334601.perfdata.host
1453428978.perfdata.service 1453730853.perfdata.host 1454032726.perfdata.service 1454334615.perfdata.host
1453428993.perfdata.host 1453730853.perfdata.service 1454032741.perfdata.host 1454334616.perfdata.service
1453428993.perfdata.service 1453730868.perfdata.host 1454032742.perfdata.service 1454334630.perfdata.host
1453429008.perfdata.host 1453730868.perfdata.service 1454032756.perfdata.host 1454334631.perfdata.service
1453429008.perfdata.service 1453730883.perfdata.service 1454032756.perfdata.service 1454334645.perfdata.host
1453429023.perfdata.host 1453730884.perfdata.host 1454032771.perfdata.host 1454334646.perfdata.service
1453429023.perfdata.service 1453730898.perfdata.host 1454032771.perfdata.service 1454334660.perfdata.host
1453429038.perfdata.host 1453730898.perfdata.service 1454032786.perfdata.host 1454334661.perfdata.service
1453429038.perfdata.service 1453730913.perfdata.host 1454032786.perfdata.service 1454334675.perfdata.service
1453429053.perfdata.host 1453730913.perfdata.service 1454032801.perfdata.host 1454334676.perfdata.host
1453429053.perfdata.service 1453730928.perfdata.host 1454032802.perfdata.service 1454334690.perfdata.service
1453429068.perfdata.host 1453730928.perfdata.service 1454032816.perfdata.host 1454334691.perfdata.host
1453429068.perfdata.service 1453730943.perfdata.host 1454032816.perfdata.service 1454334705.perfdata.service
1453429083.perfdata.host 1453730944.perfdata.service 1454032831.perfdata.host 1454334706.perfdata.host
1453429083.perfdata.service 1453730958.perfdata.host 1454032831.perfdata.service 1454334720.perfdata.service
1453429098.perfdata.host 1453730958.perfdata.service 1454032846.perfdata.host 1454334721.perfdata.host
1453429098.perfdata.service 1453730973.perfdata.host 1454032846.perfdata.service
1453429113.perfdata.host 1453730973.perfdata.service 1454032861.perfdata.host
This keeps creeping up to like 2 gigs, I thought I walked through the steps again before manually to set permissions but kind of at a loss. I can try it again, just curious if you guys have seen this before. When it was originally set up it was done with the script documented and appeared to work fine.
Ever seen this before?
Re: Nagios XI Ramdisk filling up
Posted: Mon Feb 01, 2016 11:50 am
by tgriep
The buildup of files in the xidpe folder is usually caused by the perfdataproc cron not running.
Can you login as root to the Xi server, run the following and post the output?
Code: Select all
service crond status
ps -aef | grep perfdataproc
Could you post this file so we can review it?
Code: Select all
/usr/local/nagiosxi/html/config.inc.php
Re: Nagios XI Ramdisk filling up
Posted: Mon Feb 01, 2016 12:57 pm
by JakeHatMacys
Code: Select all
[root@****** libexec]# service crond status
crond (pid 2506) is running...
You have new mail in /var/spool/mail/root
[root@******* libexec]# ps -aef | grep perfdataproc
nagios 20796 20781 0 12:56 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios 20802 20796 0 12:56 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
root 31256 29899 0 12:56 pts/97 00:00:00 grep perfdataproc
[root@******** libexec]#
Re: Nagios XI Ramdisk filling up
Posted: Mon Feb 01, 2016 1:13 pm
by lmiltchev
Can you run the following commands and show the output in code wraps?
Code: Select all
grep ramdisk /usr/local/nagios/etc/nagios.cfg
ll -d /var/nagiosramdisk
ll /var/nagiosramdisk
ll /var/nagiosramdisk/spool
chage -l nagios
tail -50 /var/log/cron
Re: Nagios XI Ramdisk filling up
Posted: Mon Feb 01, 2016 1:16 pm
by lmiltchev
Change this line in the confing.inc.php:
Code: Select all
$cfg['perfdata_spool'] = 'var/nagiosramdisk/spool/perfdata/';
to:
Code: Select all
$cfg['perfdata_spool'] = '/var/nagiosramdisk/spool/perfdata/';
and restart httpd.
Re: Nagios XI Ramdisk filling up
Posted: Tue Feb 02, 2016 9:51 am
by JakeHatMacys
lmiltchev wrote:Change this line in the confing.inc.php:
Code: Select all
$cfg['perfdata_spool'] = 'var/nagiosramdisk/spool/perfdata/';
to:
Code: Select all
$cfg['perfdata_spool'] = '/var/nagiosramdisk/spool/perfdata/';
and restart httpd.
Done.
Re: Nagios XI Ramdisk filling up
Posted: Tue Feb 02, 2016 9:58 am
by JakeHatMacys
lmiltchev wrote:Can you run the following commands and show the output in code wraps?
Code: Select all
grep ramdisk /usr/local/nagios/etc/nagios.cfg
ll -d /var/nagiosramdisk
ll /var/nagiosramdisk
ll /var/nagiosramdisk/spool
chage -l nagios
tail -50 /var/log/cron
Code: Select all
grep ramdisk /usr/local/nagios/etc/nagios.cfg
service_perfdata_file=/var/nagiosramdisk/service-perfdata
host_perfdata_file=/var/nagiosramdisk/host-perfdata
check_result_path=/var/nagiosramdisk/spool/checkresults
object_cache_file=/var/nagiosramdisk/objects.cache
status_file=/var/nagiosramdisk/status.dat
temp_path=/var/nagiosramdisk/tmp
ll -d /var/nagiosramdisk
drwxrwxrwt 5 nagios nagios 180 Feb 2 09:56 /var/nagiosramdisk
Code: Select all
ll /var/nagiosramdisk
total 5200
drwxr-xr-x 2 nagios nagios 40 Dec 16 08:48 checkresults
-rw-r--r-- 1 nagios nagios 5387 Feb 2 09:56 host-perfdata
-rw-r--r-- 1 nagios nagios 2001577 Jan 29 11:21 objects.cache
-rw-r--r-- 1 nagios nagios 14519 Feb 2 09:56 service-perfdata
drwxrwxr-x 5 nagios nagios 100 Dec 16 08:49 spool
-rw-r--r-- 1 nagios nagios 3296116 Feb 2 09:56 status.dat
drwxrwxr-x 2 nagios nagios 40 Feb 1 12:19 tmp
Code: Select all
ll /var/nagiosramdisk/spool
total 0
drwxrwxr-x 2 nagios nagios 40 Feb 1 12:19 checkresults
drwxrwxr-x 2 nagios nagios 164960 Feb 2 09:56 perfdata
drwxrwxr-x 2 nagios nagios 40 Feb 2 09:56 xidpe
Code: Select all
chage -l nagios
Last password change : Sep 23, 2015
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7
Code: Select all
tail -50 /var/log/cron
Feb 2 09:50:01 esu2v236 CROND[28782]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1)
Feb 2 09:50:01 esu2v236 CROND[28764]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1)
Feb 2 09:50:01 esu2v236 CROND[28775]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Feb 2 09:50:01 esu2v236 CROND[28784]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Feb 2 09:50:01 esu2v236 CROND[28772]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Feb 2 09:50:01 esu2v236 CROND[28794]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Feb 2 09:50:01 esu2v236 CROND[28795]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Feb 2 09:51:01 esu2v236 CROND[7782]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Feb 2 09:51:01 esu2v236 CROND[7783]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Feb 2 09:51:01 esu2v236 CROND[7785]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Feb 2 09:51:01 esu2v236 CROND[7788]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Feb 2 09:51:01 esu2v236 CROND[7790]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Feb 2 09:51:01 esu2v236 CROND[7793]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Feb 2 09:51:01 esu2v236 CROND[7786]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Feb 2 09:51:01 esu2v236 CROND[7795]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Feb 2 09:52:01 esu2v236 CROND[30423]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Feb 2 09:52:01 esu2v236 CROND[30429]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Feb 2 09:52:01 esu2v236 CROND[30428]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Feb 2 09:52:01 esu2v236 CROND[30430]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Feb 2 09:52:01 esu2v236 CROND[30427]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Feb 2 09:52:01 esu2v236 CROND[30431]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Feb 2 09:52:01 esu2v236 CROND[30426]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Feb 2 09:52:01 esu2v236 CROND[30425]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Feb 2 09:53:01 esu2v236 CROND[3914]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Feb 2 09:53:01 esu2v236 CROND[3917]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Feb 2 09:53:01 esu2v236 CROND[3915]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Feb 2 09:53:01 esu2v236 CROND[3921]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Feb 2 09:53:01 esu2v236 CROND[3922]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Feb 2 09:53:01 esu2v236 CROND[3919]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Feb 2 09:53:01 esu2v236 CROND[3930]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Feb 2 09:53:01 esu2v236 CROND[3927]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Feb 2 09:54:01 esu2v236 CROND[9014]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Feb 2 09:54:01 esu2v236 CROND[9015]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Feb 2 09:54:01 esu2v236 CROND[9017]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Feb 2 09:54:01 esu2v236 CROND[9022]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Feb 2 09:54:01 esu2v236 CROND[9024]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Feb 2 09:54:01 esu2v236 CROND[9018]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Feb 2 09:54:01 esu2v236 CROND[9023]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Feb 2 09:54:01 esu2v236 CROND[9020]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Feb 2 09:55:01 esu2v236 CROND[15814]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Feb 2 09:55:01 esu2v236 CROND[15817]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Feb 2 09:55:01 esu2v236 CROND[15816]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Feb 2 09:55:01 esu2v236 CROND[15818]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Feb 2 09:55:01 esu2v236 CROND[15819]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1)
Feb 2 09:55:01 esu2v236 CROND[15823]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Feb 2 09:55:01 esu2v236 CROND[15826]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Feb 2 09:55:01 esu2v236 CROND[15829]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1)
Feb 2 09:55:01 esu2v236 CROND[15825]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Feb 2 09:55:01 esu2v236 CROND[15828]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Feb 2 09:55:01 esu2v236 CROND[15827]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Re: Nagios XI Ramdisk filling up
Posted: Tue Feb 02, 2016 12:20 pm
by tgriep
Can you run the following and post the output?
Code: Select all
grep perfdata_spool_dir /usr/local/nagios/etc/pnp/npcd.cfg
ps -ef |grep npcd
ls /var/nagiosramdisk/spool/xidpe | wc -l
ls /var/nagiosramdisk/spool/perfdata/ | wc -l
ls /var/nagiosramdisk/spool/checkresults/ | wc -l
I just want to check a few more settings, etc...
Thanks
Re: Nagios XI Ramdisk filling up
Posted: Tue Feb 02, 2016 1:29 pm
by JakeHatMacys
tgriep wrote:Can you run the following and post the output?
Code: Select all
grep perfdata_spool_dir /usr/local/nagios/etc/pnp/npcd.cfg
ps -ef |grep npcd
ls /var/nagiosramdisk/spool/xidpe | wc -l
ls /var/nagiosramdisk/spool/perfdata/ | wc -l
ls /var/nagiosramdisk/spool/checkresults/ | wc -l
I just want to check a few more settings, etc...
Thanks
Sure:
Code: Select all
grep perfdata_spool_dir /usr/local/nagios/etc/pnp/npcd.cfg
# perfdata_spool_dir - where we can find the
# perfdata_spool_dir = </path/to/directory/>
perfdata_spool_dir = /var/nagiosramdisk/spool/perfdata/
# in 'perfdata_spool_dir'
Code: Select all
ps -ef |grep npcd
nagios 16440 1 0 Jan11 ? 00:00:27 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
root 26610 25535 0 13:28 pts/94 00:00:00 grep npcd
ls /var/nagiosramdisk/spool/xidpe | wc -l
0
ls /var/nagiosramdisk/spool/perfdata/ | wc -l
0
ls /var/nagiosramdisk/spool/checkresults/ | wc -l
0
Re: Nagios XI Ramdisk filling up
Posted: Tue Feb 02, 2016 1:55 pm
by tgriep
Thanks, It looks like the files are being processed now.
If everything is working for you, shall we close this post?