perfdata not running

[onzyone]

Hello There,

I have an strange issue with graph data.
1/ I have restated all of the services a number of times (nagios, npcd, ndo2db)
2/ I have also noticed that files are been written into: /usr/local/nagios/var/spool/xidpe. (I have also sopped all services / removed this folder and recreated it per the FAQ)
3/ More over I have also bumped up the logging level in npcd to 2, and I see it polling /usr/local/nagios/var/spool/perfdata/ every 15 sec per the config

So I guess I am not sure how the files should get from xidpe folder to perfdata folder to get processed by the npcd ...
we have traced it down to this not working ... nothing in the httpd logs and we have restarted httpd many times

perfdataproc.php
$dest=grab_array_var($cfg,'perfdata_spool',"/usr/local/nagios/var/spool/perfdata/");
$xidpe=grab_array_var($cfg,'xidpe_dir',"/usr/local/nagios/var/spool/xidpe/");


Running the 7 node free license of Nagios XI 2011R3.3 (Nagios 3.4.1)
On RHEL 6.2 box

Thanks,
Jason.

[mguthrie]

Access the Admin menu in XI, do all of the subsystem processes appear to be running ok?

[onzyone]

*update*
when manually moving all the files located in this folder /usr/local/nagios/var/spool/xidpe/ to /usr/local/nagios/var/spool/perfdata/
npcd picks them up and proccess then and I get the graphs that I want.

So the issue is with the php "cron" file. I can put a linux cron work around in place, but I need to know what happened here if we were to extend our license past 7 nodes.

All green:

Process Settings
External Commands
Active Service Checks
Passive Service Checks
Active Host Checks
Passive Host Checks
Notifications
Event Handlers
Flap Detection
Performance Data
Service Obsession
Host Obsession
Last Updated: 2012-12-18 08:37:08


This how ever
green:
Monitoring Engine
Performance Grapher
Database Backend

All red (last ran 4 days ago)
Database Maintenance
Command Subsystem
Event Manager
Feed Processor
Report Engine
Cleaner
Nonstop Operations Manager
System Statistics

[mguthrie]

Try:

Code: Select all

service crond restart

Also, make sure that all of the log files in /usr/local/nagiosxi/var are owned nagios:nagios.

Can you post the contents of /etc/cron.d/nagiosxi

[onzyone]

Code: Select all

$ service crond restart
Stopping crond:                                            [  OK  ]
Starting crond:                                            [  OK  ]
$ ps -ef |grep crond |grep -v grep
root     17687     1  0 11:28 ?        00:00:00 crond

(this looks like it didn't help. It was the first thing that I did before replying to this post and the files are still building up in xipde folder)


looks like all files are nagios:users, but this has been the case since install in Sep.

Code: Select all

-rw-r--r--  1 nagios users      0 Dec 13 18:59 nom.log
-rw-r--r--  1 nagios users      0 Dec 13 18:59 reportengine.log
-rw-r--r--  1 nagios users      0 Dec 13 18:59 corelog.diff
-rw-r--r--  1 nagios users      6 Dec 13 18:59 corelog.data
-rwxrwxr-x  1 nagios users   1268 Dec 13 18:59 corelog.newobjects
-rw-r--r--  1 nagios users      0 Dec 14 03:45 cleaner.log
-rw-r--r--  1 nagios users    200 Dec 14 03:45 cleaner.log-20121214.gz
-rw-r--r--  1 nagios users      0 Dec 14 03:45 cmdsubsys.log
-rw-r--r--  1 nagios users     47 Dec 14 03:45 cmdsubsys.log-20121214.gz
-rw-r--r--  1 nagios users      0 Dec 14 03:45 dbmaint.log
-rw-r--r--  1 nagios users      0 Dec 14 03:45 eventman.log
-rw-r--r--  1 nagios users    652 Dec 14 03:45 dbmaint.log-20121214.gz
-rw-r--r--  1 nagios users     43 Dec 14 03:45 eventman.log-20121214.gz
-rw-r--r--  1 nagios users      0 Dec 14 03:45 feedproc.log
-rw-r--r--  1 nagios users     45 Dec 14 03:45 feedproc.log-20121214.gz
-rw-r--r--  1 nagios users      0 Dec 14 03:45 perfdataproc.log
-rw-r--r--  1 nagios users    165 Dec 14 03:45 perfdataproc.log-20121214.gz
-rw-r--r--  1 nagios users      0 Dec 14 03:45 recurringdowntime.log
-rw-r--r--  1 nagios users    122 Dec 14 03:45 recurringdowntime.log-20121214.gz
-rw-r--r--  1 nagios users      0 Dec 14 03:45 sysstat.log
-rw-r--r--  1 nagios users   1377 Dec 14 03:45 sysstat.log-20121214.gz
drwxr-xr-x  2 nagios nagios  4096 Dec 17 14:03 subsys

cat /etc/cron.d/nagiosxi

Code: Select all

# /etc/cron.d/nagiosxi: crontab fragment for nagiosxi

# Backup MySQL & PostgreSQL Databases
0   7 * * * root   /root/scripts/automysqlbackup
0   8 * * * root   /root/scripts/autopostgresqlbackup

*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
01  * * * * nagios /usr/local/nagiosxi/cron/recurringdowntime.pl > /usr/local/nagiosxi/var/recurringdowntime.log 2>&1

[mguthrie]

Lets try the following:

Code: Select all

chown nagios.nagios /usr/local/nagiosxi/var/*.log

Then check to see if the jobs start to kick off after a minute or so with the following command:

Code: Select all

ps aux | grep php

FYI, there is also a 20 node license available if you just need a small instance.

[onzyone]

This didn't work either ... (didn't see any php commands). Is there a way to check the cron logs?

Code: Select all

-rw-r--r--  1 nagios nagios     0 Dec 13 18:59 nom.log
-rw-r--r--  1 nagios nagios     0 Dec 13 18:59 reportengine.log
-rw-r--r--  1 nagios users      0 Dec 13 18:59 corelog.diff
-rw-r--r--  1 nagios users      6 Dec 13 18:59 corelog.data
-rwxrwxr-x  1 nagios users   1268 Dec 13 18:59 corelog.newobjects
-rw-r--r--  1 nagios nagios     0 Dec 14 03:45 cleaner.log
-rw-r--r--  1 nagios users    200 Dec 14 03:45 cleaner.log-20121214.gz
-rw-r--r--  1 nagios nagios     0 Dec 14 03:45 cmdsubsys.log
-rw-r--r--  1 nagios users     47 Dec 14 03:45 cmdsubsys.log-20121214.gz
-rw-r--r--  1 nagios nagios     0 Dec 14 03:45 dbmaint.log
-rw-r--r--  1 nagios nagios     0 Dec 14 03:45 eventman.log
-rw-r--r--  1 nagios users    652 Dec 14 03:45 dbmaint.log-20121214.gz
-rw-r--r--  1 nagios users     43 Dec 14 03:45 eventman.log-20121214.gz
-rw-r--r--  1 nagios nagios     0 Dec 14 03:45 feedproc.log
-rw-r--r--  1 nagios users     45 Dec 14 03:45 feedproc.log-20121214.gz
-rw-r--r--  1 nagios nagios     0 Dec 14 03:45 perfdataproc.log
-rw-r--r--  1 nagios users    165 Dec 14 03:45 perfdataproc.log-20121214.gz
-rw-r--r--  1 nagios nagios     0 Dec 14 03:45 recurringdowntime.log
-rw-r--r--  1 nagios users    122 Dec 14 03:45 recurringdowntime.log-20121214.gz
-rw-r--r--  1 nagios nagios     0 Dec 14 03:45 sysstat.log
-rw-r--r--  1 nagios users   1377 Dec 14 03:45 sysstat.log-20121214.gz

I looked at some of the gz files and found that there was this error in there "mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory" (this started back on the 10th of Dec, and the issue that we are taking about started on the 13th at around 19:00. I know for sure that this folder exists and that there are files in it.

Code: Select all

zcat /usr/local/nagiosxi/var/perfdataproc.log-20121210.gz
Outbound data DISABLED Mon, 10 Dec 2012 03:40:01 -0500
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory

DONE. Processed 0 files.

zcat /usr/local/nagiosxi/var/perfdataproc.log-20121214.gz
Outbound data DISABLED Thu, 13 Dec 2012 18:59:01 -0500
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory
mv: cannot stat `/usr/local/nagios/var/spool/xidpe/*': No such file or directory

looks like this may be the issue:

Code: Select all

Dec 18 13:35:01 priappvmhpchn01 crond[3115]: (nagios) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)

$ chage -l nagios
Last password change					: Sep 14, 2012
Password expires					: [b]Dec 13, 2012[/b]
Password inactive					: Mar 18, 2013
Account expires						: never
Minimum number of days between password change		: 1
Maximum number of days between password change		: 90
Number of days of warning before password expires	: 14

*update*

I updated nagios user with the following command and it is now working ... (lol one of the oldest issues in the book sorry for taking so much of your time)

Code: Select all

chage -I -1 -m 0 -M 99999 -E -1 nagios

Thank you very much for your help. I will look at the small cluster 20 node lic as well.

[mguthrie]

Well hey, it's always better to have it working, regardless of the cause! I think it would have been a few more replies before we would have looked there, so nice catch!