NagiosXI checks "an error occurred processing your request"

[QS1]

Nagios XI 2014R1.4
Centos 5.4 VM (32 bit)

On 23SEP2014 I made a large amount of changes to the config file for host/service cleanup and saved changes. I then proceeded to upgrade to R1.4 (from 2.9). The upgrade failed as the configuration file was in error. When I logged into the site it appears the upgrade was successful as 1.4 appears at the bottom left of the GUI. I had to perform several other Write/Verify steps through the CCM and remove additional services that apparently had no hosts.

I am able to apply the configuration successfully but am noticing the checks are all reporting last check of 23SEP. when I force a check through the GUI I receive a generic error "an error occurred processing your request"

I ran through the ./upgrade process again with a successful exit but I still receive the same error when trying to force a check. I've rebooted the VM and see the same results.

[QS1]

Reading other similar posts I checked /usr/local/nagios/var/rw/ and do not have a nagios.cmd file created. I created a new text file called nagios.cmd and started the nagios service..the nagios.cmd file is no longer present when the nagios service is restarted

[Box293]

I would re-run the upgrade again to ensure it is completing correctly.

[QS1]

As explained in the original post the upgrade was re-run and completed successfully. Are you suggesting I perform the upgrade a third time?

[Box293]

Sorry I totally missed that.

The nagios.cmd is created when nagios starts and removed when it stops.

Code: Select all

service nagios status
nagios (pid 24369) is running...

ls -al /usr/local/nagios/var/rw/nagios.cmd
prw-rw---- 1 nagios nagcmd 0 Sep 22 10:32 /usr/local/nagios/var/rw/nagios.cmd

service nagios stop
Stopping nagios: .done.

ls -al /usr/local/nagios/var/rw/nagios.cmd
ls: cannot access /usr/local/nagios/var/rw/nagios.cmd: No such file or directory

If the file does not exist when nagios is running perhaps there is a permission issue.

What is the output of these commands:

Code: Select all

ls -al /usr/local/nagios/var/
ls -al /usr/local/nagios/var/rw/

[QS1]

I am running the last commands given as the nagios.cmd file is present when nagios service is stopped..and removed when nagios service is running


Stopping nagios: /etc/init.d/nagios: line 67: kill: (15595) - No such process
done.
[root@nagiosxi ~]# service nagios start
Starting nagios: done.
[root@nagiosxi ~]# ls -al /usr/local/nagios/var
total 32284
drwxrwxr-x 6 nagios nagios 4096 Sep 24 11:08 .
drwxr-xr-x 9 nagios nagios 4096 Sep 23 15:15 ..
drwxrwxr-x 2 nagios nagios 69632 Sep 23 00:00 archives
-rw-rw-r-- 1 nagios users 10507 Sep 23 15:35 host-perfdata
-rw-r--r-- 1 nagios nagios 6 Sep 24 11:08 nagios.lock
-rw-r--r-- 1 root root 5 Feb 24 2012 nagios.lock~
-rw-rw-r-- 1 nagios nagios 2149074 Sep 24 11:08 nagios.log
-rw------- 1 nagios nagios 712704 Nov 1 2013 nagios.tmp7rbJzk
-rw-r--r-- 1 nagios nagios 5 Sep 24 10:21 ndo2db.lock
-rw-rw-r-- 1 nagios nagios 0 Sep 23 11:57 ndomod.tmp
srwxr-xr-x 1 nagios nagios 0 Sep 24 10:21 ndo.sock
-rw-r--r-- 1 nagios nagios 2253612 Sep 24 10:21 npcd.log
-rw-r--r-- 1 nagios nagios 10485876 Sep 19 00:10 npcd.log.old
-rw-r--r-- 1 nagios nagios 3619394 Sep 23 11:57 objects.cache
-rw-rw-rw- 1 nagios nagios 6913302 Sep 23 15:35 perfdata.log
-rw------- 1 nagios users 6708704 Sep 24 11:08 retention.dat
drwxrwsrwx 2 nagios nagcmd 4096 Sep 24 12:01 rw
-rw-rw-r-- 1 nagios users 2745 Sep 23 15:35 service-perfdata
drwxrwxr-x 5 nagios nagios 4096 Mar 7 2011 spool
drwxr-xr-x 2 nagios nagios 4096 Sep 23 15:35 stats
You have new mail in /var/spool/mail/root
[root@nagiosxi ~]# ls -al /usr/local/nagios/var/rw/
total 584
drwxrwsrwx 2 nagios nagcmd 4096 Sep 24 12:01 .
drwxrwxr-x 6 nagios nagios 4096 Sep 24 11:08 ..
srw-rw---- 1 nagios nagcmd 0 Sep 23 11:57 live
-rw-r--r-- 1 nagios nagcmd 114 Sep 24 12:52 nagios.cmd
srw-rw---- 1 nagios nagcmd 0 Sep 24 11:08 nagios.qh
-rw-rw-r-- 1 nagios nagcmd 570223 Sep 24 12:00 nsca.dump
You have new mail in /var/spool/mail/root
[root@nagiosxi ~]#

[Box293]

nagios.cmd file is present when nagios service is stopped..and removed when nagios service is running

I suspect you have more than one instance of nagios running.

What is the output of these commands:

Code: Select all

service nagios status
ps -Aef | grep nagios
service nagios stop
service nagios status
ps -Aef | grep nagios

[QS1]

..I start nagios (done)..I immediately run service status (not running)

You have new mail in /var/spool/mail/root
[root@nagiosxi ~]# service nagios start
Starting nagios: done.
[root@nagiosxi ~]# service nagios status
nagios is not running
You have new mail in /var/spool/mail/root
[root@nagiosxi ~]# service nagios start
Starting nagios: done.
[root@nagiosxi ~]# service nagios status
nagios is not running
[root@nagiosxi ~]# ps -Aef | grep nagios
nagios 3865 1 0 10:21 ? 00:00:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios 3875 1 0 10:21 ? 00:00:00 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
avahi 3959 1 0 10:21 ? 00:00:00 avahi-daemon: running [nagiosxi.local]
postgres 7391 3714 0 10:32 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(48417) idle
postgres 7427 3714 0 10:32 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(48418) idle
postgres 7428 3714 0 10:32 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(48419) idle
postgres 7434 3714 0 10:32 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(48420) idle
postgres 7437 3714 0 10:32 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(48421) idle
postgres 7440 3714 0 10:32 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(48424) idle
postgres 7444 3714 0 10:32 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(48427) idle
postgres 7452 3714 0 10:32 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(48428) idle
postgres 7469 3714 0 10:32 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(48433) idle
postgres 7696 3714 0 10:33 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(48454) idle
postgres 7798 3714 0 10:33 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(48464) idle
postgres 8012 3714 0 10:34 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(48492) idle
postgres 8591 3714 0 10:36 ? 00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(33339) idle
nagios 11797 3792 0 14:11 ? 00:00:00 crond
nagios 11798 3792 0 14:11 ? 00:00:00 crond
nagios 11799 3792 0 14:11 ? 00:00:00 crond
nagios 11800 3792 0 14:11 ? 00:00:00 crond
nagios 11801 3792 0 14:11 ? 00:00:00 crond
nagios 11804 11798 0 14:11 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios 11806 11801 0 14:11 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios 11808 11797 0 14:11 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios 11809 11806 1 14:11 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios 11812 11804 1 14:11 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
nagios 11813 11800 0 14:11 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios 11814 11808 1 14:11 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios 11816 11813 1 14:11 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios 11818 11799 0 14:11 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 11820 11818 1 14:11 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
postgres 11835 3714 0 14:11 ? 00:00:00 postgres: nagiosxi nagiosxi 127.0.0.1(59684) idle
postgres 11839 3714 0 14:11 ? 00:00:00 postgres: nagiosxi nagiosxi 127.0.0.1(59686) idle
postgres 11842 3714 0 14:11 ? 00:00:00 postgres: nagiosxi nagiosxi 127.0.0.1(59687) idle
postgres 11845 3714 0 14:11 ? 00:00:00 postgres: nagiosxi nagiosxi 127.0.0.1(59688) idle
postgres 11849 3714 0 14:11 ? 00:00:00 postgres: nagiosxi nagiosxi 127.0.0.1(59689) idle
nagios 11963 11809 0 14:11 ? 00:00:00 sh -c /usr/bin/iostat -c 5 2 | tail --lines=2 | head --lines=1 | awk '{ print $1,$2,$3,$4,$5,$6 }'
nagios 11964 11963 0 14:11 ? 00:00:00 /usr/bin/iostat -c 5 2
nagios 11965 11963 0 14:11 ? 00:00:00 tail --lines=2
nagios 11966 11963 0 14:11 ? 00:00:00 head --lines=1
nagios 11967 11963 0 14:11 ? 00:00:00 awk { print $1,$2,$3,$4,$5,$6 }
root 11969 16432 0 14:11 pts/4 00:00:00 grep nagios
postgres 25930 3714 0 11:31 ? 00:00:00 postgres: nagiosxi nagiosxi 127.0.0.1(56878) idle
You have new mail in /var/spool/mail/root

[Box293]

What is the output of these commands?

Code: Select all

tail -n 30 /usr/local/nagios/var/nagios.log

tail -n 30 /usr/local/nagios/var/nagios.debug

tail -n 30 /var/log/messages

[QS1]

tail -n 30 /usr/local/nagios/var/nagios.log

[1411588941] wproc: Registry request: name=Core Worker 9105;pid=9105
[1411588941] Error: Could not load module '/usr/lib/check_mk/livestatus.o' -> /usr/lib/check_mk/livestatus.o: undefined symbol: last_command_check
[1411588941] Error: Failed to load module '/usr/lib/check_mk/livestatus.o'.
[1411588941] ndomod: NDOMOD 2.0.0 (02-28-2014) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
[1411588941] ndomod: Successfully connected to data sink. 0 queued items to flush.
[1411588941] ndomod registered for process data
[1411588941] ndomod registered for log data'
[1411588941] ndomod registered for system command data'
[1411588941] ndomod registered for event handler data'
[1411588941] ndomod registered for notification data'
[1411588941] ndomod registered for comment data'
[1411588941] ndomod registered for downtime data'
[1411588941] ndomod registered for flapping data'
[1411588941] ndomod registered for program status data'
[1411588941] ndomod registered for host status data'
[1411588941] ndomod registered for service status data'
[1411588941] ndomod registered for adaptive program data'
[1411588942] ndomod registered for adaptive host data'
[1411588942] ndomod registered for adaptive service data'
[1411588942] ndomod registered for external command data'
[1411588942] ndomod registered for aggregated status data'
[1411588942] ndomod registered for retention data'
[1411588942] ndomod registered for contact data'
[1411588942] ndomod registered for contact notification data'
[1411588942] ndomod registered for acknowledgement data'
[1411588942] ndomod registered for state change data'
[1411588942] ndomod registered for contact status data'
[1411588942] ndomod registered for adaptive contact data'
[1411588942] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1411588942] Error: Module loading failed. Aborting.
[root@nagiosxi ~]#

[root@nagiosxi ~]# tail -n 30 /usr/local/nagios/var/nagios.debug
tail: cannot open `/usr/local/nagios/var/nagios.debug' for reading: No such file or directory

[root@nagiosxi ~]# tail -n 30 /var/log/messages
Sep 24 16:18:36 nagiosxi xinetd[3527]: EXIT: nsca status=0 pid=13162 duration=0(sec)
Sep 24 16:18:37 nagiosxi clamd[3542]: SelfCheck: Database status OK.
Sep 24 16:20:43 nagiosxi xinetd[3527]: START: nsca pid=13680 from=10.0.21.100
Sep 24 16:20:43 nagiosxi xinetd[3527]: EXIT: nsca status=0 pid=13680 duration=0(sec)
Sep 24 16:21:09 nagiosxi xinetd[3527]: START: nsca pid=13862 from=10.0.9.72
Sep 24 16:21:09 nagiosxi xinetd[3527]: EXIT: nsca status=0 pid=13862 duration=0(sec)
Sep 24 16:22:46 nagiosxi xinetd[3527]: START: nsca pid=14229 from=10.0.9.192
Sep 24 16:22:46 nagiosxi xinetd[3527]: EXIT: nsca status=0 pid=14229 duration=0(sec)
Sep 24 16:22:50 nagiosxi xinetd[3527]: START: nsca pid=14231 from=10.0.21.100
Sep 24 16:22:50 nagiosxi xinetd[3527]: EXIT: nsca status=0 pid=14231 duration=0(sec)
Sep 24 16:24:57 nagiosxi xinetd[3527]: START: nsca pid=14711 from=10.0.21.100
Sep 24 16:24:57 nagiosxi xinetd[3527]: EXIT: nsca status=0 pid=14711 duration=0(sec)
Sep 24 16:27:04 nagiosxi xinetd[3527]: START: nsca pid=15356 from=10.0.21.100
Sep 24 16:27:04 nagiosxi xinetd[3527]: EXIT: nsca status=0 pid=15356 duration=0(sec)
Sep 24 16:27:22 nagiosxi automount[3431]: update_negative_cache: key ".directory" not found in map.
Sep 24 16:27:22 nagiosxi automount[3431]: create_udp_client:98: hostname lookup failed: No such process
Sep 24 16:27:22 nagiosxi automount[3431]: create_tcp_client:328: hostname lookup failed: No such process
Sep 24 16:27:22 nagiosxi automount[3431]: lookup_mount: exports lookup failed for .directory
Sep 24 16:27:22 nagiosxi automount[3431]: update_negative_cache: key ".directory" not found in map.
Sep 24 16:28:37 nagiosxi clamd[3542]: SelfCheck: Database status OK.
Sep 24 16:29:11 nagiosxi xinetd[3527]: START: nsca pid=15844 from=10.0.21.100
Sep 24 16:29:11 nagiosxi xinetd[3527]: EXIT: nsca status=0 pid=15844 duration=0(sec)
Sep 24 16:29:38 nagiosxi xinetd[3527]: START: nsca pid=15900 from=10.0.9.72
Sep 24 16:29:38 nagiosxi xinetd[3527]: EXIT: nsca status=0 pid=15900 duration=0(sec)
Sep 24 16:31:17 nagiosxi xinetd[3527]: START: nsca pid=16385 from=10.0.21.100
Sep 24 16:31:18 nagiosxi xinetd[3527]: EXIT: nsca status=0 pid=16385 duration=1(sec)
Sep 24 16:32:47 nagiosxi xinetd[3527]: START: nsca pid=16720 from=10.0.9.192
Sep 24 16:32:48 nagiosxi xinetd[3527]: EXIT: nsca status=0 pid=16720 duration=1(sec)
Sep 24 16:33:25 nagiosxi xinetd[3527]: START: nsca pid=16928 from=10.0.21.100
Sep 24 16:33:25 nagiosxi xinetd[3527]: EXIT: nsca status=0 pid=16928 duration=0(sec)
[root@nagiosxi ~]#