Downtime and Acknowledgement not Operating as expected

[NathanS]

For some reason, when trying to acknowledge an alert nagioxi doesn't make the expected change. Tailed the httpd error log and it's coming up with these errors on service page loads:

Code: Select all

[Mon Aug 15 13:18:23 2016] [error] [client 172.21.74.67] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-object-detail.inc.php on line 1139, referer: https://nagios.regulusgroup.net/nagiosxi/includes/components/xicore/status.php?&hostgroup=all-non-windows-hosts&show=services&hoststatustypes=0&servicestatustypes=28&serviceattr=10
[Mon Aug 15 13:18:23 2016] [error] [client 172.21.74.67] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-object-detail.inc.php on line 1139, referer: https://nagios.regulusgroup.net/nagiosxi/includes/components/xicore/status.php?&hostgroup=all-non-windows-hosts&show=services&hoststatustypes=0&servicestatustypes=28&serviceattr=10
[Mon Aug 15 13:18:23 2016] [error] [client 172.21.74.67] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/xicore/status-object-detail.inc.php on line 1139, referer: https://nagios.regulusgroup.net/nagiosxi/includes/components/xicore/status.php?&hostgroup=all-non-windows-hosts&show=services&hoststatustypes=0&servicestatustypes=28&serviceattr=10
[Mon Aug 15 13:18:41 2016] [error] [client 172.21.74.67] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-object-detail.inc.php on line 2474, referer: https://nagios.regulusgroup.net/nagiosxi/includes/components/xicore/status.php?show=servicedetail&host=clt-detour&service=OSsync+State+USG&dest=auto
[Mon Aug 15 13:18:41 2016] [error] [client 172.21.74.67] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-object-detail.inc.php on line 2474, referer: https://nagios.regulusgroup.net/nagiosxi/includes/components/xicore/status.php?show=servicedetail&host=clt-detour&service=OSsync+State+USG&dest=auto
[Mon Aug 15 13:18:41 2016] [error] [client 172.21.74.67] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/xicore/status-object-detail.inc.php on line 2474, referer: https://nagios.regulusgroup.net/nagiosxi/includes/components/xicore/status.php?show=servicedetail&host=clt-detour&service=OSsync+State+USG&dest=auto
[Mon Aug 15 13:19:10 2016] [error] [client 172.21.74.67] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-object-detail.inc.php on line 1139, referer: https://nagios.regulusgroup.net/nagiosxi/includes/components/xicore/status.php?show=services&sortby=&sortorder=asc&host=&hostgroup=&servicegroup=&search=swi82&searchButton=
[Mon Aug 15 13:19:10 2016] [error] [client 172.21.74.67] PHP Notice:  Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-object-detail.inc.php on line 1139, referer: https://nagios.regulusgroup.net/nagiosxi/includes/components/xicore/status.php?show=services&sortby=&sortorder=asc&host=&hostgroup=&servicegroup=&search=swi82&searchButton=
[Mon Aug 15 13:19:10 2016] [error] [client 172.21.74.67] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/xicore/status-object-detail.inc.php on line 1139, referer: https://nagios.regulusgroup.net/nagiosxi/includes/components/xicore/status.php?show=services&sortby=&sortorder=asc&host=&hostgroup=&servicegroup=&search=swi82&searchButton=

Profile.zip attached.

[bwallace]

I'm copying the below steps from this recent thread (same error mssg / outcome)
https://support.nagios.com/forum/viewto ... 16&t=39727

Are you using HTTPS with a self-singed certificate?
Make sure that you have your shortname and FQDN in your /etc/hosts file on your XI server as well, this is usually the cause of it.
Validate the URL in Admin > System Settings > Program URL and External URL.

[NathanS]

Indeed we're using HTTPS and a self signed certificate (though it shows as valid because it's signed with an internal CA). The cname wasn't in the hosts file, but I've updated that now with no change in behavior.

I'm able to schedule downtime, but acknowledges still don't take at all. No error messages on the gui. The acknowledge window pops up, I enter parameters, I click okay, the window goes away aaaaaand..... nothing. No change.

[bwallace]

That's odd - scheduled downtime and acknowledging are run by the cron daemon.
Try restarting cron and see if that fixes the issue by running this command:
service crond restart

Then run this command and post the output here so we can can confirm the cron daemon started:
ps aux |grep cron

[NathanS]

[[redacted]~]$ ps aux | grep cron
root 7204 0.0 0.0 116860 1224 ? Ss 07:28 0:00 crond
nagios 8024 0.0 0.0 106060 1280 ? Ss 07:29 0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios 8025 0.0 0.0 106060 1280 ? Ss 07:29 0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 8029 0.0 0.0 106060 1276 ? Ss 07:29 0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios 8030 0.3 0.1 339452 31876 ? S 07:29 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios 8032 0.2 0.1 329964 22764 ? S 07:29 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios 8034 0.1 0.1 330088 22788 ? S 07:29 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
nagios 8035 0.0 0.0 106060 1280 ? Ss 07:29 0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios 8037 0.0 0.0 106060 1284 ? Ss 07:29 0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios 8038 0.4 0.1 341648 25664 ? S 07:29 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios 8039 0.2 0.1 330140 22864 ? S 07:29 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
[redacted] 9041 0.0 0.0 103304 896 pts/0 R+ 07:29 0:00 grep cron
[[redacted]~]$

[lmiltchev]

What is the Nagios XI version that you are currently using? What kind of user (local or AD/LDAP) is trying to acknowledge the problem? Run the following commands and show the output:

Code: Select all

grep nag /etc/group
chage -l nagios
chage -l apache
grep "<username>"/usr/local/nagios/etc/cgi.cfg
ps -ef | grep "nagios/bin" | grep -v grep
grep check_external /usr/local/nagios/etc/nagios.cfg
ls -lad /usr/local/nagios/var/rw/
ls -la /usr/local/nagios/var/rw/nagios.cmd

where you substitute "<username>" with the actual username.

[NathanS]

As per my attached profile info, I'm running 5.2.9

As for the user(s), we're 100% AD. I've tried establishing new sessions, and I'm getting the same report from multiple users.

Code: Select all

$ grep nag /etc/group
nagios:x:500:nagios,apache,mail,mmcdowell
nagcmd:x:501:nagios,apache,mail,mmcdowell
shawn:x:2000:nagios
jmascio:x:1001:nagios
atyler:x:1002:nagios

Code: Select all

# chage -l nagios
Last password change					: Dec 02, 2015
Password expires					: never
Password inactive					: never
Account expires						: never
Minimum number of days between password change		: 0
Maximum number of days between password change		: 99999
Number of days of warning before password expires	: 7

Code: Select all

# chage -l apache
Last password change					: Nov 12, 2015
Password expires					: never
Password inactive					: never
Account expires						: never
Minimum number of days between password change		: -1
Maximum number of days between password change		: -1
Number of days of warning before password expires	: -1

Code: Select all

# grep -i "nathanshobe" /usr/local/nagios/etc/cgi.cfg 
authorized_for_all_host_commands=nagiosadmin,nagiosadmin,AllenTyler,NathanShobe,AliBehravesh,JenniferGanser,michaelmcdowell,tyronevaughn,shawnford,CraigFlanagan,chriscaffo,dalechatham,ImmanuJeyam,jovinlionnelrichiee,sasikumar,vijaykumar,michaeltraband,clifton.osler,MarinoLini,jasonwatson,franknowicki,michaelmcdowell,AliBehravesh,JenniferGanser,AllenTyler,NathanShobe,jasonwatson,franknowicki
authorized_for_all_hosts=nagiosadmin,nagiosadmin,AllenTyler,NathanShobe,AliBehravesh,JenniferGanser,michaelmcdowell,tyronevaughn,shawnford,CraigFlanagan,chriscaffo,dalechatham,ImmanuJeyam,jovinlionnelrichiee,sasikumar,vijaykumar,michaeltraband,clifton.osler,MarinoLini,jasonwatson,franknowicki,michaelmcdowell,AliBehravesh,JenniferGanser,AllenTyler,NathanShobe,ntramalingam,DarrellReynolds,GeneMcDonough,jeroldsalter,ceasarverdelli,raveendradvn,williammcdonald,donnaseverson,albertobuitrago,jasonwatson,franknowicki
authorized_for_all_service_commands=nagiosadmin,nagiosadmin,AllenTyler,NathanShobe,AliBehravesh,JenniferGanser,michaelmcdowell,tyronevaughn,shawnford,CraigFlanagan,chriscaffo,dalechatham,ImmanuJeyam,jovinlionnelrichiee,sasikumar,vijaykumar,michaeltraband,clifton.osler,MarinoLini,jasonwatson,franknowicki,michaelmcdowell,AliBehravesh,JenniferGanser,AllenTyler,NathanShobe,jasonwatson,franknowicki
authorized_for_all_services=nagiosadmin,nagiosadmin,AllenTyler,NathanShobe,AliBehravesh,JenniferGanser,michaelmcdowell,tyronevaughn,shawnford,CraigFlanagan,chriscaffo,dalechatham,ImmanuJeyam,jovinlionnelrichiee,sasikumar,vijaykumar,michaeltraband,clifton.osler,MarinoLini,jasonwatson,franknowicki,michaelmcdowell,AliBehravesh,JenniferGanser,AllenTyler,NathanShobe,ntramalingam,DarrellReynolds,GeneMcDonough,jeroldsalter,ceasarverdelli,raveendradvn,williammcdonald,donnaseverson,albertobuitrago,jasonwatson,franknowicki
authorized_for_configuration_information=nagiosadmin,nagiosadmin,AllenTyler,NathanShobe,AliBehravesh,JenniferGanser,michaelmcdowell,tyronevaughn,shawnford,CraigFlanagan,chriscaffo,dalechatham,ImmanuJeyam,jovinlionnelrichiee,sasikumar,vijaykumar,michaeltraband,clifton.osler,MarinoLini,jasonwatson,franknowicki,michaelmcdowell,AliBehravesh,JenniferGanser,AllenTyler,NathanShobe,ntramalingam,DarrellReynolds,GeneMcDonough,jeroldsalter,ceasarverdelli,raveendradvn,williammcdonald,donnaseverson,albertobuitrago,jasonwatson,franknowicki
authorized_for_system_commands=nagiosadmin,nagiosadmin,AllenTyler,NathanShobe,AliBehravesh,JenniferGanser,michaelmcdowell,tyronevaughn,shawnford,CraigFlanagan,chriscaffo,dalechatham,ImmanuJeyam,jovinlionnelrichiee,sasikumar,vijaykumar,michaeltraband,clifton.osler,MarinoLini,jasonwatson,franknowicki,michaelmcdowell,AliBehravesh,JenniferGanser,AllenTyler,NathanShobe,jasonwatson,franknowicki
authorized_for_system_information=nagiosadmin,nagiosadmin,AllenTyler,NathanShobe,AliBehravesh,JenniferGanser,michaelmcdowell,tyronevaughn,shawnford,CraigFlanagan,chriscaffo,dalechatham,ImmanuJeyam,jovinlionnelrichiee,sasikumar,vijaykumar,michaeltraband,clifton.osler,MarinoLini,jasonwatson,franknowicki,michaelmcdowell,AliBehravesh,JenniferGanser,AllenTyler,NathanShobe,jasonwatson,franknowicki

Code: Select all

# ps -ef | grep "nagios/bin" | grep -v grep
nagios    2212     1  0 Aug15 ?        00:00:04 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
nagios    2561     1  0 Aug15 ?        00:00:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios   22819     1  0 Aug15 ?        00:10:56 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   22821 22819  0 Aug15 ?        00:00:20 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22822 22819  0 Aug15 ?        00:00:19 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22823 22819  0 Aug15 ?        00:00:35 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22824 22819  0 Aug15 ?        00:00:19 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22825 22819  0 Aug15 ?        00:00:25 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22826 22819  0 Aug15 ?        00:00:30 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22827 22819  0 Aug15 ?        00:00:20 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22828 22819  0 Aug15 ?        00:00:25 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22829 22819  0 Aug15 ?        00:00:35 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22830 22819  0 Aug15 ?        00:00:25 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22831 22819  0 Aug15 ?        00:00:21 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22832 22819  0 Aug15 ?        00:00:30 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22835  2561  0 Aug15 ?        00:00:26 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios   22836 22835  1 Aug15 ?        00:16:57 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios   22887 22819  0 Aug15 ?        00:00:02 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

Code: Select all

# grep check_external /usr/local/nagios/etc/nagios.cfg 
check_external_commands=1

Code: Select all

# ls -lad /usr/local/nagios/var/rw/
drwxrwsr-x 2 nagios nagcmd 4096 Aug 15 13:33 /usr/local/nagios/var/rw/

Code: Select all

# ls -la /usr/local/nagios/var/rw/nagios.cmd
prw-rw---- 1 nagios nagcmd 0 Aug 16 07:51 /usr/local/nagios/var/rw/nagios.cmd

[NathanS]

I don't know why, but after making some changes in the CCM and applying the new config, it's now accepting acknowledgements.

Will report back if it continues to act up.

[lmiltchev]

Interesting... Keep an eye on it, and let us know if you run into the same issue. We will keep the thread open for a while.