Nagios 4.x CGI commands

[sebastiaopburnay]

Hi!

I have begun using the more recent Nagios' 4.x and I'm understanding it to be quite different from the previous 3.x versions, specially in terms of its Core Workers.

I'm having trouble with the cgi commands, the ones submitted via web-interface.

For example, when attempting to force a host check with 'Re-schedule the next check of this host' option, the browser hangs and I notice that a 'cmd.cgi' process is running for each command that is hanging, that process runs under the www-data user:

Code: Select all

root@myNagios4Server:~# ps -eF | grep nagios
nagios     878     1  0  6358  1156   1 14:54 ?        00:00:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
www-data  1344  1249  0  1609  1192   2 15:09 ?        00:00:00 /usr/local/nagios/sbin/cmd.cgi
www-data  1415  1250  0  1609  1196   2 15:14 ?        00:00:00 /usr/local/nagios/sbin/cmd.cgi
root      1417  1188  0  2027   936   2 15:14 pts/0    00:00:00 grep --color=auto nagios

There must be some kind of configuration/ownership/permission that I've not set right

Thank you in advance for your help

[abrist]

What are the permissions on the cgis?

Code: Select all

ls -la /usr/local/nagios/sbin/

Group settings?

Code: Select all

grep nag /etc/group

Did you specify a command group when you compiled nagios 4?

[sebastiaopburnay]

Yes abrist, I've installed the nagios-4.0.2 and after I had to install the «patch» nagios-4.0.2rc1.

What are the permissions on the cgis?

Code: Select all

ls -la /usr/local/nagios/sbin/

CGIs:

Code: Select all

root@myServer:/# ls -la /usr/local/nagios/sbin/
total 4072
drwxrwxr-x 2 nagios nagios   4096 Feb 18 12:16 .
drwxr-xr-x 8 root   root     4096 Feb 18 12:15 ..
-rwxrwxr-x 1 nagios nagios 293696 Feb 18 12:16 avail.cgi
-rwxrwxr-x 1 nagios nagios 288008 Feb 18 12:16 cmd.cgi
-rwxrwxr-x 1 nagios nagios 260848 Feb 18 12:16 config.cgi
-rwxrwxr-x 1 nagios nagios 305952 Feb 18 12:16 extinfo.cgi
-rwxrwxr-x 1 nagios nagios 256936 Feb 18 12:16 histogram.cgi
-rwxrwxr-x 1 nagios nagios 236328 Feb 18 12:16 history.cgi
-rwxrwxr-x 1 nagios nagios 232208 Feb 18 12:16 notifications.cgi
-rwxrwxr-x 1 nagios nagios 228072 Feb 18 12:16 outages.cgi
-rwxrwxr-x 1 nagios nagios 232184 Feb 18 12:16 showlog.cgi
-rwxrwxr-x 1 nagios nagios 305976 Feb 18 12:16 status.cgi
-rwxrwxr-x 1 nagios nagios 252912 Feb 18 12:16 statusmap.cgi
-rwxrwxr-x 1 nagios nagios 248632 Feb 18 12:16 statuswml.cgi
-rwxrwxr-x 1 nagios nagios 232216 Feb 18 12:16 statuswrl.cgi
-rwxrwxr-x 1 nagios nagios 256840 Feb 18 12:16 summary.cgi
-rwxrwxr-x 1 nagios nagios 244544 Feb 18 12:16 tac.cgi
-rwxrwxr-x 1 nagios nagios 265136 Feb 18 12:16 trends.cgi

Group settings?

Code: Select all

grep nag /etc/group

Did you specify a command group when you compiled nagios 4?

Group Settings

Code: Select all

root@myServer:/# grep nag /etc/group
nagios:x:1001:nagios,www-data
nagcmd:x:1002:nagios,www-data

[abrist]

Could you try to submit a check again, and then get a tail of:

Code: Select all

tail -25 /var/log/messages
tail -25 /usr/local/nagios/var/nagios.log

[sebastiaopburnay]

Sorry for the long time to reply, It's been a crazy day at the offoce

Well seen abrist.

The forced re-schedule gave the same kind of error output either in /var/log/syslog and /usr/local/nagios/var/nagios.log:

Code: Select all

May 26 18:56:02 MyNagiosServer nagios: EXTERNAL COMMAND: SCHEDULE_FORCED_SVC_CHECK;MyDBServer;C-NDOUtils_Size;1401126961
May 26 18:56:03 MyNagiosServer nagios: EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;MyDBServer;C-NDOUtils_Size;OK;OK - NDOUtils size is 2.37 GB
May 26 18:56:03 MyNagiosServer nagios: Error: External command failed -> PROCESS_SERVICE_CHECK_RESULT;MyDBServer;C-NDOUtils_Size;OK;OK - NDOUtils size is 2.37 GB
May 26 18:56:03 MyNagiosServer nagios: External command error: Command failed

It seems to me that the check is NRPE check retreives normal results, but somehow, Nagios is not able to process them

I will investigate, and try to fix it on my own.

Either way, your knowlege and ideas are welcome.

Best regards,
sebastiaopburnay

[sebastiaopburnay]

This is really annoying,

The nagios service starts OK, it launches the core workers, but I still can not force command execution from CGUI (Web-Interface).

It even gives an error upon service checks made by Nagios 'natural' scheduling

Does this ring a bell?

Here I post a log form a Nagios' restart:

Code: Select all

May 27 15:16:23 MYNAGIOSSERVER nagios: Caught SIGTERM, shutting down...
May 27 15:16:23 MYNAGIOSSERVER nagios: Successfully shutdown... (PID=1668)
May 27 15:16:23 MYNAGIOSSERVER nagios: Event broker module 'NERD' deinitialized successfully.
May 27 15:16:23 MYNAGIOSSERVER nagios: Warning: external_command_buffer_slots is deprecated and will be removed. All commands are always processed upon arrival
May 27 15:16:23 MYNAGIOSSERVER nagios: Nagios 4.0.2 starting... (PID=1841)
May 27 15:16:23 MYNAGIOSSERVER nagios: Local time is Tue May 27 15:16:23 WEST 2014
May 27 15:16:23 MYNAGIOSSERVER nagios: LOG VERSION: 2.0
May 27 15:16:23 MYNAGIOSSERVER nagios: qh: Socket '/usr/local/nagios/var/rw/nagios.qh' successfully initialized
May 27 15:16:23 MYNAGIOSSERVER nagios: qh: core query handler registered
May 27 15:16:23 MYNAGIOSSERVER nagios: nerd: Channel hostchecks registered successfully
May 27 15:16:23 MYNAGIOSSERVER nagios: nerd: Channel servicechecks registered successfully
May 27 15:16:23 MYNAGIOSSERVER nagios: nerd: Channel opathchecks registered successfully
May 27 15:16:23 MYNAGIOSSERVER nagios: nerd: Fully initialized and ready to rock!
May 27 15:16:23 MYNAGIOSSERVER nagios: wproc: Successfully registered manager as @wproc with query handler
May 27 15:16:23 MYNAGIOSSERVER nagios: wproc: Registry request: name=Core Worker 1842;pid=1842
May 27 15:16:23 MYNAGIOSSERVER nagios: wproc: Registry request: name=Core Worker 1847;pid=1847
May 27 15:16:23 MYNAGIOSSERVER nagios: wproc: Registry request: name=Core Worker 1844;pid=1844
May 27 15:16:23 MYNAGIOSSERVER nagios: wproc: Registry request: name=Core Worker 1843;pid=1843
May 27 15:16:23 MYNAGIOSSERVER nagios: wproc: Registry request: name=Core Worker 1845;pid=1845
May 27 15:16:23 MYNAGIOSSERVER nagios: wproc: Registry request: name=Core Worker 1846;pid=1846
May 27 15:16:24 MYNAGIOSSERVER nagios: Successfully launched command file worker with pid 1848
May 27 15:16:54 MYNAGIOSSERVER nagios: EXTERNAL COMMAND: SCHEDULE_FORCED_SVC_CHECK;OTHERSERVER;DNV-PING;1401200213
May 27 15:16:56 MYNAGIOSSERVER nagios: EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;MYNAGIOSSERVER;C-Current_Load;OK;OK - load average: 0.01, 0.02, 0.03
May 27 15:16:56 MYNAGIOSSERVER nagios: Error: External command failed -> PROCESS_SERVICE_CHECK_RESULT;MYNAGIOSSERVER;C-Current_Load;OK;OK - load average: 0.01, 0.02, 0.03
May 27 15:16:56 MYNAGIOSSERVER nagios: External command error: Command failed
May 27 15:16:58 MYNAGIOSSERVER nagios: EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;OTHERSERVER;DNV-PING;OK;PING OK - Packet loss = 0%, RTA = 0.31 ms
May 27 15:16:58 MYNAGIOSSERVER nagios: Error: External command failed -> PROCESS_SERVICE_CHECK_RESULT;OTHERSERVER;DNV-PING;OK;PING OK - Packet loss = 0%, RTA = 0.31 ms
May 27 15:16:58 MYNAGIOSSERVER nagios: External command error: Command failed

[sreinhardt]

It seems that you are hitting a bug with core 4 that we recently found with submitting immediate checks. I know one of our core devs has been working on it. Let me get an update from him.

[sebastiaopburnay]

It seems that you are hitting a bug with core 4 that we recently found with submitting immediate checks. I know one of our core devs has been working on it. Let me get an update from him.

OK, I'll keep waiting.

Worst case scenario, I'll downgrade to a 3.x version

Best regards,
sebastiaopburnay

[sreinhardt]

Downgrading would technically work, but we don't want you to have to do that! Just to give you an update, the patch has been submitted, and we are doing some internal testing before the next core release. If you absolutely need it now, you can pull from github at https://github.com/NagiosEnterprises/na ... ult-output

This is the specific patch: https://github.com/NagiosEnterprises/na ... a91e548dbc

[sebastiaopburnay]

(...) we are doing some internal testing before the next core release. If you absolutely need it now, you can pull from github at https://github.com/NagiosEnterprises/na ... ult-output

This is the specific patch: https://github.com/NagiosEnterprises/na ... a91e548dbc

Maybe I'll wait for a newer Nagios' core version release

I'll keep this post open until a stable version fixes this issue