Event Handler Command not executing

[socgenindia]

Hi,

We have event handler enabled for one service , when service reaches the hard state nagios.log shows event handler has been evoked , but it doesnt execute the script inside the eventhandler.

/var/log/message

May 18 16:29:37 <nagios server> nagios: SERVICE EVENT HANDLER: <server name>;Lin_Process_cron;CRITICAL;HARD;1;SG_event_handler_cron

nagios.log

[1495117777] SERVICE EVENT HANDLER:<server name> ;Lin_Process_cron;CRITICAL;HARD;1;SG_event_handler_cron


command.cfg

SG_event_handler_cron /produits/nagiosXI/nagios/etc/script/cron.sh

sample script.

cat /produits/nagiosXI/nagios/etc/script/cron.sh

#!/bin/bash

echo "executing the script"

echo "Hi" > /produits/nagiosXI/nagios/etc/script/cron.txt


could you please help in this.

[tgriep]

When a script it run by Nagios, it is run as the nagios user account so the script that you have created has to be able to be executed by the nagios user as well as the commands in the script and any folders the script needs to access.
In your example, verify that the /produits/nagiosXI/nagios/etc/script/cron.sh script can be run by the nagios user and that the folder - file /produits/nagiosXI/nagios/etc/script/cron.txt can be written by the nagios user.

To test this, login as root to the Nagios server and change to the nagios user account by running

Code: Select all

su nagios

Then try and run the script by running the following

Code: Select all

/produits/nagiosXI/nagios/etc/script/cron.sh

If it fails, verify that the file and folders are accessible be the nagios user and that should fix the issue for you.

[socgenindia]

With nagios user i am able to run that script.

[dwhitfield]

What is the output of ll /produits/nagiosXI/nagios/etc/script/cron.sh?

Also, can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the ***Download Profile*** button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+, Show Profile if older) in the thread (that will at least get us some info). This will give us access to many of the logs we would otherwise ask for individually. If security is a concern, you can unzip the profile take out what you like, and then zip it up again. We may end up needing something you remove, but we can ask for that specifically.

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

[socgenindia]

Hello,

I have PM the nagios profile to you.

when i run the script using the nagios user it just create the file cron.txt
echo "Hi" > /produits/nagiosXI/nagios/etc/script/cron.txt.

However we run the nagios with different user and with that user also i am able to run the script.

ocsp , notification are working fine , but not event handler.

where do we find the log what actually event handler is doing , i mean how it is executing the script.

I have enabled the debug option , but still unable to find the issue.

[dwhitfield]

In order to get good logs, you'll need to upgrade to 5.4.4...

OR you can modify your /etc/cron.d/nagiosxi. Before I give you instructions for this, please note that you will either need to set up a log rotation policy, or you will need to change this back.

Replace your current /etc/cron.d/nagiosxi with the following:

Code: Select all

# /etc/cron.d/nagiosxi: crontab fragment for nagiosxi

# Backup MySQL & PostgreSQL Databases
0   7 * * * root   /root/scripts/automysqlbackup
0   7 * * * root   /root/scripts/autopostgresqlbackup > /dev/null 2>&1

*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php >> /usr/local/nagiosxi/var/dbmaint.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1
01  * * * * nagios /usr/local/nagiosxi/cron/recurringdowntime.pl >> /usr/local/nagiosxi/var/recurringdowntime.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1

Of particular interest is /usr/local/nagiosxi/var/event_handler.log but I am giving you all of them because often these logs go together when trying to piece together what is actually happening.

Does it work if you run nagios with the appropriate user?

What's the output of the following?

Code: Select all

su nagios
umask

I mentioned this in the PM, but will mention it again. Can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the ***Download Profile*** button towards the top. Since you are on an older version, I can tell you that it is NOT "Show Profile".

If you cannot download the profile, that is probably due to a modified sudoers. Instructions for resolving that are at https://support.nagios.com/kb/article.p ... ategory=44

[socgenindia]

Hi,

Please find the command output
su nagios
umask

0066.

when i checked in eventman.log only global event handlers are processed , not local event handlers.


.....PROCESS EVENT: ID=8489033, SOURCE=2, TYPE=2, TIME=2017-05-22 15:23:15
*** GLOBAL HANDLER...
Array
(
[event_id] => 8489033
[event_source] => 2
[event_type] => 2
[event_time] => 2017-05-22 15:23:15
[event_meta] => Array
(
[notification-type] => service
[contact] => nagiosadmin
[contactemail] => nagios@localhost
[type] => PROBLEM
[escalated] => 0
[author] =>
[comments] =>
[host] => <server name>
[hostaddress] => <server IP>
[hostalias] => <server name>
[hostdisplayname] => <server name>
[service] => Lin_Process_cron
[hoststate] => UP
[hoststateid] => 0
[servicestate] => CRITICAL
[servicestateid] => 2
[lastservicestate] => CRITICAL
[lastservicestateid] => 2
[servicestatetype] => HARD
[currentattempt] => 1
[maxattempts] => 1
[serviceeventid] => 1730653
[serviceproblemid] => 794910
[serviceoutput] => PROCS CRITICAL: 1 process with command name crond
[longserviceoutput] =>
[datetime] => Mon May 22 15:23:15 CEST 2017
)

[logging_enabled] => 1
)
*** GLOBAL HANDLER (snmptrapsender)...

[tgriep]

Can you post the full command definition for the Event Handler command as well as the service check that is using the Event Handler so we can check them for accuracy?

Then, login to the Nagios server in a shell, run the following and post the output.

Code: Select all

su nagios
ls -l /produits/nagiosXI/nagios/etc/script/cron.sh
bash -x /produits/nagiosXI/nagios/etc/script/cron.sh

[socgenindia]

Hi,

Please find the output

ls -l /produits/nagiosXI/nagios/etc/script/cron.sh
-rwxrwxr-x 1 apache monlc 94 May 22 12:44 /produits/nagiosXI/nagios/etc/script/cron.sh

bash -x /produits/nagiosXI/nagios/etc/script/cron.sh
+ echo 'executing the script'
executing the script
+ echo Hi

cat /produits/nagiosXI/nagios/etc/script/cron.sh

#!/bin/bash

echo "executing the script"

echo "Hi" > /usr/local/nagios/etc/script/cron.txt

In our setup nagios runs with mlcadm user ...


with mlcadm user when i run this below command .
bash-4.2$ ls -l /produits/nagiosXI/nagios/etc/script/cron.sh
-rwxrwxr-x 1 apache monlc 94 May 22 12:44 /produits/nagiosXI/nagios/etc/script/cron.sh

bash-4.2$ bash -x /produits/nagiosXI/nagios/etc/script/cron.sh
+ echo 'executing the script'
executing the script
+ echo Hi

bash-4.2$ id
uid=54966(mlcadm) gid=501(monlc) groups=501(monlc),48(apache),320(outils),504(nagcmd),69172(mysql),69332(postgresql)
bash-4.2$

bash-4.2$ id nagios
uid=18961(nagios) gid=501(monlc) groups=501(monlc),504(nagcmd),53702(nagios)

Note : - Event handlers are not working since 2 month , Earlier it worked.

[tgriep]

The testing that you did looks like the script ran without any errors as the mlcadm user.
It could be a configuration setting in Nagios XI and you can either open up a ticket by emailing to [email protected] with a link to this post so we can continue on there.
If you go that route, email in your System Profile.
If you do not want to open up a ticket, can you PM me the System Profile?

To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and email it in.