Alert not executing script

[randall.schievelbein]

I have a problem where when I "activate" an alert which executes a script it does not execute the script when the alert is triggered. The alert shows that it was run in the alert history and that the log threshold was met for the script to be executed but the script was never executed.

When I click the "Run the alert now" the script executes and runs as expected but when the script is supposed to be automatically executed by the alert it doesn't seem to ever be executed.

[cdienger]

What version of XI and OS is this? Can you provide a screenshot of the alert settings?

[randall.schievelbein]

Did you mean what version of LS?

We are on Nagios Log Server version 2.1.6 running on a server with CentOS Linux release 7.7.1908 (Core)

Here are the settings for the alert

Nagios Alert Screenshot.PNG

And here is the alert history showing the alert was triggered as expected. The script wasn't executed on any of these.

Nagios Alert History.PNG

[ssax]

What is the output of this command?

Code: Select all

ls -lh /usr/local/nagioslogserver/scripts/CAP_Network.sh

Make sure it's:

Code: Select all

chmod 0554 /usr/local/nagioslogserver/scripts/CAP_Network.sh
chown nagios.nagios /usr/local/nagioslogserver/scripts/CAP_Network.sh

Please attach the script as well:

Code: Select all

/usr/local/nagioslogserver/scripts/CAP_Network.sh

Does it work if you manually call it as the nagios user?

Code: Select all

su - nagios
/usr/local/nagioslogserver/scripts/CAP_Network.sh "Nagios - High Log Count"

[randall.schievelbein]

So this script is pretty simple, most of the real work is done by a python script that is called by this CAP_Network.sh script. Below are the file permissions for both files.

Nagios Alert chmod.png

When I run this command the script executes as expected:
sudo su -c "/usr/local/nagioslogserver/scripts/CAP_Network.sh \"Nagios - High Log Count\"" -s /bin/sh nagios

I'm unable to the run the su command as you provided due to lacking a password that is prompted. I assume the above command essentially does the same thing but I could be wrong.

The script CAP_Network.sh is attached

[cdienger]

The nagios account should not have a password and this is likely the reason for failure. Edit /etc/sudoers and make sure this is at the bottom:

Code: Select all

User_Alias NAGIOSLOGSERVER=nagios
User_Alias NAGIOSLOGSERVERWEB=apache
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/bin/systemctl start httpd
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/bin/systemctl stop httpd
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/bin/systemctl restart httpd
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/bin/systemctl reload httpd
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/bin/systemctl status httpd
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/change_timezone.sh
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/reconfigure_ncpa.sh
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_listener start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_listener stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_listener restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_listener reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_listener status
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_passive start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_passive stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_passive restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_passive reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_passive status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/bin/systemctl start httpd
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/bin/systemctl stop httpd
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/bin/systemctl restart httpd
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/bin/systemctl reload httpd
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/bin/systemctl status httpd
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/get_logstash_ports.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/profile.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/reconfigure_ncpa.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_listener start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_listener stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_listener restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_listener reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_listener status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_passive start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_passive stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_passive restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_passive reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_passive status

Run the below to check and set expiration settings on the account if the above doesn't help:

Code: Select all

chage -u nagios
chage -I -1 -m 0 -M 99999 -E -1 nagios

And also look in /var/log/crond if it still isn't working.

[randall.schievelbein]

I checked the /etc/sudoers file and it already had the lines at the bottom that you mentioned.

This got me curious so I dug around in our /etc/passwd file and noticed that the nagios gid didn't match the uid. I'm not super knowledgeable about Linux, I just know enough to do basic navigation around the filesystem so I'm not sure if this would be an issue?

Nagios passwd.PNG

When I run the below command in the scripts directory I get the below result for the CAP_Network.sh file.

Code: Select all

ls -ln

Nagios ls ln result.PNG

[ssax]

Try this for CAP_Network.sh:

Code: Select all

#!/bin/bash

phpbin=$(which php)
sendmailbin=$(which sendmail)
pythonbin=$(which python)

alert_name="$1"
alert_name_encoded=$($phpbin -r "echo urlencode(\"${alert_name}\");")

echo "Subject: Nagios CAP Network Alert Triggered" | $sendmailbin -F "Nagios Log Server" randall.schievelbein@cmc.com

$pythonbin /usr/local/nagioslogserver/scripts/parse_alert.py "CAP - Network" "${alert_name_encoded}" > /dev/null 2>&1

echo "Subject: Nagios CAP Network Alert Finished Executing" | $sendmailbin -F "Nagios Log Server" randall.schievelbein@cmc.com

I would also test it like this:

Code: Select all

#!/bin/bash
echo "MADE IT HERE" >> /usr/local/nagioslogserver/var/TESTING.txt
phpbin=$(which php)
sendmailbin=$(which sendmail)
pythonbin=$(which python)

alert_name="$1"
alert_name_encoded=$($phpbin -r "echo urlencode(\"${alert_name}\");")

echo "Subject: Nagios CAP Network Alert Triggered" | $sendmailbin -F "Nagios Log Server" randall.schievelbein@cmc.com

$pythonbin /usr/local/nagioslogserver/scripts/parse_alert.py "CAP - Network" "${alert_name_encoded}" > /dev/null 2>&1

echo "Subject: Nagios CAP Network Alert Finished Executing" | $sendmailbin -F "Nagios Log Server" randall.schievelbein@cmc.com

Then force the alert to kick off and then check this file to see if it's created (if it is we know the script is at least getting called):

Code: Select all

/usr/local/nagioslogserver/var/TESTING.txt

Additionally, include the output of this command:

Code: Select all

grep nag /etc/group

[randall.schievelbein]

Sorry for the super late reply. I was out sick the last 2 weeks and was just now able to get back to this.

This issue has been resolved. The main issue turned out to be a silent failure in my main python script that was creating the illusion that it wasn't even being executed. Nothing appears to have been wrong on the Nagios side of things.

I really appreciate the help in debugging this issue. The tip about testing the script by outputting to a file was really helpful in finding the failure in my main script.


Thanks,
Randall Schievelbein

[ssax]

That's okay, glad you're feeling better!

Thanks for the update, I'm glad it's working for you now, let us know when we're okay to lock this up and mark it as resolved.