Command works on CLI but not in SNMPTT event handler

[swolf]

Your guess sounds pretty reasonable to me.

In the name of due diligence, I have to ask you a few more questions:

Is there anything interesting happening on your XI filesystem? Any soft links/disk mounts that might have strange permissions?

What linux distribution is your server running on?

Can you verify the permissions on your scripts?

Code: Select all

ls -l /usr/local/nagiosxi/scripts/nxti.php
ls -l /usr/local/nagios/libexec/submit_passive_service.sh
groups snmptt

When you send a trap, are you seeing any errors in your other logs? I'd at least check /var/log/messages and possibly also your apache error log (path varies by distro, but on CentOS it's /var/log/httpd/error_log).

Lastly, and this is optional depending on your team's security restrictions/personal security preferences, but would you be able to try giving snmptt a shell and running the SNMPTT EXEC commands as them?

Code: Select all

sed -Ei 's#snmptt(.*)/sbin/nologin#snmptt\1/bin/bash#' /etc/passwd
su snmptt

and when finished:

Code: Select all

sed -Ei 's#snmptt(.*)/bin/bash#snmptt\1/sbin/nologin#' /etc/passwd

Hopefully we'll either see the issue or be able to re-create a bug with this information

[gavnor]

Debian GNU/Linux 9 (stretch) running as a VM on a Xen Hypervisor. The file system is ext4 on an LVM.

Code: Select all

-rwxr-xr-x 1 nagios nagios 1325 Apr 25 15:47 /usr/local/nagiosxi/scripts/nxti.php
-rwxr-xr-x 1 www-data nagcmd 1185 May  7 12:45 /usr/local/nagios/libexec/submit_passive_service.sh
snmptt : snmptt nagios nagcmd

These are the only logs from /var/log/messages that I'm unsure of and occurred around the time of the trap.

Code: Select all

May  7 14:30:20 nagiosxi ndo2db: Trimming timedevents.
May  7 14:30:20 nagiosxi ndo2db: Trimming systemcommands.
May  7 14:30:20 nagiosxi ndo2db: Trimming servicechecks.
May  7 14:30:20 nagiosxi ndo2db: Trimming hostchecks.
May  7 14:30:20 nagiosxi ndo2db: Trimming eventhandlers.

Nothing recorded in /var/log/apache2/error.log at the time of the trap.

Running as snmptt and executing the command via CLI the passive check was received and the service updated.

[swolf]

Those messages look normal to me. I'm setting up a debian system to see if I can re-create your issue. I'll update this thread again soon, hopefully with a solution.

EDIT: I am able to reproduce the issue on my system

[swolf]

Thought I'd give a proper update here:

There were a couple of different issues that I found.

First, I found that the /usr/local/nagiosxi/scripts/nxti.php script has a bug. This is fixed in the attached script, and will at least allow the script to work from the CLI. Note that you'll need to change the file extension back to .php

nxti.txt

Second, there are some strange permissions set by default in debian, which prevent the apache user from sending an SNMP trap. These commands fixed it for me:

Code: Select all

chmod g+w /var/lib/snmp
usermod -a -G Debian-snmp www-data

My hope is that these two changes will at least get your service status changes working. I also found a third issue, which may be a bug in snmptt, and which is still preventing the received traps table from working properly. If you don't care about the details, feel free to skip to the very bottom of this post:
---
I manually changed the EXEC command to the following:

Code: Select all

php /usr/local/nagiosxi/scripts/nxti.php --event_name="$N"  --event_oid="$i" --numeric_oid="$o" --symbolic_oid="$O" --community="$C" --trap_hostname="$R" --trap_ip="$aR" --agent_hostname="$A" --agent_ip="$aA" --category="$c" --severity="$s" --uptime="$T" --datetime="$x $X" --unixtime="$@" --bindings="$+*" >> /usr/local/nagiosxi/var/NXTI_Write_Test

Running snmptt in debug mode, I got the following command:

Code: Select all

EXEC line(s):
EXEC command:php /usr/local/nagiosxi/scripts/nxti.php --event_name="NXTI_Event_1"  --event_oid=".1.3.6.1.4.1.8072.2.3.0.1" --numeric_oid=".1.3.6.1.4.1.8072.2.3.0.1" --symbolic_oid="netSnmpExampleHeartbeatNotification" --community="" --trap_hostname="localhost" --trap_ip="127.0.0.1" --agent_hostname="localhost" --agent_ip="127.0.0.1" --category="NXTI Test Event" --severity="Normal" --uptime="0:19:24:20.02" --datetime="2019-05-08 10:33:05" --unixtime="1557329585" --bindings="netSnmpExampleHeartbeatRate:123456" >> /usr/local/nagiosxi/var/NXTI_Write_Test
snmptt
Warning: require_once(/usr/local/nagiosxi/html/config.inc.php): failed to open stream: Permission denied in /usr/local/nagiosxi/scripts/nxti.php on line 6

Fatal error: require_once(): Failed opening required '/usr/local/nagiosxi/scripts/../html/config.inc.php' (include_path='.:/usr/share/php') in /usr/local/nagiosxi/scripts/nxti.php on line 6

But after giving snmptt a shell, I found that the command worked as expected:

Code: Select all

php /usr/local/nagiosxi/scripts/nxti.php --event_name="$N"  --event_oid="$i" --numeric_oid="$o" --symbolic_oid="$O" --community="$C" --trap_hostname="$R" --trap_ip="$aR" --agent_hostname="$A" --agent_ip="$aA" --category="$c" --severity="$s" --uptime="$T" --datetime="$x $X" --unixtime="$@" --bindings="$+*"
debug: required files and connected to db
/* other debugging output */

---
Anyways, let me know how those first two changes work out for you, and I'll continue to work at this for a bit, and see if I can get in contact with the original author of snmptt

[gavnor]

Thank you very much for the update.

I updated the nxti.php file as directed.

I then ran the following:

Code: Select all

chmod g+w /var/lib/snmp
usermod -a -G Debian-snmp www-data

And also gave snmptt a shell:

Code: Select all

usermod -s /bin/bash snmptt

After analyzing the changes to the php command, I appended

Code: Select all

>> /usr/local/nagiosxi/var/NXTI_Write_Test

but nothing was written.

After making these changes, the trap event still doesn't work correctly. I can now run the php command from the CLI and produce a result in the received traps log but that command doesn't trigger a service status update. I can still run the static submit script from the command line and produce a status update but neither of those commands trigger by the trap receiver.

Please let me know if this is where we are in the troubleshooting or if I'm missing another step somewhere.

[swolf]

Please let me know if this is where we are in the troubleshooting

It doesn't necessarily look inconsistent. Here's where I am:

1. the PHP script (nxti.php) now works when run from the CLI, whether as root or as snmptt

As far as I can tell we both have that working

2. The received traps table does not update when a trap is sent

It seems we both have this issue as well

3. The snmp trap should still "fire".

i.e., if you do something like

Code: Select all

tail -f /var/log/snmptt/snmptt.log

You should see a new log entry every time you click the 'send test trap' button or run the snmptrap command from your command line.

If #3 matches then our systems are at least the same. If #3 doesn't work, run this and let me know the output:

Code: Select all

su www-data
snmptrap -v 2c -c public localhost '' NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification netSnmpExampleHeartbeatRate i 123456

[gavnor]

Thank you for clarifying. I believe our results are aligned.

[swolf]

Okay, as a temporary fix, you can make the following changes:

Change the permissions of config.inc.php to be world-readable

Code: Select all

chmod 644 /usr/local/nagiosxi/html/config.inc.php

This creates a minor security vulnerability on your system, but as long as you're only using your server to run Nagios XI, you should be okay.

You'll also want to modify /usr/local/nagiosxi/scripts/reset_config_perms.sh, changing line 76 from

Code: Select all

/bin/chmod 640 /usr/local/nagiosxi/html/config.inc.php

to

Code: Select all

/bin/chmod 644 /usr/local/nagiosxi/html/config.inc.php

in order to avoid having to make this change again later.

We will try to create a more secure solution for this issue, but for now this should allow you to use the application as expected.
Let me know if you have any issues with this or if you have any other questions.

[gavnor]

I've made the recommended changes and I now see traps logging in the received traps log however services aren't updating.

Passive Service Setup:
Host Name: $aR (IP)
Service Description: LostPowerEvent
Severity: Parse Severity Level ($s)
Service Output: SNMP Trap Received at $@ with variables $+*

Service Definition:

Code: Select all

define service {
    host_name                <ip>
    service_description      LostPowerEvent
    use                      xiwizard_passive_service
    is_volatile              1
    max_check_attempts       1
    check_interval           1
    retry_interval           1
    check_period             xi_timeperiod_24x7
    notification_interval    60
    notification_period      xi_timeperiod_24x7
    contacts                 nagiosadmin
    stalking_options         n
    _xiwizard                passivecheck
    register                 1
}

The received traps log shows the events received by my host IP and registered as Critical Severity but my Service Status is still "No check results for service yet..."

Is there something I'm missing or are there more permissions that need adjusted?

[swolf]

My apologies, I missed one thing. We'll also need to add read/write permissions to nagios.cmd to submit passive check results.

Code: Select all

chmod 666 /usr/local/nagios/var/rw/nagios.cmd

I imported your service definition and trap definition from page 1 and this worked on my test machine. If there is somehow still an issue, I'd expect the result to at least show up in Admin -> Monitoring Config -> Unconfigured Objects.