python script as event handler won't write to file

[aramisbackwards]

Hi, I'm having a very weird issue that I've been trying to debug for the past two days and I simply can't figure out for the life of me what' s going on.

Debug level is set to -1, Verbosity is set to 2.

The python code has been stripped down to simply just printing to a file. The script is chmodded to 777 and the file should be saved in the libexec folder at /usr/local/nagios/libexec.
If I run the script standalone it works perfectly.

Code: Select all

#!/usr/bin/python
import os
import sys
import datetime

def main():
    print sys.argv[1:]
    with open('new_file.txt', 'a+') as newfile:
        newfile.write(str(sys.argv[1:]))
        newfile.write(str(datetime.datetime.now()))


if __name__ == '__main__':
    main()       

My object definition

Code: Select all

define host {
	check_period			24x7
	use			                generic-switch
	check_interval			2
	notification_interval         5
	notification_period		24x7
	retry_interval			1
	check_command		check-host-alive
	hostgroups			nagios-routers
	contact_groups			admins
	alias			                test-router
	parents			        network-monitor
	host_name			42947219231x5
	address			        *.*.*.*
	event_handler			test_python
	max_check_attempts		5
	_parent_id			42947219020x5
	notification_options		d,r,u,
}

My command config

Code: Select all

define command {
	command_name 	test_python
	command_line	/usr/bin/python $USER1$/go.py "$HOSTNAME$" "$HOSTADDRESS$" "$HOSTSTATE$" "$HOSTATTEMPT$" "$_HOSTPARENT_ID$" "$HOSTSTATETYPE$"
}

And here is the log whenever the event handler fires (from journalctl -xe):

Code: Select all

Feb 01 10:49:12 nagios-server.novalocal nagios[19862]: wproc: Registry request: name=Core Worker 19865;pid=19865
Feb 01 10:49:12 nagios-server.novalocal nagios[19862]: wproc: Registry request: name=Core Worker 19866;pid=19866
Feb 01 10:49:12 nagios-server.novalocal nagios[19862]: wproc: Registry request: name=Core Worker 19867;pid=19867
Feb 01 10:49:12 nagios-server.novalocal nagios[19862]: wproc: Registry request: name=Core Worker 19864;pid=19864
Feb 01 10:49:12 nagios-server.novalocal nagios[19862]: Successfully launched command file worker with pid 19868
Feb 01 10:50:01 nagios-server.novalocal sshd[19890]: Connection closed by 127.0.0.1 [preauth]
Feb 01 10:50:13 nagios-server.novalocal nagios[19862]: HOST ALERT: 42947219231x5;DOWN;SOFT;1;CRITICAL - Host Unreachable (*.*.*.*)
Feb 01 10:50:13 nagios-server.novalocal nagios[19862]: HOST EVENT HANDLER: 42947219231x5;DOWN;SOFT;1;test_python
Feb 01 10:51:01 nagios-server.novalocal sshd[19933]: Connection closed by 127.0.0.1 [preauth]
Feb 01 10:51:16 nagios-server.novalocal nagios[19862]: HOST ALERT: 42947219231x5;DOWN;SOFT;2;CRITICAL - Host Unreachable (*.*.*.*)
Feb 01 10:51:16 nagios-server.novalocal nagios[19862]: HOST EVENT HANDLER: 42947219231x5;DOWN;SOFT;2;test_python
Feb 01 10:51:18 nagios-server.novalocal sudo[19946]:   centos : TTY=pts/0 ; PWD=/usr/local/nagios/libexec ; USER=root ; COMMAND=/bin/vim :q
Feb 01 10:51:29 nagios-server.novalocal sshd[19957]: Connection closed by 127.0.0.1 [preauth]
Feb 01 10:51:42 nagios-server.novalocal sudo[19978]:   centos : TTY=pts/0 ; PWD=/usr/local/nagios/libexec ; USER=root ; COMMAND=/bin/vim migrate_router.py
Feb 01 10:52:01 nagios-server.novalocal sshd[19988]: Connection closed by 127.0.0.1 [preauth]
Feb 01 10:52:19 nagios-server.novalocal nagios[19862]: HOST ALERT: 42947219231x5;DOWN;SOFT;3;CRITICAL - Host Unreachable (*.*.*.*)
Feb 01 10:52:19 nagios-server.novalocal nagios[19862]: HOST EVENT HANDLER: 42947219231x53;DOWN;SOFT;3;test_python

Nothing remotely relevant in nagios.debug.

I cannot figure out why the file doesn't write but there are no errors reported in the log. Does it run the python script from a different shell?

EDIT: I just tried the following:

1. replaced the script with a bash script that calls the python script like (python my_script.py)
2. In the error log it says "stderr line 01: python: can't open file 'my_script.py': [Errno 2] No such file or directory" despite them being in the same directory.

What exactly is happening with event handlers? Should I avoid using them and just use check-commands instead?

[aramisbackwards]

Okay, strange, I found out by searching for my file that nagios event handler was writing to a temp folder, and missed this line in the config:

Code: Select all

# TEMP PATH
# This is path where Nagios can create temp files for service and
# host check results, etc.

temp_path=/tmp

Process of elimination will eventually have to work I guess, but there's still an issue: Changing the path still has nagios serving event_handler output to the /tmp folder on CentOS7. Not sure why that happens and how to configure it.

[rkennedy]

Does it work properly if you define a direct path instead of relying on the temp dir / a relative path?

[aramisbackwards]

Actually even if I change the folder it still writes to temp. Right now my config is

Code: Select all

temp_path=/home/centos/testing/logs

but my intuition tells me that perhaps my nagios might not have permissions to cd into /home/centos and so it defaults to temp...maybe?

I'll try the relative path

[rkennedy]

That's possible, could always test with su - nagios and then attempt to write there. If you run the python script from the CLI does it work still, and where does it write the file?

[aramisbackwards]

Doesn't look like permissions are the issue. I tried this with a fresh install of 4.2.4, and have nagios configured as having root privileges. It still writes to temp no matter what I give it in the directive. Other thing I noticed is that it looks like nagios runs the event handlers from a subshell, since I checked the environment variables in the event handler script and they are completely different. I'm not sure if this is because the subshell doesn't have proper permissions or something but I'm not too familiar with subshells (provided that's the case) so I'll have to do some research.

The script output when run manually from the CLI goes to the same libexec folder or anywhere that I specify, as expected.

As a work around I'm going to try to just move every the output manually from tmp to somewhere else (it'll let me know if I have the proper permissions as well).

[rkennedy]

I'm not sure what's going on in your environment, but I just tested your python script at two layers and it worked fine writing to an absolute path. The test check command, and also as an event handler.

Code: Select all

[root@centos7 test]# cat new_file_remote.txt
['test3']2017-02-03 10:26:44.562488['web']2017-02-03 10:29:48.252323['event']2017-02-03 10:31:36.155222

permissions:

Code: Select all

[root@centos7 test]# ls -la
total 16
drwxrwxrwx.  2 root root   68 Feb  3 10:26 .
drwxrwxrwt. 18 root root 4096 Feb  3 10:31 ..
-rwxrwxrwx.  1 root root  138 Feb  3 10:32 new_file_remote.txt
-rwxrwxrwx.  1 root root   62 Feb  3 10:24 new_file.txt
-rwxrwxrwx.  1 root root  292 Feb  3 10:26 test.py

test.py

Code: Select all

[root@centos7 test]# cat test.py
#!/usr/bin/python
import os
import sys
import datetime

def main():
    print sys.argv[1:]
    with open('/tmp/test/new_file_remote.txt', 'a+') as newfile:
        newfile.write(str(sys.argv[1:]))
        newfile.write(str(datetime.datetime.now()))


if __name__ == '__main__':
    main()

I then went as far as changing it to the libexec directory, and it worked fine -

Code: Select all

    with open('/usr/local/nagios/libexec/test.txt', 'a+') as newfile:

Code: Select all

[root@centos7 test]# cat /usr/local/nagios/libexec/test.txt
['cli']2017-02-03 10:34:37.331644['event']2017-02-03 10:34:55.759223

I would verify all of your permissions are setup properly for the nagios user.

[aramisbackwards]

what is the path you specified in your nagios.cfg file? specifying a file path in the script isn't the issue, it is specifying the output of the directive in the config file

[rkennedy]

I thought you mentioned that the output in the python script wasn't respecting absolute paths as well, which is why I setup my testing.

Code: Select all

temp_path=/tmp

[aramisbackwards]

sorry, if that's the case i misspoke.

What I basically mean is that if I change

Code: Select all

temp_path =/tmp

to anything else (for example)

Code: Select all

 temp_path=/a/different/path

and then just execute an event handler and write to some file, regardless of what I change it to the folder will always be /tmp. It's really just a matter of seeing if that functionality is working or not (for just me or in general im not sure), not that I couldn't find a workaround by just specifying it in my python code