Nagios external commands getting processed intermittently

[sidd]

I have a script that runs every 10 mins that updates the Host Comments field in Nagios. There are about 600 hosts and I am seeing some issues. The script deletes the previous comments and adds new ones. The issue I am seeing is that the comments are getting deleted intermittently.

I am using Nagios 3.2.2

Here is the script -

Code: Select all

FILE = open("/var/log/nagios/rw/nagios.cmd","w")
  NOW = int(time.time())
  HOSTS = ["host1", "host2",etc]

  for hosts in HOSTS:
                DEL_COMMENT = "["+str(NOW)+"] DEL_ALL_HOST_COMMENTS;"+hosts+"\n"
                FILE.write(DEL_COMMENT)
  FILE.close()

The Nagios log shows that the comment was read -

Code: Select all

 [1402964971] EXTERNAL COMMAND: DEL_ALL_HOST_COMMENTS;host1
  [1402964971] EXTERNAL COMMAND: DEL_ALL_HOST_COMMENTS;host2
  [1402964971] EXTERNAL COMMAND: DEL_ALL_HOST_COMMENTS;host3

[sreinhardt]

How long have you been running this script, and how long has it been having issues? Have you made any changes, updates, etc since the issues started?

[sidd]

I just started yesterday. It seems the problem is not with the scripts but the way the hosts are setup in Nagios or something. When I issue an external command just to a single host that is having issues with comments, to delete all comments, it doesn't work. Same thing with the WebUI. The "Delete all comments" for certain hosts doesn't work. I can delete comments using comment ids.

I ended up getting the comment ids using live status and writing a script around that.

I don't have the rights to change the debug mode for Nagios otherwise I would have posted 128 debug level.

[eloyd]

"And nothing's changed."

I know you didn't say that, but that's something I hear all the time.

It's time to start figuring out what changed on your system. Does the package updater run automatically, possibly having changed one of the packages on your machine? Did your code repository get updated and now your shell script is broken? Lots of things along those lines. Also, make sure you're not running out of disk space and/or quota on the filesystem.

These are long shots, but if it was working and now it's not working, then something changed. Figure out what that was.

[slansing]

Hey sidd,

Did you come up with anything? Find out who may have altered system level options?