Project mass process_service_check_result

[WillemDH]

Hello,

First of all a happy New Year to all of you...

To start 2014 with some new ideas......
At the moment each of our Windows servers has two passive services, one named EVT_Application and one named EVT_System that receives passive events from the application or system eventlog.
We would really like to build a bash script that executes a process_service_check_result with OK result for each passive service staring with EVT_ in warning state. At the moment I have a quick action through the actions component pointing to a bash script that will do this for one service.

Code: Select all

HOSTNAME=$1
SERVICENAME=$2
now=`date +%s`
commandfile='/usr/local/nagios/var/rw/nagios.cmd'

/usr/bin/printf "[%lu] PROCESS_SERVICE_CHECK_RESULT;$HOSTNAME;$SERVICENAME;0;OK - Manual Reset\n" $now > $commandfile

But as we are continously adding more hosts, we tend to receive more and more passive service checks. As we do not want to make freshness checks or maximum age passive results, for different reasons, resetting the state of our EVT_* services would need to be done by a manual action.

As my bash knowledge is limited and I have no idea where to start, I would like to ask some questions:

- Is my above idea realistic? Or is it too complex?
- is it possible to somehow list all the services found in "Open Service Problems"? As I do not want to reset passive services that are not in the "Open service Problems" view, I would need to create an array of all services starting with EVT_* in warning state in "Open Service Problems" with their respective host and than loop through the array and send each item to the commandfile.

Any help / advice is very welcome.

Willem

[abrist]

As long as my assumption is correct, and you want to consider warnings as OK, you could do this a few ways:
1) By setting the check thresholds to eliminate warnings.
2) With event handlers: http://assets.nagios.com/downloads/nagi ... ndlers.pdf
3) Or, more easily with the negate plugin wrapper: https://www.nagios-plugins.org/doc/man/negate.html

[WillemDH]

Hmm, sorry but you assumption is not correct. I just want to create a manual action to reset all services whose name is starting with EVT_* and are in "Open service problems" to an OK state. I do not want this to be an automatic action (eg event handlers, negate check or sth like that)

So to repost my question:

I would need to create a bash script that lists all the services staring with EVT_* found in "Open Service Problems" with their respective host and than loop through the array and send each item to the commandfile like this:

/usr/bin/printf "[%lu] PROCESS_SERVICE_CHECK_RESULT;$HOSTNAME;$SERVICENAME;0;OK - Manual Reset\n" $now > $commandfile

[abrist]

Hmm, sorry but you assumption is not correct.

Well, you know what they say about assumptions. . .

The best way to do this may involve wgeting the backend XML, parsing it in bash, and then iterating through the parsed list. The following backend url is most likely the one you want to use:

Code: Select all

http://<xi ip>/nagiosxi/backend/?cmd=getservicestatus&brevity=3&current_state=ne:0&name=lk:EVT_

You will want to also include a username (someone with rights to the EVT_ boxes) and their ticketid in the get request.
Write your bash script to wget/curl the xml, parse it and iterate through the "host_name" and "name" (servicename) keys, running the process command on each pair. If you need further assistance - let me know. Cheers.

[WillemDH]

Andy,

I think we are getting in the right direction. I updated the service status url and added a user and ticket. I get only two results, but I should have gotten 30 results. Only 2 EVT_Appliaction services are shown. I temporary used nagiosadmin as user, so permissions should be ok.
The url looks like this atm:
http://servername/nagiosxi/backend/?cmd ... izjdajdfij

When I changed it and use the full seervice name like this:
http://servername/nagiosxi/backend/?cmd ... izjdajdfij

I get all the EVT_Application services.
Seems the name=lk:EVT_& somehow does not suffice to get EVT_Application and EVT_System

Well this doesn't have to be an issue to continue working on the script, but I thought I'd let you know. I think I'll make two scripts anyway, one to reset all EVT_Application services and one to reset all EVT_System services.

Grtz

[WillemDH]

Andy,

Fantastic, I managed to create the scripts I need This will save us a lot of time... Here is the code if anyone would like to dos something similar:

Code: Select all

#!/bin/bash
# This script will automatically reset all EVT_System services to an OK state
# 20140110 => Willem D'Haese => Script creation

now=`date +%s`
commandfile='/usr/local/nagios/var/rw/nagios.cmd'
dir=/usr/local/nagios/libexec/digipolis

rdom () { local IFS=\> ; read -d \< E C ;}
# rdom () { local IFS=\< ; read -d \> C E ;}

wget -O $dir/servicestatus.xml  "http://srvnagios01/nagiosxi/backend/?cmd=getservicestatus&brevity=3&current_state=ne:0&name=lk:EVT_System&username=nagiosadmin&ticket=blabla"

while rdom; do
	if [[ $E = host_name ]]; then
		/usr/bin/printf "[%lu] PROCESS_SERVICE_CHECK_RESULT;$C;EVT_System;0;OK - Manual Reset\n" $now > $commandfile
	fi
done < $dir/servicestatus.xml

Now I would just need some way to let people execute this script from the XI gui. I could use a dummy host and attach a quick action that points to the sccript to it, but I wonder if there is some better way, like for example a dashlet I can place on admins homepage or something. Any suggestions?

[WillemDH]

Seems I still have some issue. The script won't execute when initiated from quick actions. it works fine from cli. I added some kind of logging

Code: Select all

nowymd=$(date '+%Y-%m-%d -- %H:%M')
echo "$nowymd : /usr/local/nagios/libexec/dig_ext_cmd_svc_submit_ok_all_evt_application.sh execution started!" >> /var/log/dig-reset-evt-app.log

When executed fromcli, the log gets an entry the script starts, but when I click on the action component link, nothing happens. I checked permissions and they seem fine. (755) The permissions on the digipolis directory also seems fine. The servicestatusxml is not created however, but he never gets there. Is there something I'm missing?

[abrist]

Can you try running the script from the cli a few times as the users "nagios" and "apache"?

[WillemDH]

Andy,

I found the problem. It was indeed still a permissions problem on the servicestatux.xml generated. I'mm too much used working with Windows ntfs security and was thinking if I put permissions on the folder, the permissions would be inherited by the containing files.

So now I'm able to execute it from a dummy host I created. Is there some way to put a link to this script in some sort of a dashlet?

Grtz

[abrist]

I found the problem.

Great!

Is there some way to put a link to this script in some sort of a dashlet?

Possibly, though we do not have any documentation for dashlet creation. Using the "actions" component may be your best option.