shutdown 3 servers when UPS low

[jriker1]

I have 3 servers connected to a single UPS. I am monitoring it's charge left with check_ups. When it's down to 30% it goes critical. I could send nrpe command but it's running the service against the PowerWare UPS not the computers. How would I properly script an eventhandler or something to shutdown 3 computers when the power gets to 30%? Note it's a mix of 2 windows and a Linux box. Here is my current service:

# Battery Charge level left
define service{
use generic-service
host_name PowerWare 9125
service_description Battery Charge Left
check_command check_ups!public!charge!70!30
normal_check_interval 15
retry_check_interval 3
}

Thanks.

JR

[rkennedy]

Assuming you already have NRPE commands / scripts defined for the command 'shutdown' setup to trigger, you would go this route.

Create a bash script, that first off detects what state the service is changing in to. Then, just have it execute as NRPE normally would as commands in your bash script. Just because it's on a different service, doesn't mean the Nagios machine can't execute /usr/local/nagios/libexec/check_nrpe -H x.x.x.x -c shutdown from the CLI still.

Once that's created, assign it as an event_handler to the service.

[jriker1]

Assuming you already have NRPE commands / scripts defined for the command 'shutdown' setup to trigger, you would go this route.

Create a bash script, that first off detects what state the service is changing in to. Then, just have it execute as NRPE normally would as commands in your bash script. Just because it's on a different service, doesn't mean the Nagios machine can't execute /usr/local/nagios/libexec/check_nrpe -H x.x.x.x -c shutdown from the CLI still.

Once that's created, assign it as an event_handler to the service.

Thanks for the reply. I wrote up the shutdown script, modified from another one I had, however partly no to your inquiry. The systems I need to shutdown here are virtual machines. Treating them like any other machine, however they do not have NSCLIENT++ installed so if there is another suggestion open to it. Installing NSCLIENT++ is not even vaguely a big deal but always interested in options.

Thanks.

JR

For reference here is the simple event handler I created:

Code: Select all

!/bin/sh
#
# Event handler script for restarting the server on the local machine
#
# use variables for arguments
SERVICESTATE=$1
SERVICESTATETYPE=$2
SERVICEATTEMPT=$3

# we don't want to restart if current status is OK
if [ "$SERVICESTATE" != "OK" ] ; then
  # proceed only if we're in soft transition state
  if [ "$SERVICESTATETYPE" == "SOFT" ] ; then
    # proceed only if this is 3rd attempt, restart
    if [ "$SERVICESTATEATTEMPT" == "3" ] ; then
      /usr/local/nagios/libexec/check_nrpe -H 192.168.0.x -c shuterdown
      /usr/local/nagios/libexec/check_nrpe -H 192.168.0.x -c shuterdown
      /usr/local/nagios/libexec/check_nrpe -H 192.168.0.x -c shuterdown
    fi
  fi
fi
exit 0

Assuming this won't go straight to a hard state.

Also this may be a challenge. If Nagios is in VM, how do I shut down both the Nagios VM and it's parent server? Maybe tell the parent server to shutdown, have it's shutdown script have a 2 minute pause in it, and then trigger the Nagios instance to shutdown?

[rkennedy]

Assuming you already have NRPE commands / scripts defined for the command 'shutdown' setup to trigger, you would go this route.

Create a bash script, that first off detects what state the service is changing in to. Then, just have it execute as NRPE normally would as commands in your bash script. Just because it's on a different service, doesn't mean the Nagios machine can't execute /usr/local/nagios/libexec/check_nrpe -H x.x.x.x -c shutdown from the CLI still.

Once that's created, assign it as an event_handler to the service.

Thanks for the reply. I wrote up the shutdown script, modified from another one I had, however partly no to your inquiry. The systems I need to shutdown here are virtual machines. Treating them like any other machine, however they do not have NSCLIENT++ installed so if there is another suggestion open to it. Installing NSCLIENT++ is not even vaguely a big deal but always interested in options.

Thanks.

JR

For reference here is the simple event handler I created:

Code: Select all

!/bin/sh
#
# Event handler script for restarting the server on the local machine
#
# use variables for arguments
SERVICESTATE=$1
SERVICESTATETYPE=$2
SERVICEATTEMPT=$3

# we don't want to restart if current status is OK
if [ "$SERVICESTATE" != "OK" ] ; then
  # proceed only if we're in soft transition state
  if [ "$SERVICESTATETYPE" == "SOFT" ] ; then
    # proceed only if this is 3rd attempt, restart
    if [ "$SERVICESTATEATTEMPT" == "3" ] ; then
      /usr/local/nagios/libexec/check_nrpe -H 192.168.0.x -c reboot
      /usr/local/nagios/libexec/check_nrpe -H 192.168.0.x -c reboot
      /usr/local/nagios/libexec/check_nrpe -H 192.168.0.x -c reboot
    fi
  fi
fi
exit 0

Assuming this won't go straight to a hard state.

This should work, looks like the obstacle now is figuring out how to shut them down without NSClient++, and then we can change the /usr/local/nagios/libexec/check_nrpe -H 192.168.0.x -c reboot part. To help gather a bit more info:
- What hyper visor are these VM's running on?
- Are they on a domain?
- Any kind of ways to remotely reboot them from another windows computer that has NSClient++ installed?
-- shutdown -m might work (new to me - http://davidvielmetter.com/tricks/four- ... s-machine/)
-- https://technet.microsoft.com/en-us/sys ... sexec.aspx (can't recall if this has a shutdown option, but very useful in a windows environment)
-- Powershell 'restart-computer' command (http://davidvielmetter.com/tricks/four- ... s-machine/)

[jriker1]

Here's your answers:

- What hyper visor are these VM's running on? They are running on standard VMware Workstation 12 running on a Windows box.
- Are they on a domain? All are on a domain, except the Nagios instance is Linux.
- Any kind of ways to remotely reboot them from another windows computer that has NSClient++ installed? Maybe. Probably. Only negative is will there be any permission challenges.
-- shutdown -m might work (new to me - http://davidvielmetter.com/tricks/four- ... s-machine/)
-- https://technet.microsoft.com/en-us/sys ... sexec.aspx (can't recall if this has a shutdown option, but very useful in a windows environment) Won't touch the PSExec stuff. Thought it was the coolest stuff and then found out it needed some special admin privs to run you couldn't pass or do thru a NSClient call. Switched to the built in Windows shutdown and been all good ever since
-- Powershell 'restart-computer' command (http://davidvielmetter.com/tricks/four- ... s-machine/)

[rkennedy]

Ah. PSExec is pretty interesting, it's been a while since I've done windows administration though.

Looks like you're left with a few options -
1. Try to shut down on the Workstation level (found these links, https://www.vmware.com/support/develope ... ommand.pdf http://www.bloggerbaru.com/how-to-start ... ing-vmrun/)
2. shutdown -m
3. 'restart-computer' from powershell.

I haven't used any of the three, so it is going to take looking at your environment to see what would work for you. Based on the route you decide to go, you'll just want to call those commands from a Windows machine currently running NSClient++. (that way Nagios not being a windows machine, and not being on the domain become irrelevent)

I'm interested to see how this works out, so let us know if you run into any issues. The last resort, if the 3 above options won't work, is simply installing NSClient++ on the 3 client machines, and just using a simple batch script to run shutdown.

[jriker1]

Thanks for the tips I will report back shortly. I think I will go with the vmrun command as it can suspend the process and then resume when turned on again. Manually but can be done then without needing to "shutdown" the whole system. Plus the computer VM is running on has NSClient++ already. I will report back with my final solution all around. Only item I need to figure out now:

Suspend VM1 (Windows)
Suspend VM2 (Windows)
Uhhh, VM3 is Nagios
Physical server 1 is running all the VM's.

So thinking that I need to send a shutdown command to physical server 1 that has like a 2 minute delay in it. Then send the suspend command to Nagios VM. Then hopefully nagios will suspend and then after that the server will shutdown.

[tmcdonald]

Definitely keep us posted!