adding hosts outside of NagiosXI from script

[tgroten]

Greetings,

I have a plethora of hosts that need to be added/deleted dynamically to/from my NagiosXI instance, however, I cannot use the UI, Auto-Discovery or the Bulk Import Tool. Here is the situation: We have our servers in AWS EC2, managed by Rightscale for auto-scaling deployments. At a certain threshold in our RS array, it kicks off the server build templates/scripts and creates X number of instances, but also when a different threshold is reached, it will kick off server decommissioning scripts and the instances go away. As you can see, this makes monitoring a nightmare for consistency.

I would love to see a function that could be called from a shell/python/perl script that would add itself upon creation to NagiosXI, set the correct templates based on the AWS EC2 tag I have set for "role" and a 2nd script I could add to the decomm scripts that would remove it from monitoring all together. I've looked at the Bulk Import Tool and it has the capabilities of reading all of the parameters I need to add/organize hosts from a CSV, but I need a programmatic way of calling it at the command line, since I am not going to be awake at 3am to manually import 600+ new servers that just got deployed because our load increased, only to have them go away when the load eventually decreases and all of the "terminated" servers are throwing up unavailability alerts.

I imagine there is a way to call something like the nagiosql_delete_host.php script manually from the box itself, as it gets decommed, but I can't find anything out there to add the host itself, initially. All of the config generators out there are for Nagios Core and all of the configs I have looked at on the XI server have an admonishment to not change anything, because those files get dynamically created/overwritten periodically. Is there anything in your bag of tricks that can get me through this major headache? Bear in mind, I don't have a lot of hair left to be pulled out, as I have been doing to try and figure this out for weeks now.

Thanks,

-Todd Groten

[jsmurphy]

That set up certainly would be a nightmare without automation!

If you generate nagios host configs in /usr/local/nagios/etc/import/ and then run /usr/local/nagiosxi/scripts/reconfigure_nagios.sh it will import all the host config files into your XI set up. Alternatively you can also place them in /usr/local/nagios/etc/static/; placing them in this directory is a bit easier to manage with a script and less moving parts to go wrong but you won't be able to manage them with the XI configuration utility.

[tgroten]

Thanks for the idea! Though, I have a few questions pertaining to its use...

1) Would you recommend running it at timed intervals? Each server that gets auto scaled would create its own config file upon deployment and drops that file into the import folder on the XI server to be picked up by the cronjob. Would having multiple config files in the import folder be an issue?

2) What would be the best frequency to have this cronjob interval set at? I can imagine that running it regularly might be a performance hit to XI, but would it also make the XI system unavailable during its run?

3) Is it a better idea to run the config create and reconfig task at the server creation time? The only problem I might see with this method is if I have 500+ servers auto-scaled at once and them ALL trying to reconfigure at once. Correct?

Do those sound like valid concerns with automating deployment activities for our servers? What's the best practice for the reconfigure script with large scale deployment automation, in your opinion?

Thanks,

-Todd

[mguthrie]

Hi Todd,

We have had users set up automation like this, but it does require some customization and mods, and we're able to offer *some* support for this, but as you might guess, we place no guarantees on modified installs. With that said, here are some responses to your questions.

1) Would you recommend running it at timed intervals? Each server that gets auto scaled would create its own config file upon deployment and drops that file into the import folder on the XI server to be picked up by the cronjob. Would having multiple config files in the import folder be an issue?

I would make two suggestions here. Multiple config files are just fine, but keep them as simple as possible to prevent import issues. Do as much as you can with templating. There shouldn't be a lot of downtime with XI, it runs the import, writes them to file, verifies the config, and if verification succeeds, then nagios will be restarted. It should also be noted that if the config verification fails XI will roll back to the last known good configuration in order to maximize uptime for the monitoring engine.

2) What would be the best frequency to have this cronjob interval set at? I can imagine that running it regularly might be a performance hit to XI, but would it also make the XI system unavailable during its run?

The biggest thing you'll have to figure out, is the maximum amount of hosts you can import at once. PHP has built-in timeouts and memory limits specified in the /etc/php.ini file. If you exceed one of these the script will just timeout. I'd crank these numbers up for safety.

3) Is it a better idea to run the config create and reconfig task at the server creation time? The only problem I might see with this method is if I have 500+ servers auto-scaled at once and them ALL trying to reconfigure at once. Correct?

It can be done, just give PHP a lot of memory and processing time to complete it all. : )