THOUGHTS - Managing a distributed NAGIOS Setup

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
Locked
mitchsmith
Posts: 33
Joined: Mon Apr 04, 2011 7:26 am

THOUGHTS - Managing a distributed NAGIOS Setup

Post by mitchsmith »

Hi,

Just posting my thoughts really, and any input would be appreciated.

Here is my brain dump.....
I currently use nagios core to monitor multiple sites globally, and am in the process of distributing this using NSCA.

The idea is, currently I use nagios to monitor a very specific application server stack, and that one benefit of distributing this to local IT would allow (as well as the reduced load on the Central Nagios server), the local IT teams to extend my foundation NAGIOS configuration to monitor there other IT Systems.

I am considering using, source control to ensure that configurations are kept consistent, where by the Local NAGIOS server could perform a nightly SVN update to get any new cfg files, plugins. Then Reload nagios.

I have a basic .cfg file structure that allows me to distribute my core configuration, however I can see issue arrising if local IT staff edit the nagios.cfg to include their extended configuration (That I do not care about - I will provide a template that will ensure Obsess over services and hosts for their extension configs, is disabled)

My thoughts are:

If I created a dir within each site called Local, in which I could include a nagiosLocal.cfg which could look something like this;

Code: Select all

cfg_dir=/etc/nagios/local/commands
cfg_dir=/etc/nagios/local/services
cfg_dir=/etc/nagios/local/hosts
And Instruct the local IT teams to ensure that their configurations should be kept in this DIR only. As It would not be included in the SVN update process

Does anyone have any insight in to this... at the moment this is just my thoughts, I am open to suggestions on improving this idea.

Mitchell
mguthrie
Posts: 4380
Joined: Mon Jun 14, 2010 10:21 am

Re: THOUGHTS - Managing a distributed NAGIOS Setup

Post by mguthrie »

That strikes me as a pretty solid solution. Do you have any concerns about the local teams maintaining their configs properly and preventing Nagios from restarting correctly?

The monitoring structure you're describing is one that I'm familiar with. A classic passive distributed setup is one where local teams handle alerts, performance graphs, etc, and then the essential services get pushed up to the central server. I wouldn't call myself an expert on distributed monitoring by any means, but your plans seems fairly solid from what I can tell.
mitchsmith
Posts: 33
Joined: Mon Apr 04, 2011 7:26 am

Re: THOUGHTS - Managing a distributed NAGIOS Setup

Post by mitchsmith »

Yes,

my only concern is local IT teams altering the core check configuration that is required.

I have developed all the functionality, this is now the final step, before switching the new distributed monitoring to be PRODUCTION.

I will probably develop the script to update to include some notification, of error on reload.

Code: Select all

psudocode

svn update /etc/nagios/global
svn update /etc/nagios/siteName
.
.
.
if (NAGIOS PRE-FLIGHT CHECK RETURNS " EVERYTHING OK")
---- RELOAD NAGIOS COMMAND
else
 ----mail -s “SITE NAME - NAGIOS AUTOMATED UPDATE FAIL” me@myDomain.com
hopefully I could then at the least leave NAGIOS running with its existing config, to monitor, and investigate the cause of the error in configuration.
mguthrie
Posts: 4380
Joined: Mon Jun 14, 2010 10:21 am

Re: THOUGHTS - Managing a distributed NAGIOS Setup

Post by mguthrie »

I suppose you could also just have an nrpe or passive check to make sure the nagios process is running on all of the remote servers as well.
Locked