THOUGHTS - Managing a distributed NAGIOS Setup

An open discussion forum for obtaining help with Nagios Core. Nagios Core users of all experience levels are welcome here. Subforum have been created for the discussion of Nagios Core and Nagios Plugin development.

NOTE: The SourceForge.net mailing lists have been deprecated in favor of this forum in order to expedite support and provide additional features not available on the old mailing list.

THOUGHTS - Managing a distributed NAGIOS Setup

Postby mitchsmith » Fri Jul 22, 2011 5:15 am

Hi,

Just posting my thoughts really, and any input would be appreciated.

Here is my brain dump.....
I currently use nagios core to monitor multiple sites globally, and am in the process of distributing this using NSCA.

The idea is, currently I use nagios to monitor a very specific application server stack, and that one benefit of distributing this to local IT would allow (as well as the reduced load on the Central Nagios server), the local IT teams to extend my foundation NAGIOS configuration to monitor there other IT Systems.

I am considering using, source control to ensure that configurations are kept consistent, where by the Local NAGIOS server could perform a nightly SVN update to get any new cfg files, plugins. Then Reload nagios.

I have a basic .cfg file structure that allows me to distribute my core configuration, however I can see issue arrising if local IT staff edit the nagios.cfg to include their extended configuration (That I do not care about - I will provide a template that will ensure Obsess over services and hosts for their extension configs, is disabled)

My thoughts are:

If I created a dir within each site called Local, in which I could include a nagiosLocal.cfg which could look something like this;
Code: Select all
cfg_dir=/etc/nagios/local/commands
cfg_dir=/etc/nagios/local/services
cfg_dir=/etc/nagios/local/hosts


And Instruct the local IT teams to ensure that their configurations should be kept in this DIR only. As It would not be included in the SVN update process

Does anyone have any insight in to this... at the moment this is just my thoughts, I am open to suggestions on improving this idea.

Mitchell
mitchsmith
 
Posts: 33
Joined: Mon Apr 04, 2011 7:26 am

Re: THOUGHTS - Managing a distributed NAGIOS Setup

Postby mguthrie » Fri Jul 22, 2011 9:06 am

That strikes me as a pretty solid solution. Do you have any concerns about the local teams maintaining their configs properly and preventing Nagios from restarting correctly?

The monitoring structure you're describing is one that I'm familiar with. A classic passive distributed setup is one where local teams handle alerts, performance graphs, etc, and then the essential services get pushed up to the central server. I wouldn't call myself an expert on distributed monitoring by any means, but your plans seems fairly solid from what I can tell.
mguthrie
 
Posts: 4384
Joined: Mon Jun 14, 2010 10:21 am

Re: THOUGHTS - Managing a distributed NAGIOS Setup

Postby mitchsmith » Fri Jul 22, 2011 9:57 am

Yes,

my only concern is local IT teams altering the core check configuration that is required.

I have developed all the functionality, this is now the final step, before switching the new distributed monitoring to be PRODUCTION.

I will probably develop the script to update to include some notification, of error on reload.
Code: Select all
psudocode

svn update /etc/nagios/global
svn update /etc/nagios/siteName
.
.
.
if (NAGIOS PRE-FLIGHT CHECK RETURNS " EVERYTHING OK")
---- RELOAD NAGIOS COMMAND
else
----mail -s “SITE NAME - NAGIOS AUTOMATED UPDATE FAIL” me@myDomain.com


hopefully I could then at the least leave NAGIOS running with its existing config, to monitor, and investigate the cause of the error in configuration.
mitchsmith
 
Posts: 33
Joined: Mon Apr 04, 2011 7:26 am

Re: THOUGHTS - Managing a distributed NAGIOS Setup

Postby mguthrie » Fri Jul 22, 2011 3:55 pm

I suppose you could also just have an nrpe or passive check to make sure the nagios process is running on all of the remote servers as well.
mguthrie
 
Posts: 4384
Joined: Mon Jun 14, 2010 10:21 am


Return to Nagios Core

Who is online

Users browsing this forum: MSN [Bot], pepe_carlos and 36 guests