Page 1 of 1

Splunk on Nagios (To monitor Nagiosxi)

Posted: Mon Jul 29, 2013 7:08 pm
by KiwiBloke
Hi,

We have Nagiosxi and we also have Splunk. We use both independently, with a little bit of overlap depending on whats best for what.

What I would like to do is have a fallback for Nagiosxi (as we use Nagiosxi to monitor Splunk services and processes). So i plan to install the Splunk universal forwarder onto the Nagiosxi server and create a server profile for it so that I can manage it via the Splunk deployment server.

All we need/want to do is monitor the various Nagiosxi log files. and then monitor them within Splunk for errors. Can you advise which log files we should be targeting and perhaps what would be some basic things to watch for? (other than things like "error")

Cheers,

KB

Re: Splunk on Nagios (To monitor Nagiosxi)

Posted: Tue Jul 30, 2013 9:36 am
by yancy
KiwiBloke,

you'll probably want to start with nagios.log and nagios.debug

Code: Select all

/usr/local/nagios/var/nagios.log
/usr/local/nagios/var/nagios.debug
Also, depending on the granularity, you can change the logging level as defined here:

http://nagios.sourceforge.net/docs/3_0/configmain.html

I also noticed you can have Nagios log to syslog which may simplify things for use with Splunk.

-Yancy

Re: Splunk on Nagios (To monitor Nagiosxi)

Posted: Tue Jul 30, 2013 10:01 am
by abrist
For system health:
dmesg or /var/log/messages - mostly looking for "seg fault"

Performance data:
/usr/local/nagios/var/perfdata.log - Looking for timeouts: "TIMEOUT: Please check your npcd.cfg"
/usr/local/nagios/var/npcd.log - Looking for max load threshold warnings: "NPCD: WARN: MAX load reached:"

Apache logs:
/var/log/httpd/error_log - Looking for "Error", and you could most likely overlook the warnings.
/var/log/httpd/access_log - May be useful if you use ssl only, or wish to see a certain IPs usage of XI.

Nagios Core log:
/usr/local/nagios/var/nagios.log - this list of things to watch out for in here is big, and rather environment specific. Many of the things were alerting of will start with "Error".

Mysqld:
/var/log/mysqld.log - Looking for "crashed"

Re: Splunk on Nagios (To monitor Nagiosxi)

Posted: Tue Jul 30, 2013 5:24 pm
by KiwiBloke
Hi,

Thanks guys this is great stuff.

Cheers,

KB.