Doubled Entries in /var/log/messages

[nseltzer]

Hey all,

There is a strange quirk where /var/log/messages is doubling all Nagios-related entries.

Code: Select all

Feb  6 12:14:15 papmonmp10 nagios: SERVICE ALERT: hqsadbp0;SNMP - CPU Usage;OK;HARD;5;8 CPU, average load 28.1% < 95% : OK 
Feb  6 12:14:15 papmonmp10 nagios: SERVICE ALERT: hqsadbp0;SNMP - CPU Usage;OK;HARD;5;8 CPU, average load 28.1% < 95% : OK 

Feb  6 12:15:51 papmonmp10 nagios: SERVICE ALERT: HQSADBT0;SNMP - Process - NTPD;OK;HARD;5;1 process matching ntpd (> 0) 
Feb  6 12:15:51 papmonmp10 nagios: SERVICE ALERT: HQSADBT0;SNMP - Process - NTPD;OK;HARD;5;1 process matching ntpd (> 0)

I've exhausted my Google-fu and have perused the forums to no avail.

Thanks,



Nic Seltzer

System Info:

Code: Select all

$ uname -a
Linux papmonmp10.cabelas.corp 2.6.18-308.24.1.el5 #1 SMP Wed Nov 21 11:42:14 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

$cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.8 (Tikanga)

Manual or VM install? Manual XI install

[lmiltchev]

Can you run the following command and post the output?

Code: Select all

ps -ef | grep /bin/nagios

[nseltzer]

Code: Select all

$ sudo ps -ef | grep /bin/nagios | grep -v grep
nagios    5406     1  3 10:25 ?        00:04:55 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

Nic

[slansing]

I believe this is something that Core does by default I will check into it though.

[lmiltchev]

Open the nagios.cfg file in a text editor and change this line:

Code: Select all

use_syslog=1

to this

Code: Select all

use_syslog=0

Then restart nagios service:

Code: Select all

service nagios restart

See if this is going to remove the double entries.

[nseltzer]

It was successful in one respect. No new Nagios-related entries were being populated in to /var/log/messages. Any ideas?

[lmiltchev]

I'm sorry about that - I guess this is not what you really want (to disable syslog facility altogether). You can change it back:

Code: Select all

use_syslog=1
service nagios restart

Do you see the same messages in the nagios log?

Code: Select all

tail /usr/local/nagios/var/nagios.log

Are these active checks are snmp traps?

[nseltzer]

We're only seeing errors in nagios.log. Everything nagios-related is being duplicated in /var/log/messages, for example:

Code: Select all

Feb  6 12:54:18 papmonmp10 nagios: Warning: Host 'shqdbadm02' has no default contacts or contactgroups defined!
Feb  6 12:54:18 papmonmp10 nagios: Warning: Host 'shqdbadm02' has no default contacts or contactgroups defined!
Feb  6 12:54:18 papmonmp10 nagios: Finished daemonizing... (New PID=6605) 
Feb  6 12:54:18 papmonmp10 nagios: Finished daemonizing... (New PID=6605) 

At this time, I don't believe we have any snmptraps going to this host, so all check statuses that we're seeing would be from active hosts.

Nic

[lmiltchev]

Are the errors in the nagios.log also duplicated (like the ones in the /var/log/messages)?

[nseltzer]

Nope. They appear to be groovy.

Code: Select all

[1321647824] Warning: Return code of 127 for check of service 'Current Load' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1321647884] Warning: Return code of 127 for check of service 'Current Users' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1321647944] Warning: Return code of 127 for check of service 'PING' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1321648002] Caught SIGTERM, shutting down...
[1321648002] Successfully shutdown... (PID=3136)

Nic