Page 1 of 1

Logfile check sending alert from old time stamp

Posted: Sun Feb 11, 2018 9:58 pm
by ericssonvietnam
Below is my logfile configuration file which i am using but i am receiving alert of older timestamp as logfile on server is not updating properly
Below are the log and my observation can you suggest on the same
[nagios@HNRBT1 conf]$ cat wrapperstop.conf
$seekfilesdir = '/home/nagios/logfile_basedir/tmp';
$protocolsdir = '/home/nagios/logfile_basedir/tmp';
$scriptpath = '/home/nagios/bin';
#$MACROS = { CL_DISK01 => "/dev/dsk/c0d1", CL_DISK02 => "/dev/dsk/c0d2" };

@searches = (
{
tag => 'MRTB_Jboss_wrapperstop',
criticalpatterns => ['Wrapper Stopped'] ,
okpatterns => ['"Server startup'] ,
logfile => '/opt/mrbt/jboss/server/all/log/wrapper.log'
#rotation => 'SUSE',
},
########MRBT_tomcat
{
tag => 'MRBT_Tomcat__wrapperstop',
criticalpatterns => ['Wrapper Stopped'] ,
okpatterns => ['"Server startup'] ,
logfile => '/opt/mrbt/tomcat/logs/wrapper.log'
}

);
+++++++++++++++++++++++++++++++++++++++++++
[nagios@HNRBT1 conf]$ ls -ltr /opt/mrbt/jboss/server/all/log/wrapper.log
-rw-r--r-- 1 root root 5316021 Feb 12 07:30 /opt/mrbt/jboss/server/all/log/wrapper.log
[nagios@HNRBT1 conf]$ date
Mon Feb 12 09:32:37 ICT 2018
[nagios@HNRBT1 conf]$ grep -i "Wrapper Stopped" /opt/mrbt/jboss/server/all/log/wrapper.log
STATUS | wrapper | 2017/05/16 01:08:33 | <-- Wrapper Stopped
STATUS | wrapper | 2017/05/26 01:12:21 | <-- Wrapper Stopped
STATUS | wrapper | 2017/06/15 01:01:30 | <-- Wrapper Stopped
STATUS | wrapper | 2017/08/08 01:03:15 | <-- Wrapper Stopped
STATUS | wrapper | 2017/08/08 03:29:24 | <-- Wrapper Stopped
STATUS | wrapper | 2017/08/11 02:12:35 | <-- Wrapper Stopped
STATUS | wrapper | 2017/08/25 04:06:38 | <-- Wrapper Stopped
STATUS | wrapper | 2017/10/16 23:56:20 | <-- Wrapper Stopped
STATUS | wrapper | 2017/10/17 14:50:29 | <-- Wrapper Stopped
STATUS | wrapper | 2017/10/31 04:53:11 | <-- Wrapper Stopped
STATUS | wrapper | 2018/01/16 00:38:13 | <-- Wrapper Stopped
STATUS | wrapper | 2018/01/16 00:52:45 | <-- Wrapper Stopped
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[nagios@HNRBT1 conf]$ ls -ltr /opt/mrbt/jboss/server/all/log/wrapper.log
-rw-r--r-- 1 root root 5316021 Feb 12 07:30 /opt/mrbt/jboss/server/all/log/wrapper.log
[nagios@HNRBT1 conf]$ date
Mon Feb 12 09:39:15 ICT 2018
[nagios@HNRBT1 conf]$ grep -i "Wrapper Stopped" /opt/mrbt/tomcat/logs/wrapper.log
STATUS | wrapper | 2016/08/24 01:44:16 | <-- Wrapper Stopped
STATUS | wrapper | 2016/08/24 02:26:58 | <-- Wrapper Stopped
[nagios@HNRBT1 conf]$ ls -ltr /opt/mrbt/tomcat/logs/wrapper.log
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Alert i attached for the same

Can you suggest what we can do in this case

Re: Logfile check sending alert from old time stamp

Posted: Mon Feb 12, 2018 3:56 pm
by cdienger
The first time check_logfile is run, it will search the entire file and note the end of the log in the seek file. Subsequent searches will start at the offset set in the seek file(/home/nagios/logfile_basedir/tmp contains the seek files). So just running the check again should clear this critical message.

From the command's documenation page at https://labs.consol.de/nagios/check_logfiles/index.html:
In principle check_logfiles scans a log file until the end-of-file is reached. The offset will then be saved in a so-called seekfile. The next time check_logfiles runs, this offset will be used as the starting position inside the log file. In the event that a rotation has occurred in the meantime, the rest of the rotated archive will be scanned also.

Re: Logfile check sending alert from old time stamp

Posted: Mon Feb 12, 2018 10:10 pm
by ericssonvietnam
cdienger wrote:The first time check_logfile is run, it will search the entire file and note the end of the log in the seek file. Subsequent searches will start at the offset set in the seek file(/home/nagios/logfile_basedir/tmp contains the seek files). So just running the check again should clear this critical message.

From the command's documenation page at https://labs.consol.de/nagios/check_logfiles/index.html:
In principle check_logfiles scans a log file until the end-of-file is reached. The offset will then be saved in a so-called seekfile. The next time check_logfiles runs, this offset will be used as the starting position inside the log file. In the event that a rotation has occurred in the meantime, the rest of the rotated archive will be scanned also.
Hi Cdinger
i understood the working of the plugin but still my concern is we haven't changed anything with conf file or plugin but we still get these type of alert from different nodes in certain period of time can you suggest if we can fix this out anyhow.

Re: Logfile check sending alert from old time stamp

Posted: Tue Feb 13, 2018 10:39 am
by cdienger
There may be problems writing or reading the seek file. Do you see a seek file in '/home/nagios/logfile_basedir/tmp'; ? Does it get updated with offset information when the plugin is run? Is the file maybe getting deleted?

Re: Logfile check sending alert from old time stamp

Posted: Mon Mar 12, 2018 12:54 am
by ericssonvietnam
cdienger wrote:There may be problems writing or reading the seek file. Do you see a seek file in '/home/nagios/logfile_basedir/tmp'; ? Does it get updated with offset information when the plugin is run? Is the file maybe getting deleted?

Again i have observed the similar issue where the alarm persist for time period longer than 30 min which is defined in the logfile conf

-bash-3.00$ cat logfile_basedir/conf/WSMS_Major.conf
$seekfilesdir = '/export/home/nagios/logfile_basedir/tmp';
$protocolsdir = '/export/home/nagios/logfile_basedir/tmp';
$scriptpath = '/export/home/nagios/bin';
#$MACROS = { CL_DISK01 => "/dev/dsk/c0d1", CL_DISK02 => "/dev/dsk/c0d2" };

@searches = (
{
tag => 'WSMS_Major',
criticalpatterns => ['MAJOR'] ,
options => 'sticky=3600,noprotocol,report=long,allyoucaneat',
logfile => '/export/home/projadm/bin/SYSTEM.LOG'

}
);

I think there is some issue with plugin or something as i am getting these types of issue from long time.

Re: Logfile check sending alert from old time stamp

Posted: Mon Mar 12, 2018 1:14 pm
by cdienger
options => 'sticky=3600,noprotocol,report=long,allyoucaneat',

The above would set the sticky value to an hour. To set it to a half hour use:

options => 'sticky=1800,noprotocol,report=long,allyoucaneat',