Nagios Log monitoring

[RIDS_I2MP]

Hello Team,

We are currently monitoring custom logs on our Windows servers. We monitor the logs using check_log3 plugin. Below is an example for the same:

check_log = scripts\check_log3.exe -l "D:\\Oracle\\Middleware\\user_projects\\DataMan0\\diagnostics\\logs\\services\\HyS9aifWeb_DataMan0-syserr.txt" -p "com.hyperion.reporting.util.HyperionReportException: Error connecting to database connection NIPPERA3: EPMHFM-69052 : The system was unable to open the application" -s "c:\seek1" -c 1

We add above lines in our nsclient.ini file and we get the output. We are using NSCP-0.5.2.35-x64 version.

In above example, -p denotes the string that we want to check in the log file and for multiple strings, we use multiple -p option.

Now, we have a new requirement in which we have to monitor the below three lines:

**** info Fri Aug 09 00:37:21 BST 2019 1565307441491 /atg/commerce/catalog/custom/CatalogMaintenanceService Service Lock CatalogMaintenance released for service CatalogMaintenance.
**** info Fri Aug 09 00:37:21 BST 2019 1565307441522 /ricoh/webshop/commerce/catalog/CatalogMaintenanceServiceScheduler DataSource before switch : DataSourceB datasource after switch: DataSourceA
**** info Fri Aug 09 00:37:21 BST 2019 1565307441522 /ricoh/webshop/commerce/catalog/CatalogMaintenanceServiceScheduler Exiting with datasource : DataSourceA

Here, we want to receive the alert only if these three lines comes in the same order, time/date will be changed.

But, if any of these lines is not present, Nagios should not throw any alert.
In other words, we want to receive the alert only if these 3 lines comes in the same sequence.

Please let me know how we can achieve this using check_log3 plugin.

[ssax]

I'm trying to think how this would be doable, it's essentially just a multiline regex match so that would need to be defined, which will be complicated, you will need to use these options:

Code: Select all

Search pattern control:

-p, --pattern=

    The regular expression to scan for in the log file.  If specified more
    than once, the patterns will be combined into an expression of the form
    'pattern1|pattern2|pattern3|...' (but also see the -A option).

-P, --patternfile=

    File containing regular expressions, one per line, which will be combined
    into an expression of the form 'line1|line2|line3|...' (but also see -A).

-A, --and

    Use AND instead of OR to combine multiple patterns specified via the -p or
    -P options.  A line must match all patterns to be counted as a match.

    This is equivalent to '(?=.*pattern1)(?=.*pattern2)(?=.*pattern3)...'.

-n, --negpattern=

    The regular expression to skip in the log file.  Can be specified multiple
    times, in which case they will be combined as 'pat1|pat2|pat3|...'.

-f, --negpatternfile=

    Specifies a file with regular expressions which will all be skipped.

-i, --case-insensitive

    Do a case insensitive scan.

I'll lab it up and take a crack at it but if you are a regex master, let me know!

If you don't hear from me by midday Monday, please reply to this post so that it pops back up on my dashboard.

EDIT: Will there ever been additional, non-matching lines between them or will they be always one after the other in order?

[RIDS_I2MP]

Hi,

Thanks for the reply!!

It will always be in the same manner, always one after the other in order, no lines would come in middle. If anything comes, Nagios should ignore it.

[ssax]

I have been unable to get multiline match to work on this, that multiline regex is besting me.

tried this form:

Code: Select all

m/blah/

Also tried this form:

Code: Select all

/(?m)^\*\*\* info(?m)^\*\*\* info(?m)^\*\*\* info/

Are you open to using another plugin such as this?

https://labs.consol.de/nagios/check_logfiles/index.html

[RIDS_I2MP]

Hello,

Yes we are good to use it.
Please help us to understand how we can work on it.

[ssax]

I'm having trouble with this one as well...

I'm having trouble just matching this text with either of them:

/tmp/TEST

Code: Select all

test1
test2
test3

I can get it to work with grep but neither plugin works:

Code: Select all

grep -Pzo '^test1$\n^test2$\n^test3$' /tmp/TEST

[RIDS_I2MP]

Hello,

So, there is no way to get this monitored?

[ssax]

There is always a way to monitor something, the question is if there's an existing plugin that supports it AND that you can get to work. I'm currently unable to get the multi-line matching working with the current plugins (even though they support regular expressions, it doesn't mean they support multi-line match regular expressions) and I've spent a bunch of time on them trying to do it.

Technically, this works and you should be able to write a plugin around it:

Code: Select all

grep -Pzo '^\*\*\*\* info \w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} \w{3,4} \d{4} \d{13} \/atg\/commerce\/catalog\/custom\/CatalogMaintenanceService Service Lock CatalogMaintenance released for service CatalogMaintenance.$\n^\*\*\*\* info \w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} \w{3,4} \d{4} \d{13} \/ricoh\/webshop\/commerce\/catalog\/CatalogMaintenanceServiceScheduler DataSource before switch : DataSourceB datasource after switch: DataSourceA$\n^\*\*\*\* info \w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} \w{3,4} \d{4} \d{13} \/ricoh\/webshop\/commerce\/catalog\/CatalogMaintenanceServiceScheduler Exiting with datasource : DataSourceA $' YOURLOGFILE

This is the complete regex line from that command:

Code: Select all

^\*\*\*\* info \w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} \w{3,4} \d{4} \d{13} \/atg\/commerce\/catalog\/custom\/CatalogMaintenanceService Service Lock CatalogMaintenance released for service CatalogMaintenance.$\n^\*\*\*\* info \w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} \w{3,4} \d{4} \d{13} \/ricoh\/webshop\/commerce\/catalog\/CatalogMaintenanceServiceScheduler DataSource before switch : DataSourceB datasource after switch: DataSourceA$\n^\*\*\*\* info \w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} \w{3,4} \d{4} \d{13} \/ricoh\/webshop\/commerce\/catalog\/CatalogMaintenanceServiceScheduler Exiting with datasource : DataSourceA $

Please test that regex in your plugin to see if it works (and link me to the specific check_log3.exe file you are using (or attach it)).