Match related logs in Nagios Log Server

[li_alm]

Hello,

I need an alert that matches a sequnce of events (a sequence of message logs).
For a single ID, I have multiple message logs, and each message log has an event type.
E.g.:
message1= ID1 eventx
message2= ID1 eventy
message3= ID2 eventx
The correct flow is eventx and eventy for every ID, so ID1 is fine, while ID2 has an issue.
(the IDs and the events are fields that can be used in queries)

At the end of the day, I want to know all the IDs that have received eventx, but haven't received eventy.
In the above example: I want to find out about ID2.

Q1: Is there a way to do this in the nagios web interface?

Q2: If I cannot do this in the nagis interface, is there a helpful mechanism available directly in elasticsearch?

Thank you.
Regards,
Liviu

[li_alm]

Q3: A more simplified question: how can I find every ID that doesn't have an "eventy" message received?

Later edit: OK, I do not think everything is clear. I will try to explain things in a better way.
The log messages I am interested in have 2 important fields: messageID & eventID.
For a messageID, I receive multiple messages logs in nagios, each message having an eventID.
Eg:
messageID=13 eventID=100 .... (other fields)
messageID=13 eventID=101 ....
messageID=14 eventID=100 ....
messageID=14 eventID=103 ....

I want an alert that tells me whether there is a messageID that hasn't received an eventID=101.

Hope thing are more clear now.
Liviu

[mcapra]

I don't think that's something you can do with a simple Lucene search. I'm fairly certain it can be done in ElasticSearch using aggregations but don't have the time to lab it up right now:
https://www.elastic.co/guide/en/elastic ... ucket.html

Unfortunately I don't have a very good answer for reporting that information via the Nagios Log Server GUI. I can think of ways to restructure the messages via some sort of Logstash filter transformations (something like messageID=13 eventIDs=[100, 101, 102]) which would be a format Lucene can work with, but the work required would almost definitely be non-trivial.

[dwhitfield]

If I understand the question properly, you can do this with a grok filter. There's some info on them at https://assets.nagios.com/downloads/nag ... ilters.pdf

As far as the types of things you should look at while writing the filter, https://support.nagios.com/forum/viewto ... 37&t=31227 may be instructive.

[li_alm]

Thank you for you answers.

@mcapra: Yes, I also found the aggregations mechanisms.
Using aggregations, I can group by messageID and by eventID (using 2 aggregations - 1 inner aggregation) and I can find out the actual count for eventID=101 for every messageID.
Then, I need to parse the results (using my own script) to find the messageIDs that have count=0 for eventID=101.
My question was: is there a way to directly find the messageIDs missing eventID=101, without any further steps?
Also, it would have been useful to manage this in the web interface, so I can easily configure an email alert, but I understand it is not possible.

@dwhitfield: I do not think a grok filter can help me. As I understand, the grok filter is used to process one _single_ message, not relations between multiple messages, as I need.

[mcapra]

My question was: is there a way to directly find the messageIDs missing eventID=101, without any further steps?

Using aggregations alone, not that I am aware of (granted I am no ElasticSearch expert). If you could cross-reference an arbitrary number of buckets (though I think that's ES 2.x+) I think it could be done that way. Doing relational things with a non-relational database is always tricky.

If it were my problem to solve, I'd write a ruby filter in Logstash and have that be responsible for collecting all events on the basis of their message ID, then appending some sort of array field like eventIDs to each message before they get written to the database. This would assume the messages come in-order of messageID though (eg 15 -> 16 -> 17 -> 18 ... etc) and that there aren't any out-of-sequence messages being shipped by nxlog (eg 15 -> 13 -> 15 -> 14 -> 13 ... etc). You could then flush your queue/stack mechanism when a new messageID is received.

I imagine you could do the same sort of grouping->appending->writing pipe in nxlog itself, though my understanding of Ruby/Logstash is better than my understanding of nxlog so I chose the Ruby route

So once you had a field like this:

Code: Select all

eventIDs=[101, 102, 103, 104]

I think you could run a Lucene search like this to check if a given message has an eventID 101 but not an eventID 103:

Code: Select all

(eventIDs:101]) AND (NOT eventIDs:103)

Though this is all without testing in a lab environment. I could be wrong about stuff.

[dwhitfield]

If you could cross-reference an arbitrary number of buckets (though I think that's ES 2.x+) I think it could be done that way.

I believe 2.1: https://www.elastic.co/guide/en/elastic ... tion-2.1.0

We don't have roadmap for NLS 3.0 yet, but I suspect this might appear there. I suppose it's possible it'll appear in NLS 2.1, which would be fun since normally the numbers don't line up.

For clarity, NLS 2.0 is scheduled for release before the end of the year.