Nagios Support Forum

Hello,

We have a situation where we require Nagios Log Server to send us an alert when -- ERROR -- (spaces before and after ERROR) is found in the logs... But when I create an alert for the same, I get false alarms which has WARN and some even don't have even a letter related to the search string give and even I see the same when I search in the Dashboards...

Can there be some help in resolving the issue and creating an alert with "-- ERROR --" please?

Thanks!

Srinivas Mandalika

Spaces in Elasticsearch can be tricky. Can you share some samples of the sorts of logs you're trying to alert on?

Could you also share the query associated with your alert? If you're not sure how to export the raw query, a screenshot of the search+filters displayed on a dashboard with the query loaded should be sufficient.

Below is one of the log in the log file which has info that I am looking to create alert... I am searching for -- ERROR --

O 2017-06-21 14:16:02.456 -- ERROR -- [DefaultMessageListenerContainer-9] com.simon.eventmsg.service.src.PublishSourceEventService: Exception org.springframework.orm.hibernate4.HibernateSystemException: Could not access JTA Transaction to register synchronization; nested exception is org.hibernate.engine.transaction.jta.platform.spi.JtaPlatformException: Could not access JTA Transaction to register synchronization occurred processing with jmsListener Source System: Salesforce. Queue: app.eventmsg.sfevent.default publishedEvent PublishedEventDTO [sourceSystem=Salesforce, sourceSystemContext=null, eventId=a2B16000001ppX6EAI, eventType=Object Updated, sourceType=Opportunity, eventContext=0061600000twSiMAAU, status=New, createdDate=2017-06-21T18:10:11.000Z, createdById=00530000003ncsWAAQ, workQueue=app.eventmsg.sfevent.default]

Please find the screenshot of the alert that I created...

Srinivas Mandalika

It's likely picking up the severity level but possibly another field. Try adding a filter by clicking the little magnifying glass next to the message field and modifying the filter accordingly.

Nope, even after I apply, I get false alarms...

No success...

Srinivas Mandalika

Rather than being at the mercy of what analyzer Elasticsearch is using (which I suspect is the issue), another option might be to use a grok filter to parse out the severity directly from your Java logs:

http://grokdebug.herokuapp.com/

Using the source logs as your input and this for the pattern: I get the following out: Applied as a Logstash filter (which can be added/removed/changed from the Configuration page in Nagios Log Server), it might look like this (you don't need to use [host] if you have a more convenient field to filter on): This would make your search much easier since you could search the java_severity field directly like so: You could even build out some slick dashboards to give a visual breakdown of different severity messages. Or if you wanted to get more granular, you could grok filter the class paths and see which parts of your Spring application are causing the most trouble.

Typically if there is an all-important piece of some logs that needs to be analyzed, I recommend people isolate that specific piece of information via Logstash filtering into it's own field.

Thanks for the suggestion, mcapra!

Did mcapra's suggestion help?

I am bad at this grok... So, I think I am unable to get the output what is expected...

Srinivas Mandalika

Well, the grok is already done; It's a few posts up

All you need to figure out is how you're isolating the "Java logs" traffic. If it's a single application server with a single IP address (or hostname), you should be able to plug that in for the [host].

You could just as easily throw that grok filter into a filter rule without further worry, but then it would apply to all of your logs, not just the Java logs. Saying if the logs are coming from my Java source as a part of your filter is a good way to narrow things down a bit.

Writing Logstash input/filter/output rules can be a bit intimidating at first, but it's a really really powerful part of the product.

I mean I am unable to add source code as my input filter... I am stuck at very basic step... I added given code as input "--\s%{DATA:java_severity}\s--" ... That is my level of understanding in this Grok..

Srinivas Mandalika