Tomcat logging with alerts

[shabbaynwa]

Hello,

New to Nagios Log Server as we have had a recent demo and believe this product could work for our environments we really like the ability to run to one log management tool vs scrolling through logs on each individual server.

We heavily use Apache Tomcat for our applications and I was able to follow the instructions to begin importing the logs into the server, I see some logs come in but how would I set up alerts? For example we like to be notified if the following threads (below) begin to pile up and since this could happen multiple times a day we only want to be notified for any new stuck threads.

Lets say I want a warning email for the below for 1 stuck thread but then a critical error alert if it jumped to lets say 5? How would I accomplish something like this?

WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8083-exec-132] (id=[18646020]) has been active for [123,134] milliseconds () to serve the same request for [] and may be stuck (configured threshold for this StuckThreadDetectionValve is [120] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.

[ssax]

Go to Dashboards, search for "and may be stuck" in the query input box.

In the top right click, query manager, input a name like "Tomcat Stuck", make it global, and click the Create button.

Now to go to Alerting:
- Click New Alert
- Name it
- Type Query
- Query - Select the one you created
- Check Interval 5m
- Lookback Peroiod 5m
- Thresholds 1 5
- Select whatever notification method you'd like
- Click the Create Alert button

[shabbaynwa]

Thank you for this we will be implementing this once we find out what happened to our application servers as they have stopped reporting into in the log server thanks for these detailed steps much appreciated.

[ssax]

Please create another thread or ticket for that if you're having issues and we can help you debug.

Reserve this one for the Tomcat issue.

Include any messages from /var/log/messages on your application servers related to rsyslog (if that's what you're using).

You can also test that packets are being received with this command:
- Run this on the Log Server and change X.X.X.X to one of your application servers

Code: Select all

tcpdump -nni any host X.X.X.X