Setting up Alerts - a more efficient way?

[tomslmonitor]

I am currently setting up alerts and I was wondering if there was a way to set up a filter more efficiently than what I had planned.

So I have a general query which returns a bunch of events which all have different sources (SourceModuleName), I have around 50 log files. The way I thought about implementing this was to filter each log and create a single alert for each source, as you can image this is tedious and hard to update if I needed to in the future.

The alerts I am planning on setting up will be along the lines of 10 events in 10 minutes for a source, is there a more efficient way to set these alerts up?

Thanks!

[jolson]

is there a more efficient way to set these alerts up?

Typically alerts are set up reactively - that is, there is a dangerous log that came in and you want to be alerted if it arrives again (or too often). A good example might be *alert me if there are more than 20 SSH failed logins within the last 5 minutes* - this could be a filter that uses a pool of SSH fails gathered from every server reporting to Nagios Log Server.

If you're concerned about a rise in logs and want to monitor _every_ host for a spike in alerts (20+ over the last five minutes) the only option at the moment is to make an individual alert for each server.

My question to you is: what kind of alert are you trying to create? We might be able to come up with a solution if we know exactly what you'd like to do. Thanks!

Jesse

[tomslmonitor]

Hi Jesse, thanks for the quick response.

I have a grok filter in place at the moment which I've setup as fieldname 'WWaited', this is just an int.

The query I have is:

Code: Select all

+WWaited: [10000 TO *]

Essentially I have multiple logs coming in but is it possible to set up one alert that will be able to alert me when one log source gets more than 10 events in 10 minutes.

I'd like to avoid setting up 50 alerts filtering on 'SourceModuleName' if possible.

Thanks

[tmcdonald]

I don't think this is going to work, unfortunately. The alerting logic depends on the dashboard being used, and there is no way to say "This host OR this host OR this host" in a given dashboard.

We can put in a feature request for AND/OR logic in alerts possibly, but at the moment it's looking like separate queries.

[tomslmonitor]

Thanks for the response,

With that not being a solution, I'd have to be setting up a lot of alerts.

Is there a cap on alerts and upon setting up a couple hundred alerts (if there is no cap) where would I mostly take a performance hit if any?

Thanks in advanced

[jolson]

Is there a cap on alerts and upon setting up a couple hundred alerts (if there is no cap) where would I mostly take a performance hit if any?

In my experience there's likely a cap, but I have yet to see a client hit it. @jklre has several thousand alerts set up, and I don't think he's had any problems.

[tomslmonitor]

Thanks Jolson,

While on the topic of alerts, just playing around with some of the email templates macros.

Are there additional macros available? I found the %uniquehosts% to be helpful but I'd like to display SourceModuleName in the alerts if possible.

[jolson]

The only macros currently supported are the ones listed when you click the 'Macros' button. Could you list the macros that you'd like to see included in the future?

[tomslmonitor]

At the moment being able to have something similar to %uniquehosts% but for SourceModuleName (which shows top 10 sourcemodulename's matching the alert threshold).

I do realize this is a newly added additional and hopefully more macros will be released, but looking forward to future macros.

[jolson]

Understood - thanks for the feedback! I'll pass it on to the developers. Thanks.