Logstash stuck?

[CFT6Server]

Reviewing the Log server cluster this morning and realized the one of the nodes was no longer receiving from our firewall. I restarted the logstash service and seems like they are now coming back in. Review of the logstash logs did not indicated any errors, actually there wasn't much events at all.

So I have two questions:
1. Are there any other logging that I can look at to find out what happened?
2. How do I monitoring for this? The alerting component can alert for a threshold reached, can it do alerting when say there are 0 entries for a duration of 1hour?

Thanks.

[jolson]

Are there any other logging that I can look at to find out what happened?

The logstash logs would certainly be the place to look if it were stuck - though I've never seen it crash without errors before. Are you sure that the errors aren't in one of the rotated log files in /var/log/logstash/?

The alerting component can alert for a threshold reached, can it do alerting when say there are 0 entries for a duration of 1hour?

This can be done. When setting up an alert, the following syntax is valid:

Range definition Generate an alert if x...
10 < 0 or > 10, (outside the range of {0 .. 10})
10: < 10, (outside {10 .. ∞})
~:10 > 10, (outside the range of {-∞ .. 10})
10:20 < 10 or > 20, (outside the range of {10 .. 20})
@10:20 ≥ 10 and ≤ 20, (inside the range of {10 .. 20})

So to alert on 0 entries found, you can set up an alert as follows:

2015-07-29 10_55_44-Alerting • Nagios Log Server - Firefox Developer Edition.png