False positive alerts?

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
Jklre
Posts: 163
Joined: Wed May 28, 2014 1:56 pm

False positive alerts?

Post by Jklre »

I have an alert that started acting up yesterday. When i manually look at the query results I see the expected data coming back. This is a negative alert watching a heartbeat message for if there is less than 3 messages in the past 90 minutes. Which should ideally never occur unless there is an issue but we are seeing some false positives showing up.

Alert Name:
OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT
Check Interval 90m
Lookback Period 90m

Thresholds:
3: 3: # of events
Alert Method
Email

Here is histogram of the alerts query.
results.jpg
Here are some of the false positives we are receiving:
2016-05-15T01:22:05.253-07:00 ALERT Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned CRITICAL: 1 matching entries found |logs=...
2016-05-20T18:23:39.233-07:00 ALERT Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned CRITICAL: 2 matching entries found |logs=...


Most of the other audit logs are coming back properly (see below)

Code: Select all

2016-05-26T13:33:27.182-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-26T12:02:38.456-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-26T10:31:38.037-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-26T09:00:53.941-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-26T07:30:13.701-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-26T05:59:19.073-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-26T04:28:39.464-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-26T02:58:34.085-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-26T01:27:43.409-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 23 matching entries found |logs=23;3:...	 
2016-05-25T23:56:53.225-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 25 matching entries found |logs=25;3:...	 
2016-05-25T22:26:08.250-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T20:55:28.560-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T19:24:37.964-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T17:54:13.535-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 11 matching entries found |logs=11;3:...	 
2016-05-25T16:23:33.909-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T14:52:43.760-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T13:21:48.129-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T11:51:03.582-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T10:20:13.277-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T08:49:28.483-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T07:18:43.670-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T05:48:03.169-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T04:17:13.564-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T02:46:42.962-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-25T01:16:13.449-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 19 matching entries found |logs=19;3:...	 
2016-05-24T23:45:23.909-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T22:14:20.816-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T20:43:30.401-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T19:13:18.399-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T17:42:14.310-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 15 matching entries found |logs=15;3:...	 
2016-05-24T16:11:14.285-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T14:41:14.049-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T13:11:01.198-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T11:40:09.086-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T10:09:54.042-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T08:38:53.893-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T07:08:48.565-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T05:37:49.295-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T04:06:49.132-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T02:35:49.096-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-24T01:04:48.971-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 22 matching entries found |logs=22;3:...	 
2016-05-23T23:34:04.656-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T22:04:04.163-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T20:33:13.870-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T19:02:20.495-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T17:31:20.644-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 18 matching entries found |logs=18;3:...	 
2016-05-23T16:00:28.764-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T14:29:33.692-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T12:58:43.390-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T11:27:53.831-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T09:57:08.288-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T08:26:30.191-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T06:55:20.660-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T05:24:29.655-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T03:53:29.281-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T02:22:33.463-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-23T00:51:43.612-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 22 matching entries found |logs=22;3:...	 
2016-05-22T23:20:55.193-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T21:50:18.544-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T20:19:34.878-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T18:48:43.484-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T17:17:44.106-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 21 matching entries found |logs=21;3:...	 
2016-05-22T15:46:53.659-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T14:16:08.980-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T12:45:21.071-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T11:14:39.065-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T09:43:55.700-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T08:13:09.088-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T06:42:18.598-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T05:11:29.082-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T03:41:11.256-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T02:10:38.877-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-22T00:39:39.897-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 21 matching entries found |logs=21;3:...	 
2016-05-21T23:08:48.386-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-21T21:38:05.740-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-21T20:07:13.723-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-21T18:36:23.169-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-21T17:05:34.072-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 25 matching entries found |logs=25;3:...	 
2016-05-21T15:34:48.999-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-21T14:03:59.535-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-21T12:33:08.540-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-21T11:02:13.293-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-21T09:31:13.732-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-21T08:00:23.374-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-21T06:29:34.186-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-21T04:58:49.897-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-21T03:28:31.551-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-21T01:57:03.778-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-21T00:26:29.082-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 21 matching entries found |logs=21;3:...	 
2016-05-20T22:56:24.001-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T21:25:31.753-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T19:54:35.888-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T18:23:39.233-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned CRITICAL: 2 matching entries found |logs=...	 
2016-05-20T16:53:24.477-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T15:22:38.827-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T13:52:23.687-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T12:22:18.746-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-20T10:51:38.445-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T09:21:14.296-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T07:50:43.965-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T06:20:31.587-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T04:49:36.747-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T03:18:52.962-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T01:47:54.084-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-20T00:17:43.766-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 21 matching entries found |logs=21;3:...	 
2016-05-19T22:47:23.999-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-19T21:16:31.950-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-19T19:46:23.365-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-19T18:16:09.053-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 3 matching entries found |logs=3;3:;3...	 
2016-05-19T16:46:03.963-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-19T15:15:54.162-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-19T13:45:02.357-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-19T12:14:53.714-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-19T10:44:28.838-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-19T09:14:19.006-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-19T07:43:54.382-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-19T06:13:02.618-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-19T04:42:49.249-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-19T03:11:47.964-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 25 matching entries found |logs=25;3:...	 
2016-05-19T01:41:14.015-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-19T00:11:08.130-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 22 matching entries found |logs=22;3:...	 
2016-05-18T22:40:15.754-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T21:10:08.878-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T19:39:54.064-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T18:09:44.096-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 6 matching entries found |logs=6;3:;3...	 
2016-05-18T16:39:43.190-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T15:09:42.438-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T13:39:29.084-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T12:09:19.212-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T10:38:54.026-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T09:08:43.683-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T07:38:38.881-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T06:08:09.003-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T04:37:48.304-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T03:07:15.693-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-18T01:36:56.509-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-18T00:05:44.160-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 23 matching entries found |logs=23;3:...	 
2016-05-17T22:35:38.786-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-17T21:05:23.522-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-17T19:35:13.898-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-17T18:05:11.640-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 7 matching entries found |logs=7;3:;3...	 
2016-05-17T16:35:09.067-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-17T15:04:43.386-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-17T13:34:28.977-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-17T12:04:04.284-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-17T10:34:03.743-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-17T09:03:34.091-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-17T07:33:18.822-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-17T06:03:04.016-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-17T04:32:43.211-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-17T03:02:41.399-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-17T01:32:33.468-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 25 matching entries found |logs=25;3:...	 
2016-05-17T00:02:24.130-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 25 matching entries found |logs=25;3:...	 
2016-05-16T22:32:23.464-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T21:02:19.539-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T19:31:25.735-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T18:01:09.061-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 9 matching entries found |logs=9;3:;3...	 
2016-05-16T16:30:53.491-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T15:00:38.500-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T13:30:04.582-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T11:59:49.598-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T10:29:18.786-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T08:58:33.945-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T07:28:14.342-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T05:57:52.051-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T04:27:38.389-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T02:57:13.412-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-16T01:26:43.591-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 23 matching entries found |logs=23;3:...	 
2016-05-15T23:56:19.140-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T22:26:19.400-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T20:56:04.039-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T19:25:38.936-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T17:55:08.369-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 12 matching entries found |logs=12;3:...	 
2016-05-15T16:24:48.999-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T14:54:48.729-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T13:24:31.242-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T11:54:09.289-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T10:23:41.864-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T08:53:24.347-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T07:23:20.720-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T05:53:04.145-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T04:22:35.904-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T02:52:08.458-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-15T01:22:05.253-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned CRITICAL: 1 matching entries found |logs=...	 
2016-05-14T17:14:14.522-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 22 matching entries found |logs=22;3:...	 
2016-05-14T15:43:41.481-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-14T14:13:18.852-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-14T12:42:56.769-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-14T11:12:38.625-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-14T09:42:33.206-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-14T08:11:54.345-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-14T06:41:26.840-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-14T05:11:03.942-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-14T03:40:32.145-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-14T02:10:19.029-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-14T00:39:37.840-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 21 matching entries found |logs=21;3:...	 
2016-05-13T23:09:14.286-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T21:38:36.954-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T20:08:09.246-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T18:37:56.376-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T17:07:24.272-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 24 matching entries found |logs=24;3:...	 
2016-05-13T15:36:34.099-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T13:56:09.229-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T12:25:19.324-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T10:55:14.527-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T09:25:08.973-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T07:55:04.551-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T06:24:08.866-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T04:53:14.964-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T03:22:14.148-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T01:51:19.563-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-13T00:20:37.430-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 21 matching entries found |logs=21;3:...	 
2016-05-12T22:50:34.583-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T21:19:33.908-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T19:48:30.619-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T18:17:29.232-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 3 matching entries found |logs=3;3:;3...	 
2016-05-12T16:46:54.266-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T15:16:04.333-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T13:45:04.724-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T12:14:54.031-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T10:44:10.586-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T09:12:54.964-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-12T07:42:51.929-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T07:42:51.897-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T06:11:49.302-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T04:40:59.188-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T03:10:33.949-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-12T01:40:34.003-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-12T00:09:37.083-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 22 matching entries found |logs=22;3:...	 
2016-05-11T22:38:39.609-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-11T21:08:11.803-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-11T19:37:41.513-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-11T18:07:13.350-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 6 matching entries found |logs=6;3:;3...	 
2016-05-11T16:36:14.055-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-11T15:05:40.436-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-11T13:35:34.865-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-11T12:05:29.597-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-11T10:35:19.747-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-11T09:04:39.216-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-11T07:34:31.038-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-11T06:03:44.643-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-11T04:33:04.664-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-11T03:02:35.615-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-11T01:32:34.281-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 25 matching entries found |logs=25;3:...	 
2016-05-11T00:01:39.709-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 25 matching entries found |logs=25;3:...	 
2016-05-10T22:30:39.412-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T20:59:54.296-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T19:29:40.754-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T17:59:24.540-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 9 matching entries found |logs=9;3:;3...	 
2016-05-10T16:29:24.030-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T14:58:49.900-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T13:27:48.952-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T11:56:54.490-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T10:26:54.284-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T08:56:11.089-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T07:25:34.474-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T05:54:49.634-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T04:24:49.244-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T02:53:54.247-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-10T01:23:30.411-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 22 matching entries found |logs=22;3:...	 
2016-05-09T23:52:29.482-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T22:21:34.578-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T20:50:44.082-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T19:19:40.080-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T17:49:08.814-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 12 matching entries found |logs=12;3:...	 
2016-05-09T16:18:14.533-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T14:47:14.406-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T13:16:14.590-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T11:45:45.903-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T10:15:44.413-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T08:44:44.072-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T07:13:44.234-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T05:42:49.585-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T04:12:09.352-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-09T02:41:14.265-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-09T01:10:29.345-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 21 matching entries found |logs=21;3:...	 
2016-05-08T23:39:34.451-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-08T22:09:02.838-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-08T20:38:08.831-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-08T19:07:29.409-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-08T17:36:34.748-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 15 matching entries found |logs=15;3:...	 
2016-05-08T16:05:39.258-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-08T14:35:04.409-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-08T13:04:09.129-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-08T11:33:24.094-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-08T10:03:19.363-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-08T08:32:50.802-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-08T07:01:49.374-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-08T05:30:53.888-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-08T04:00:53.919-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-08T02:30:27.744-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-08T00:59:34.155-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 21 matching entries found |logs=21;3:...	 
2016-05-07T23:28:39.275-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T21:57:44.090-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T20:26:54.388-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T18:56:54.720-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T17:26:04.952-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 18 matching entries found |logs=18;3:...	 
2016-05-07T15:55:09.720-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T14:24:09.351-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T12:53:09.016-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T11:22:40.351-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T09:52:38.965-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T08:21:43.940-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T06:50:43.907-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T05:20:08.824-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T03:49:09.278-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T02:18:09.521-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-07T00:48:09.289-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 21 matching entries found |logs=21;3:...	 
2016-05-06T23:17:41.347-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-06T21:47:34.494-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-06T20:16:39.596-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-06T18:45:44.485-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-06T17:14:49.316-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 22 matching entries found |logs=22;3:...	 
2016-05-06T15:44:19.264-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-06T14:14:19.130-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-06T12:43:19.173-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-06T11:13:09.718-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 26 matching entries found |logs=26;3:...	 
2016-05-06T09:42:34.909-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...	 
2016-05-06T08:12:29.011-07:00	ALERT	Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...
You do not have the required permissions to view the files attached to this post.
Top
Jklre
Posts: 163
Joined: Wed May 28, 2014 1:56 pm

Re: False positive alerts?

Post by Jklre »

here is a longer view of the manual results sorted by hour.
results1.jpg
You do not have the required permissions to view the files attached to this post.
Top
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:
Contact hsmith

Re: False positive alerts?

Post by hsmith »

The sample size of 2 isn't giving a ton of information.

Code: Select all

    2016-05-20T18:23:39.233-07:00   ALERT   Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned CRITICAL: 2 matching entries found |logs=...   
    2016-05-20T16:53:24.477-07:00   ALERT   Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...
These seemed to run at the right times...

Code: Select all

    2016-05-15T01:22:05.253-07:00   ALERT   Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned CRITICAL: 1 matching entries found |logs=...   
    2016-05-14T17:14:14.522-07:00   ALERT   Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 22 matching entries found |logs=22;3:...
These ones seem to have a major jump in time.

Have you upgraded to 1.4.1 recently? That would give us something to go off of.
Former Nagios Employee.
me.
Top
Jklre
Posts: 163
Joined: Wed May 28, 2014 1:56 pm

Re: False positive alerts?

Post by Jklre »

hsmith wrote:The sample size of 2 isn't giving a ton of information.

Code: Select all

    2016-05-20T18:23:39.233-07:00   ALERT   Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned CRITICAL: 2 matching entries found |logs=...   
    2016-05-20T16:53:24.477-07:00   ALERT   Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 27 matching entries found |logs=27;3:...
These seemed to run at the right times...

Code: Select all

    2016-05-15T01:22:05.253-07:00   ALERT   Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned CRITICAL: 1 matching entries found |logs=...   
    2016-05-14T17:14:14.522-07:00   ALERT   Alert Name OPCOM*W*PROCES*ediWatchDog* *HEARTBEAT returned OK: 22 matching entries found |logs=22;3:...
These ones seem to have a major jump in time.

Have you upgraded to 1.4.1 recently? That would give us something to go off of.

We are still running 1.4.0. its interesting I'm seeing a lot more e-mail alerts than what the audit log shows.
Alerts.jpg
You do not have the required permissions to view the files attached to this post.
Top
Jklre
Posts: 163
Joined: Wed May 28, 2014 1:56 pm

Re: False positive alerts?

Post by Jklre »

I found the root cause.

It looks like after 5/25 the two nodes stopped talking to each other. Since the checks are load balanced it would only alert when the check was looking on the 2nd node that wasn’t syncing. I was able to get the databases to talk to each other again. We only noticed this check because it was a negative check and was looking for the absence of logs.

I’m going to setup some other alerting to detect this in the future. We had a network indecent that i believe caused this.

Thank you!
Top
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:
Contact hsmith

Re: False positive alerts?

Post by hsmith »

I'm glad to see you didn't find a bug one this one. Is this thread all right to lock up?
Former Nagios Employee.
me.
Top
Jklre
Posts: 163
Joined: Wed May 28, 2014 1:56 pm

Re: False positive alerts?

Post by Jklre »

hsmith wrote:I'm glad to see you didn't find a bug one this one. Is this thread all right to lock up?
yes please close this.

thank you.
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: False positive alerts?

Post by mcapra »

Closing this!
Former Nagios employee
https://www.mcapra.com/
Top
Locked

Return to “Nagios Log Server”