Alerts not triggert

[manfredrotzetter]

Hello,

We have the problem, that alerts are configured and when we "run the alert now" the alert is send correctly to the email address. But then the alert is never checked again. (Last Run never changes)

Parameters:
Check Interfal: 1m
Lookback Period: 5m
Thresholds: 50 2000
Alert Method: Email Users
1 User selectet
Email Template: System Default

Query:
"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1463995550226,"to":1464081950227}}},{"fquery":{"query":{"query_string":{"query":"message:(\"TS Continuity Counter\")"}},"_cache":true}}]}}}}}


Nagios Log Server Version: 1.4.1 (4 Nodes)

What is wrong?

Thanks,

Manfred

[hsmith]

Is cron running on your machine?

[manfredrotzetter]

Yes I think.

I have some corn config's for the log rotate /etc/cron.daily and this one are working properly.
Than I have this one: /etc/cron.d/nagioslogserver
On this I have to jobs and i added a 3th one:

# /etc/cron.d/nagioslogserver: crontab fragment for nagioslogserver
* * * * * nagios /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
* * * * * nagios /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
* * * * * nagios date >> /usr/local/nagioslogserver/var/nagioslogserver_cron.log 2>&1

Log file: /usr/local/nagioslogserver/var/nagioslogserver_cron.log
Wed May 25 09:44:01 CEST 2016
Wed May 25 09:45:01 CEST 2016
Wed May 25 09:46:02 CEST 2016
Wed May 25 09:47:01 CEST 2016
Wed May 25 09:48:01 CEST 2016
Wed May 25 09:49:01 CEST 2016
Wed May 25 09:50:01 CEST 2016

So I think that cron is working. But is it normal, that usr/local/nagioslogserver/var/jobs.log is empty (file modification date is current) ? Or better question, which job is doing the alerting stuff?

[tmcdonald]

Let's check permissions on those jobs and their logfiles, since they will help us troubleshoot once we can get them written to:

Code: Select all

ls-l /var/www/html/nagioslogserver/www/index.php 
ls -l /usr/local/nagioslogserver/var/poller.log
ls -l /usr/local/nagioslogserver/var/jobs.log
ls -ld /usr/local/nagioslogserver/var/

[manfredrotzetter]

The output of the 4 commands on the 4 nodes:

Code: Select all

[root@sbcc-log-2601 AD+tgdromar]# ls -l /var/www/html/nagioslogserver/www/index.php
-rw-r--r--. 1 root root 6517 May 10 08:21 /var/www/html/nagioslogserver/www/index.php
[root@sbcc-log-2601 AD+tgdromar]# ls -l /usr/local/nagioslogserver/var/poller.log
-rwxrwxr-x. 1 nagios nagios 290 May 25 18:47 /usr/local/nagioslogserver/var/poller.log
[root@sbcc-log-2601 AD+tgdromar]# ls -l /usr/local/nagioslogserver/var/jobs.log
-rwxrwxr-x. 1 nagios nagios 0 May 25 18:47 /usr/local/nagioslogserver/var/jobs.log
[root@sbcc-log-2601 AD+tgdromar]# ls -ld /usr/local/nagioslogserver/var/
drwxrwxr-x. 2 nagios nagios 4096 May 25 09:51 /usr/local/nagioslogserver/var/

[root@sbcc-log-2602 AD+tgdromar]# ls -l /var/www/html/nagioslogserver/www/index.php
-rw-r--r--. 1 root root 6517 May 10 08:18 /var/www/html/nagioslogserver/www/index.php
[root@sbcc-log-2602 AD+tgdromar]# ls -l /usr/local/nagioslogserver/var/poller.log
-rwxrwxr-x. 1 nagios nagios 290 May 25 18:48 /usr/local/nagioslogserver/var/poller.log
[root@sbcc-log-2602 AD+tgdromar]# ls -l /usr/local/nagioslogserver/var/jobs.log
-rwxrwxr-x. 1 nagios nagios 0 May 25 18:48 /usr/local/nagioslogserver/var/jobs.log
[root@sbcc-log-2602 AD+tgdromar]# ls -ld /usr/local/nagioslogserver/var/
drwxrwxr-x. 2 nagios nagios 4096 May 25 10:00 /usr/local/nagioslogserver/var/

[root@sbcc-log-2603 AD+tgdromar]# ls -l /var/www/html/nagioslogserver/www/index.php
-rw-r--r--. 1 root root 6517 May 10 08:16 /var/www/html/nagioslogserver/www/index.php
[root@sbcc-log-2603 AD+tgdromar]# ls -l /usr/local/nagioslogserver/var/poller.log
-rwxrwxr-x. 1 nagios nagios 290 May 25 18:49 /usr/local/nagioslogserver/var/poller.log
[root@sbcc-log-2603 AD+tgdromar]# ls -l /usr/local/nagioslogserver/var/jobs.log
-rwxrwxr-x. 1 nagios nagios 0 May 25 18:49 /usr/local/nagioslogserver/var/jobs.log
[root@sbcc-log-2603 AD+tgdromar]# ls -ld /usr/local/nagioslogserver/var/
drwxrwxr-x. 2 nagios nagios 4096 May 25 10:00 /usr/local/nagioslogserver/var/

[root@sbcc-log-2604 AD+tgdromar]# ls -l /var/www/html/nagioslogserver/www/index.php
-rw-r--r--. 1 root root 6517 May 10 08:14 /var/www/html/nagioslogserver/www/index.php
[root@sbcc-log-2604 AD+tgdromar]# ls -l /usr/local/nagioslogserver/var/poller.log
-rwxrwxr-x. 1 nagios nagios 290 May 25 18:50 /usr/local/nagioslogserver/var/poller.log
[root@sbcc-log-2604 AD+tgdromar]# ls -l /usr/local/nagioslogserver/var/jobs.log
-rwxrwxr-x. 1 nagios nagios 0 May 25 18:50 /usr/local/nagioslogserver/var/jobs.log
[root@sbcc-log-2604 AD+tgdromar]# ls -ld /usr/local/nagioslogserver/var/
drwxrwxr-x. 2 nagios nagios 4096 May 25 10:38 /usr/local/nagioslogserver/var/

[hsmith]

Can you do this for about 5 minutes, and let us know what the results are?

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log 

It's normal for this to be empty, because it gets purged every minute.

[manfredrotzetter]

No jobs

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.
Processed 0 node jobs.
Processed 0 global jobs.

[hsmith]

Can you go to Administration > System > Command Subsystem and hit the "Reset All Jobs" button, and let me know what happens?

[manfredrotzetter]

That's it. Now its working fine.

You can close this post.

Thanks a lot.

[mcapra]

Glad you were able to find a solution! Closing this up