Page 1 of 3
Strange recheck timing issue
Posted: Sun Feb 04, 2018 11:58 am
by askewdread
Hi,
I have noticed this a few times, but occasionally we are getting services that either hit the check limit or get less than the limit, but then for some reason it schedules the next check for months in the future... it sort of looks like it might be mixing up the month and day? a bit of an issue when your relying on the check. if you force a check it will then go and run it properly and schedule the next one correctly
Issue: - see the last check and next check
issue.PNG
Service config:
checks.PNG
Checks etc are all enabled..:
checklist.PNG
we are running NagiosXI version: 5.4.12 however I have seen this on older versions as well
Re: Strange recheck timing issue
Posted: Mon Feb 05, 2018 11:31 am
by kyang
Was your timezone changed recently?
Have you tried resetting the server?
Could you send me the output of these commands?
Code: Select all
ps -ef | grep nagios.cfg | grep -v grep
OR if you are using mariadb use this.
Thanks!
Re: Strange recheck timing issue
Posted: Tue Feb 06, 2018 11:50 am
by askewdread
not changed in the last year no...
I haven't restarted it yet incase there was something you guys need to get from it
Code: Select all
[root@dnzbsglnx10 ~]# ps -ef | grep nagios.cfg | grep -v grep
nagios 26894 1 2 Feb02 ? 02:25:10 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 26984 26894 0 Feb02 ? 00:00:15 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
all of those crashes are last year and have run a check and all are ok
Code: Select all
[root@dnzbsglnx11 ~]# tail -30 /var/log/mariadb/mariadb.log
171025 0:05:46 [ERROR] mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
171025 0:05:46 [ERROR] mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
171025 0:05:46 [ERROR] mysqld: Table './nagiosxi/xi_eventqueue' is marked as crashed and should be repaired
171025 0:05:46 [Warning] Checking table: './nagiosxi/xi_eventqueue'
171025 0:05:46 [Warning] Checking table: './nagios/nagios_hoststatus'
171025 0:05:47 [ERROR] mysqld: Table './nagios/nagios_programstatus' is marked as crashed and should be repaired
171025 0:05:47 [Warning] Checking table: './nagios/nagios_programstatus'
171025 0:05:47 [ERROR] mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
171025 0:05:47 [Warning] Checking table: './nagios/nagios_servicestatus'
171025 0:05:52 [ERROR] mysqld: Table './nagios/nagios_customvariablestatus' is marked as crashed and should be repaired
171025 0:05:52 [Warning] Checking table: './nagios/nagios_customvariablestatus'
171025 0:05:52 [ERROR] mysqld: Table './nagios/nagios_logentries' is marked as crashed and should be repaired
171025 0:05:52 [Warning] Checking table: './nagios/nagios_logentries'
171025 0:06:02 [ERROR] mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
171025 0:06:02 [ERROR] mysqld: Table './nagiosxi/xi_sysstat' is marked as crashed and should be repaired
171025 0:06:02 [ERROR] mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
171025 0:06:02 [ERROR] mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
171025 0:06:02 [Warning] Checking table: './nagiosxi/xi_meta'
171025 0:06:02 [ERROR] mysqld: Table './nagiosxi/xi_sysstat' is marked as crashed and should be repaired
171025 0:06:02 [Warning] Checking table: './nagiosxi/xi_events'
171025 0:06:02 [Warning] Checking table: './nagiosxi/xi_sysstat'
171025 0:06:36 [ERROR] mysqld: Table './nagios/nagios_systemcommands' is marked as crashed and should be repaired
171025 0:06:36 [Warning] Checking table: './nagios/nagios_systemcommands'
171025 0:06:36 [ERROR] mysqld: Table './nagios/nagios_eventhandlers' is marked as crashed and should be repaired
171025 0:06:36 [Warning] Checking table: './nagios/nagios_eventhandlers'
171025 0:06:37 [ERROR] mysqld: Table './nagios/nagios_statehistory' is marked as crashed and should be repaired
171025 0:06:37 [Warning] Checking table: './nagios/nagios_statehistory'
171025 0:07:27 [ERROR] mysqld: Table './nagiosxi/xi_auditlog' is marked as crashed and should be repaired
171025 0:07:27 [Warning] Checking table: './nagiosxi/xi_auditlog'
180131 11:05:52 [Warning] IP address '10.51.61.167' could not be resolved: Name or service not known
[root@dnzbsglnx11 ~]#
ive found another service with that error today as well.... it always seems to confuse the day and month
second1.PNG
Re: Strange recheck timing issue
Posted: Tue Feb 06, 2018 12:13 pm
by kyang
Could you give me the output of these commands? Thanks!
Code: Select all
echo 'select now();' | mysql -t -uroot -pnagiosxi mysql
Code: Select all
echo 'select @@system_time_zone;' | mysql -t -uroot -pnagiosxi mysql
Re: Strange recheck timing issue
Posted: Tue Feb 06, 2018 12:19 pm
by askewdread
not sure if it makes any difference but our database server is a separate server to our nagiosxi server
Code: Select all
[root@dnzbsglnx10 ~]# grep timezone /etc/php.ini
; Defines the default timezone used by the date functions
; http://php.net/date.timezone
date.timezone = Pacific/Auckland
Code: Select all
+---------------------+
| now() |
+---------------------+
| 2018-02-07 06:18:18 |
+---------------------+
Code: Select all
+--------------------+
| @@system_time_zone |
+--------------------+
| NZDT |
+--------------------+
[root@dnzbsglnx11 ~]#
Re: Strange recheck timing issue
Posted: Tue Feb 06, 2018 2:28 pm
by kyang
not sure if it makes any difference but our database server is a separate server to our nagiosxi server
Could you clarify what you mean by this?
Is your db offloaded?
Did you get those results from the same server?
Re: Strange recheck timing issue
Posted: Tue Feb 06, 2018 2:30 pm
by askewdread
kyang wrote:not sure if it makes any difference but our database server is a separate server to our nagiosxi server
Could you clarify what you mean by this?
Is your db offloaded?
Did you get those results from the same server?
yes db offloaded,
php command came from nagiosxi server, mariadb commands came from database server
Re: Strange recheck timing issue
Posted: Tue Feb 06, 2018 2:51 pm
by kyang
Thanks,
Could you run and post the
date command on your servers? Are they at the same time?
Let's also get a profile from you as well.
Nagios XI Profile --> On the XI Home Page click "Admin" > "System Profile" --> "Download Profile" button
Save the profile.zip file and upload it here or PM me.
If you receive a
PROFILE BUILD FAILED
Please follow this article,
https://support.nagios.com/kb/article.p ... ategory=44
After you PM the profile please update this thread so we can see that you have sent something, unless you post the profile on here. Thanks
UPDATE: Profile Received!
Re: Strange recheck timing issue
Posted: Tue Feb 06, 2018 2:57 pm
by askewdread
kyang wrote:Thanks,
Could you run and post the
date command on your servers? Are they at the same time?
Let's also get a profile from you as well.
Nagios XI Profile --> On the XI Home Page click "Admin" > "System Profile" --> "Download Profile" button
Save the profile.zip file and upload it here or PM me.
If you receive a
PROFILE BUILD FAILED
Please follow this article,
https://support.nagios.com/kb/article.p ... ategory=44
After you PM the profile please update this thread so we can see that you have sent something, unless you post the profile on here. Thanks
NagiosXI Server:
Code: Select all
[root@dnzbsglnx10 ~]# date
Wed Feb 7 08:53:49 NZDT 2018
DB Server:
Code: Select all
[root@dnzbsglnx11 ~]# date
Wed Feb 7 08:53:51 NZDT 2018
have pm'd the profile
Re: Strange recheck timing issue
Posted: Tue Feb 06, 2018 3:09 pm
by kyang
Thanks, could you also tell me the time on XI?
Timezone settings.
http://yourXIipAddress/nagiosxi/admin/globalconfig.php