Strange recheck timing issue

[askewdread]

Hi,

I have noticed this a few times, but occasionally we are getting services that either hit the check limit or get less than the limit, but then for some reason it schedules the next check for months in the future... it sort of looks like it might be mixing up the month and day? a bit of an issue when your relying on the check. if you force a check it will then go and run it properly and schedule the next one correctly

Issue: - see the last check and next check

issue.PNG

Service config:

checks.PNG

Checks etc are all enabled..:

checklist.PNG

we are running NagiosXI version: 5.4.12 however I have seen this on older versions as well

[kyang]

Was your timezone changed recently?

Have you tried resetting the server?

Could you send me the output of these commands?

Code: Select all

ps -ef | grep nagios.cfg | grep -v grep

Code: Select all

tail -30 /var/log/mysqld.log

OR if you are using mariadb use this.

Code: Select all

tail -30 /var/log/mariadb.log

Thanks!

[askewdread]

not changed in the last year no...

I haven't restarted it yet incase there was something you guys need to get from it

Code: Select all

[root@dnzbsglnx10 ~]# ps -ef | grep nagios.cfg | grep -v grep
nagios   26894     1  2 Feb02 ?        02:25:10 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   26984 26894  0 Feb02 ?        00:00:15 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

all of those crashes are last year and have run a check and all are ok

Code: Select all

[root@dnzbsglnx11 ~]# tail -30 /var/log/mariadb/mariadb.log
171025  0:05:46 [ERROR] mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
171025  0:05:46 [ERROR] mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
171025  0:05:46 [ERROR] mysqld: Table './nagiosxi/xi_eventqueue' is marked as crashed and should be repaired
171025  0:05:46 [Warning] Checking table:   './nagiosxi/xi_eventqueue'
171025  0:05:46 [Warning] Checking table:   './nagios/nagios_hoststatus'
171025  0:05:47 [ERROR] mysqld: Table './nagios/nagios_programstatus' is marked as crashed and should be repaired
171025  0:05:47 [Warning] Checking table:   './nagios/nagios_programstatus'
171025  0:05:47 [ERROR] mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
171025  0:05:47 [Warning] Checking table:   './nagios/nagios_servicestatus'
171025  0:05:52 [ERROR] mysqld: Table './nagios/nagios_customvariablestatus' is marked as crashed and should be repaired
171025  0:05:52 [Warning] Checking table:   './nagios/nagios_customvariablestatus'
171025  0:05:52 [ERROR] mysqld: Table './nagios/nagios_logentries' is marked as crashed and should be repaired
171025  0:05:52 [Warning] Checking table:   './nagios/nagios_logentries'
171025  0:06:02 [ERROR] mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
171025  0:06:02 [ERROR] mysqld: Table './nagiosxi/xi_sysstat' is marked as crashed and should be repaired
171025  0:06:02 [ERROR] mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
171025  0:06:02 [ERROR] mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
171025  0:06:02 [Warning] Checking table:   './nagiosxi/xi_meta'
171025  0:06:02 [ERROR] mysqld: Table './nagiosxi/xi_sysstat' is marked as crashed and should be repaired
171025  0:06:02 [Warning] Checking table:   './nagiosxi/xi_events'
171025  0:06:02 [Warning] Checking table:   './nagiosxi/xi_sysstat'
171025  0:06:36 [ERROR] mysqld: Table './nagios/nagios_systemcommands' is marked as crashed and should be repaired
171025  0:06:36 [Warning] Checking table:   './nagios/nagios_systemcommands'
171025  0:06:36 [ERROR] mysqld: Table './nagios/nagios_eventhandlers' is marked as crashed and should be repaired
171025  0:06:36 [Warning] Checking table:   './nagios/nagios_eventhandlers'
171025  0:06:37 [ERROR] mysqld: Table './nagios/nagios_statehistory' is marked as crashed and should be repaired
171025  0:06:37 [Warning] Checking table:   './nagios/nagios_statehistory'
171025  0:07:27 [ERROR] mysqld: Table './nagiosxi/xi_auditlog' is marked as crashed and should be repaired
171025  0:07:27 [Warning] Checking table:   './nagiosxi/xi_auditlog'
180131 11:05:52 [Warning] IP address '10.51.61.167' could not be resolved: Name or service not known
[root@dnzbsglnx11 ~]#

ive found another service with that error today as well.... it always seems to confuse the day and month

second1.PNG

[kyang]

Could you give me the output of these commands? Thanks!

Code: Select all

grep timezone /etc/php.ini

Code: Select all

echo 'select now();' | mysql -t -uroot -pnagiosxi mysql

Code: Select all

echo 'select @@system_time_zone;' | mysql -t -uroot -pnagiosxi mysql

[askewdread]

not sure if it makes any difference but our database server is a separate server to our nagiosxi server

Code: Select all

[root@dnzbsglnx10 ~]# grep timezone /etc/php.ini
; Defines the default timezone used by the date functions
; http://php.net/date.timezone
date.timezone = Pacific/Auckland

Code: Select all

+---------------------+
| now()               |
+---------------------+
| 2018-02-07 06:18:18 |
+---------------------+

Code: Select all

+--------------------+
| @@system_time_zone |
+--------------------+
| NZDT               |
+--------------------+
[root@dnzbsglnx11 ~]#

[kyang]

not sure if it makes any difference but our database server is a separate server to our nagiosxi server

Could you clarify what you mean by this?

Is your db offloaded?

Did you get those results from the same server?

[askewdread]

not sure if it makes any difference but our database server is a separate server to our nagiosxi server

Could you clarify what you mean by this?

Is your db offloaded?

Did you get those results from the same server?

yes db offloaded,

php command came from nagiosxi server, mariadb commands came from database server

[kyang]

Thanks,

Could you run and post the date command on your servers? Are they at the same time?

Code: Select all

date

Let's also get a profile from you as well.

Nagios XI Profile --> On the XI Home Page click "Admin" > "System Profile" --> "Download Profile" button
Save the profile.zip file and upload it here or PM me.

If you receive a PROFILE BUILD FAILED
Please follow this article,

https://support.nagios.com/kb/article.p ... ategory=44

After you PM the profile please update this thread so we can see that you have sent something, unless you post the profile on here. Thanks

UPDATE: Profile Received!

[askewdread]

Thanks,

Could you run and post the date command on your servers? Are they at the same time?

Code: Select all

date

Let's also get a profile from you as well.

Nagios XI Profile --> On the XI Home Page click "Admin" > "System Profile" --> "Download Profile" button
Save the profile.zip file and upload it here or PM me.

If you receive a PROFILE BUILD FAILED
Please follow this article,

https://support.nagios.com/kb/article.p ... ategory=44

After you PM the profile please update this thread so we can see that you have sent something, unless you post the profile on here. Thanks

NagiosXI Server:

Code: Select all

[root@dnzbsglnx10 ~]# date
Wed Feb  7 08:53:49 NZDT 2018

DB Server:

Code: Select all

[root@dnzbsglnx11 ~]# date
Wed Feb  7 08:53:51 NZDT 2018

have pm'd the profile

[kyang]

Thanks, could you also tell me the time on XI?

Timezone settings.

http://yourXIipAddress/nagiosxi/admin/globalconfig.php