Nagios Support Forum

Hi guys,

A huge number of services defined as active checks are stales.
...
[Mon Mar 18 11:23:08 2019] Warning: The results of service 'Running Processes' on host '1287000114_10_cucon-01' are stale by 0d 0h 0m 10s (threshold=0d 0h 4m 15s). I'm forcing an immediate check of the service.
[Mon Mar 18 11:23:08 2019] Warning: The results of service 'Running Processes' on host '1287000114_10_imp-01' are stale by 0d 0h 0m 6s (threshold=0d 0h 4m 15s). I'm forcing an immediate check of the service.
[Mon Mar 18 11:24:08 2019] Warning: The results of service 'Device Power Supply' on host '1287000012_00_ccme-01' are stale by 0d 0h 0m 1s (threshold=0d 0h 4m 15s). I'm forcing an immediate check of the service.
...

I was expecting these behaviors only on passive checks.

Our monitoring engine status seems ok as you can see below:

#############################################################
[root@nagios-01: /usr/local/nagios]# /usr/local/nagios/bin/nagiostats

Nagios Stats 4.4.3
Copyright (c) 2003-2008 Ethan Galstad (www.nagios.org)
Last Modified: 2019-01-15
License: GPL

CURRENT STATUS DATA
------------------------------------------------------
Status File: /usr/local/nagios/var/status.dat
Status File Age: 0d 0h 0m 6s
Status File Version: 4.4.3

Program Running Time: 1d 10h 25m 39s
Nagios PID: 29706

Total Services: 16832
Services Checked: 16832
Services Scheduled: 15491
Services Actively Checked: 16771
Services Passively Checked: 61
Total Service State Change: 0.000 / 28.680 / 0.017 %
Active Service Latency: 0.000 / 1.076 / 0.074 sec
Active Service Execution Time: 0.001 / 60.006 / 1.006 sec
Active Service State Change: 0.000 / 28.680 / 0.014 %
Active Services Last 1/5/15/60 min: 1153 / 7478 / 15148 / 16687
Passive Service Latency: 0.006 / 0.993 / 0.539 sec
Passive Service State Change: 0.000 / 25.200 / 0.677 %
Passive Services Last 1/5/15/60 min: 1 / 1 / 1 / 2
Services Ok/Warn/Unk/Crit: 16481 / 80 / 255 / 16
Services Flapping: 0
Services In Downtime: 0

Total Hosts: 2717
Hosts Checked: 2715
Hosts Scheduled: 2717
Hosts Actively Checked: 2717
Host Passively Checked: 0
Total Host State Change: 0.000 / 10.000 / 0.027 %
Active Host Latency: 0.000 / 1.057 / 0.020 sec
Active Host Execution Time: 0.000 / 5.003 / 0.164 sec
Active Host State Change: 0.000 / 10.000 / 0.027 %
Active Hosts Last 1/5/15/60 min: 688 / 2647 / 2715 / 2715
Passive Host Latency: 0.000 / 0.000 / 0.000 sec
Passive Host State Change: 0.000 / 0.000 / 0.000 %
Passive Hosts Last 1/5/15/60 min: 0 / 0 / 0 / 0
Hosts Up/Down/Unreach: 2677 / 23 / 17
Hosts Flapping: 0
Hosts In Downtime: 0

Active Host Checks Last 1/5/15 min: 746 / 2786 / 8352
Scheduled: 739 / 2746 / 8212
On-demand: 7 / 40 / 140
Parallel: 739 / 2746 / 8212
Serial: 0 / 0 / 0
Cached: 7 / 40 / 140
Passive Host Checks Last 1/5/15 min: 0 / 0 / 0
Active Service Checks Last 1/5/15 min: 1377 / 7644 / 22825
Scheduled: 1377 / 7644 / 22825
On-demand: 0 / 0 / 0
Cached: 0 / 0 / 0
Passive Service Checks Last 1/5/15 min: 1 / 1 / 1

External Commands Last 1/5/15 min: 2 / 2 / 2
############################################################################

any idea?
We are running Nagios XI 5.5.10

B.Regards

Please PM me a copy of your profile, you can download it from Admin > System Profile > Download Profile.

Additionally, please include the output of these commands:
Thank you!

Hi,
Thanks for your quick response.

My profile file:

(RECEIVED)

The results of the commands:

[root@nagios-01: ~]# date
Tue Mar 19 15:47:57 CET 2019

[root@nagios-01: ~]# ls -l /etc/localtime
lrwxrwxrwx 1 root root 35 Jul 7 2016 /etc/localtime -> /usr/share/zoneinfo/Europe/Brussels

[root@nagios-01: ~]# php -r 'echo date("D M j G:i:s T Y")."\n";'
Tue Mar 19 15:48:44 CET 2019

[root@nagios-01: ~]# grep "date.timezone =" /etc/php.ini
date.timezone = Europe/Brussels

[root@nagios-01: ~]# echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagiosxi --table
+--------------------------------------------+------------+
| Table | Size in MB |
+--------------------------------------------+------------+
| nagios_acknowledgements | 0.36 |
| nagios_commands | 0.02 |
| nagios_commenthistory | 22.36 |
| nagios_comments | 0.15 |
| nagios_configfiles | 0.00 |
| nagios_configfilevariables | 0.01 |
| nagios_conninfo | 0.65 |
| nagios_contact_addresses | 0.00 |
| nagios_contact_notificationcommands | 0.04 |
| nagios_contactgroup_members | 0.01 |
| nagios_contactgroups | 0.00 |
| nagios_contactnotificationmethods | 0.10 |
| nagios_contactnotifications | 0.11 |
| nagios_contacts | 0.01 |
| nagios_contactstatus | 0.01 |
| nagios_customvariables | 1.70 |
| nagios_customvariablestatus | 1.86 |
| nagios_dbversion | 0.00 |
| nagios_downtimehistory | 1.16 |
| nagios_eventhandlers | 0.01 |
| nagios_externalcommands | 0.41 |
| nagios_flappinghistory | 3.13 |
| nagios_host_contactgroups | 0.11 |
| nagios_host_contacts | 0.03 |
| nagios_host_parenthosts | 0.15 |
| nagios_hostchecks | 0.00 |
| nagios_hostdependencies | 0.00 |
| nagios_hostescalation_contactgroups | 0.00 |
| nagios_hostescalation_contacts | 0.00 |
| nagios_hostescalations | 0.00 |
| nagios_hostgroup_members | 0.41 |
| nagios_hostgroups | 0.04 |
| nagios_hosts | 0.54 |
| nagios_hoststatus | 1.28 |
| nagios_instances | 0.00 |
| nagios_logentries | 1044.52 |
| nagios_notifications | 0.68 |
| nagios_objects | 6.37 |
| nagios_processevents | 0.32 |
| nagios_programstatus | 0.00 |
| nagios_runtimevariables | 0.00 |
| nagios_scheduleddowntime | 0.06 |
| nagios_service_contactgroups | 0.64 |
| nagios_service_contacts | 0.16 |
| nagios_service_parentservices | 0.00 |
| nagios_servicechecks | 0.00 |
| nagios_servicedependencies | 0.00 |
| nagios_serviceescalation_contactgroups | 0.00 |
| nagios_serviceescalation_contacts | 0.00 |
| nagios_serviceescalations | 0.00 |
| nagios_servicegroup_members | 0.00 |
| nagios_servicegroups | 0.00 |
| nagios_services | 4.20 |
| nagios_servicestatus | 8.14 |
| nagios_statehistory | 762.52 |
| nagios_systemcommands | 0.03 |
| nagios_timedeventqueue | 0.00 |
| nagios_timedevents | 0.00 |
| nagios_timeperiod_timeranges | 0.03 |
| nagios_timeperiods | 0.01 |
| tbl_command | 0.04 |
| tbl_contact | 0.02 |
| tbl_contactgroup | 0.01 |
| tbl_contacttemplate | 0.01 |
| tbl_domain | 0.01 |
| tbl_host | 0.51 |
| tbl_hostdependency | 0.00 |
| tbl_hostescalation | 0.00 |
| tbl_hostextinfo | 0.00 |
| tbl_hostgroup | 0.09 |
| tbl_hosttemplate | 0.02 |
| tbl_info | 0.13 |
| tbl_lnkContactToCommandHost | 0.00 |
| tbl_lnkContactToCommandService | 0.00 |
| tbl_lnkContactToContactgroup | 0.00 |
| tbl_lnkContactToContacttemplate | 0.00 |
| tbl_lnkContactToVariabledefinition | 0.00 |
| tbl_lnkContactgroupToContact | 0.00 |
| tbl_lnkContactgroupToContactgroup | 0.00 |
| tbl_lnkContacttemplateToCommandHost | 0.00 |
| tbl_lnkContacttemplateToCommandService | 0.00 |
| tbl_lnkContacttemplateToContactgroup | 0.00 |
| tbl_lnkContacttemplateToContacttemplate | 0.00 |
| tbl_lnkContacttemplateToVariabledefinition | 0.00 |
| tbl_lnkHostToContact | 0.00 |
| tbl_lnkHostToContactgroup | 0.00 |
| tbl_lnkHostToHost | 0.04 |
| tbl_lnkHostToHostgroup | 0.08 |
| tbl_lnkHostToHosttemplate | 0.21 |
| tbl_lnkHostToVariabledefinition | 0.31 |
| tbl_lnkHostdependencyToHost_DH | 0.00 |
| tbl_lnkHostdependencyToHost_H | 0.00 |
| tbl_lnkHostdependencyToHostgroup_DH | 0.00 |
| tbl_lnkHostdependencyToHostgroup_H | 0.00 |
| tbl_lnkHostescalationToContact | 0.00 |
| tbl_lnkHostescalationToContactgroup | 0.00 |
| tbl_lnkHostescalationToHost | 0.00 |
| tbl_lnkHostescalationToHostgroup | 0.00 |
| tbl_lnkHostgroupToHost | 0.00 |
| tbl_lnkHostgroupToHostgroup | 0.02 |
| tbl_lnkHosttemplateToContact | 0.00 |
| tbl_lnkHosttemplateToContactgroup | 0.00 |
| tbl_lnkHosttemplateToHost | 0.00 |
| tbl_lnkHosttemplateToHostgroup | 0.00 |
| tbl_lnkHosttemplateToHosttemplate | 0.01 |
| tbl_lnkHosttemplateToVariabledefinition | 0.00 |
| tbl_lnkServiceToContact | 0.01 |
| tbl_lnkServiceToContactgroup | 0.00 |
| tbl_lnkServiceToHost | 0.01 |
| tbl_lnkServiceToHostgroup | 0.01 |
| tbl_lnkServiceToServicegroup | 0.00 |
| tbl_lnkServiceToServicetemplate | 0.02 |
| tbl_lnkServiceToVariabledefinition | 0.01 |
| tbl_lnkServicedependencyToHost_DH | 0.00 |
| tbl_lnkServicedependencyToHost_H | 0.00 |
| tbl_lnkServicedependencyToHostgroup_DH | 0.00 |
| tbl_lnkServicedependencyToHostgroup_H | 0.00 |
| tbl_lnkServicedependencyToService_DS | 0.00 |
| tbl_lnkServicedependencyToService_S | 0.00 |
| tbl_lnkServiceescalationToContact | 0.00 |
| tbl_lnkServiceescalationToContactgroup | 0.00 |
| tbl_lnkServiceescalationToHost | 0.00 |
| tbl_lnkServiceescalationToHostgroup | 0.00 |
| tbl_lnkServiceescalationToService | 0.00 |
| tbl_lnkServicegroupToService | 0.00 |
| tbl_lnkServicegroupToServicegroup | 0.00 |
| tbl_lnkServicetemplateToContact | 0.00 |
| tbl_lnkServicetemplateToContactgroup | 0.00 |
| tbl_lnkServicetemplateToHost | 0.00 |
| tbl_lnkServicetemplateToHostgroup | 0.00 |
| tbl_lnkServicetemplateToServicegroup | 0.00 |
| tbl_lnkServicetemplateToServicetemplate | 0.01 |
| tbl_lnkServicetemplateToVariabledefinition | 0.00 |
| tbl_lnkTimeperiodToTimeperiod | 0.00 |
| tbl_logbook | 0.02 |
| tbl_mainmenu | 0.00 |
| tbl_permission | 0.02 |
| tbl_permission_inactive | 0.02 |
| tbl_service | 0.09 |
| tbl_servicedependency | 0.00 |
| tbl_serviceescalation | 0.00 |
| tbl_serviceextinfo | 0.00 |
| tbl_servicegroup | 0.01 |
| tbl_servicetemplate | 0.02 |
| tbl_session | 0.00 |
| tbl_session_locks | 0.00 |
| tbl_settings | 0.00 |
| tbl_submenu | 0.00 |
| tbl_timedefinition | 0.03 |
| tbl_timeperiod | 0.01 |
| tbl_user | 0.01 |
| tbl_variabledefinition | 0.68 |
| xi_auditlog | 0.27 |
| xi_auth_tokens | 1.61 |
| xi_cmp_trapdata | 0.03 |
| xi_cmp_trapdata_log | 0.03 |
| xi_commands | 0.00 |
| xi_eventqueue | 3.07 |
| xi_events | 0.12 |
| xi_incidents | 0.00 |
| xi_meta | 2.82 |
| xi_options | 0.08 |
| xi_sessions | 0.03 |
| xi_sysstat | 0.09 |
| xi_usermeta | 1.75 |
| xi_users | 0.02 |
+--------------------------------------------+------------+

[root@nagios-01: ~]# echo "SELECT relname as Table, pg_size_pretty(pg_total_relation_size(relid)) As Size, pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as ExternalSize FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;" | psql nagiosxi nagiosxi
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

This latter command is it correct?

Thanks you

I included that last command just in case, you aren't running postgresql so please ignore that message.

What is the output of this command?
Also, do you have any Global Event Handlers defined/enabled in Admin > Manage Components > Global Event Handlers > Settings on any of the tabs?

Hi,

[root@nagios-01: ~]# ls -l /usr/local/nagiosxi/scripts
total 1196
drwxr-x--- 3 nagios nagios 4096 Oct 29 11:27 automation
-rwxr-x--- 1 root nagios 7625 Feb 20 12:04 backup_xi.sh
-r-xr-x--- 1 nagios nagios 8165 Feb 20 12:04 ccm_delete_object.php
-r-xr-x--- 1 nagios nagios 1041 Feb 20 12:04 ccm_export.php
-r-xr-x--- 1 nagios nagios 1537 Feb 20 12:04 ccm_import.php
-r-xr-x--- 1 nagios nagios 3180 Feb 20 12:04 ccm_snapshot.sh
-rwxr-x--- 1 root nagios 1800 Feb 20 12:04 change_timezone.sh
-rw-r----- 1 nagios nagios 2839 Feb 20 12:04 contact_notification_handler.php
-rwxr-x--- 1 nagios nagios 660 May 14 2018 export_nagiosql.sh
-rwxr-x--- 1 nagios nagios 266 Feb 20 12:04 handle_nagioscore_event.php
-rw-r----- 1 nagios nagios 3903 Feb 20 12:04 handle_nagioscore.inc.php
-rwxr-x--- 1 nagios nagios 267 Feb 20 12:04 handle_nagioscore_notification.php
-rwxr-x--- 1 nagios nagios 425 May 14 2018 import_nagiosql.sh
-rwxr-x--- 1 nagios nagios 1244 Feb 20 12:04 import_xiconfig.php
-rwxr-x--- 1 nagios nagios 93 May 14 2018 kill_rrdtool.sh
-rwxr-x--- 1 root nagios 2634 Feb 20 12:04 manage_services.sh
-rwxr-x--- 1 root nagios 3815 Feb 20 12:04 manage_ssl_config.sh
-rwxrwx--- 1 nagios nagios 160 Oct 29 08:42 nagiosql.cookies
-rwxrwx--- 1 nagios nagios 283685 Feb 20 12:04 nagiosql_defaults.sql
-rwxrwx--- 1 nagios nagios 49297 Dec 8 2017 nagiosql.delete.contact
-rwxrwx--- 1 nagios nagios 895 May 14 2018 nagiosql_delete_contact.php
-rwxrwx--- 1 nagios nagios 66150 Dec 6 2017 nagiosql.delete.host
-rwxrwx--- 1 nagios nagios 1937 May 14 2018 nagiosql_delete_host.php
-rwxrwx--- 1 nagios nagios 497 May 14 2018 nagiosql_delete_object.sh
-rwxrwx--- 1 nagios nagios 43148 Jun 7 2016 nagiosql.delete.service
-rwxrwx--- 1 nagios nagios 2766 May 14 2018 nagiosql_delete_service.php
-rwxrwx--- 1 nagios nagios 51192 Dec 8 2017 nagiosql.delete.timeperiod
-rwxrwx--- 1 nagios nagios 896 May 14 2018 nagiosql_delete_timeperiod.php
-rwxrwx--- 1 nagios nagios 1272 May 14 2018 nagiosql_exportall.php
-rwxrwx--- 1 nagios nagios 21859 Oct 29 08:42 nagiosql.export.monitoring
-rwxrwx--- 1 nagios nagios 1296 May 14 2018 nagiosql_importall.php
-rwxrwx--- 1 nagios nagios 377712 Jun 27 2018 nagiosql.import.monitoring
-rwxrwx--- 1 nagios nagios 39860 Oct 29 08:42 nagiosql.login
-rwxrwx--- 1 nagios nagios 1681 May 14 2018 nagiosql_login.php
-rwxrwx--- 1 nagios nagios 2247 May 14 2018 nagiosql_snapshot.sh
-rwxrwx--- 1 nagios nagios 200 May 14 2018 nagiosql_trim_backups.sh
-rw-r----- 1 nagios nagios 366 May 14 2018 nagiosxi_dbtype.php
-rwxr-x--- 1 nagios nagios 506 Feb 20 12:04 nom_create_nagioscore_checkpoint_cond.sh
-rwxr-x--- 1 nagios nagios 986 Feb 20 12:04 nom_create_nagioscore_checkpoint.sh
-rwxr-x--- 1 nagios nagios 785 Feb 20 12:04 nom_create_nagioscore_errorpoint.sh
-rwxr-x--- 1 nagios nagios 968 Feb 20 12:04 nom_restore_nagioscore_checkpoint.sh
-rwxr-x--- 1 nagios nagios 947 Feb 20 12:04 nom_restore_nagioscore_checkpoint_specific.sh
-rwxr-x--- 1 nagios nagios 4627 Feb 20 12:04 nom_trim_checkpoints.sh
-rwxr-x--- 1 nagios nagios 2893 May 14 2018 nom_trim_nagioscore_checkpoints.sh
-rwxr-x--- 1 nagios nagios 5510 Feb 20 12:04 nxti_import.php
-rwxr-x--- 1 nagios nagios 1325 Feb 20 12:04 nxti.php
-rwxr-x--- 1 nagios nagios 12496 Feb 20 12:04 parse_core_eventlog.php
-rwxr-x--- 1 nagios nagios 6649 Feb 20 12:04 patch_ndoutils.php
-rwxr-x--- 1 nagios nagios 971 Feb 20 12:04 reconfigure_nagios.sh
-rwxr-x--- 1 nagios nagios 4224 Feb 20 12:04 remove_historical_data.sh
-rwxr-x--- 1 root nagios 1688 Feb 20 12:04 repair_databases.sh
-rwxr-x--- 1 nagios nagios 3948 Feb 20 12:04 repairmysql.sh
-rwxr-x--- 1 root nagios 3604 Feb 20 12:04 reset_config_perms.sh
-rwxr-x--- 1 nagios nagios 1859 Feb 20 12:04 reset_nagiosadmin_password.php
-rwxr-x--- 1 nagios nagios 1874 Feb 20 12:04 restart_nagios_with_export.sh
-rwxr-x--- 1 nagios nagios 4027 Feb 20 12:04 restore_defaults.sh
-rwxr-x--- 1 nagios nagios 12752 Feb 20 12:04 restore_xi.sh
-rwxr-x--- 1 nagios nagios 1812 Feb 20 12:04 send_to_auditlog.php
-rw-r----- 1 nagios nagios 2702 May 14 2018 uninstall_xi.sh
-rwxr-x--- 1 nagios nagios 1345 Feb 20 12:04 unlock_user_account.php
-rwxr-x--- 1 nagios nagios 722 Feb 20 12:04 update_check.php
-rwxr-x--- 1 root nagios 2914 Feb 20 12:04 upgrade_to_latest.sh


Yes, we have global event handle defined for host state change and service state change: B.Regards

Please PM me these files:

Hi,

Please find below the files we use to handle host and service state change: B.Regards

After reviewing your information, this is occurring because you have enabled check_freshness in a couple host templates and service templates, to disable this (which is what the default is set to), do this:

Go to Configure > Core Config Manager > Host Templates:
- Edit the generic-host host template
- Click the Check Settings tab
- Set Check freshness to Skip
- Save

Go to Configure > Core Config Manager > Service Templates:
- Edit the generic-service service template
- Click the Check Settings tab
- Set Check freshness to Skip
- Save
- Edit the xiwizard_generic_service service template
- Click the Check Settings tab
- Set Check freshness to Skip
- Save

Go to Configure > Core Config Manager > Services:
- Search for uccx-servers
- Edit these services: CPU Usage, Physical Memory Usage, Swap Usage, and Uptime
- Click the Check Settings tab
- Set Check freshness to Skip
- Save
- Do this for the rest (Physical Memory Usage, Swap Usage, and Uptime)

Now apply configuration and you should not receive those messages anymore.

One thing to note is that you really would only use check_freshness for passive checks to make sure that a passive result has been received in the last X minutes, since your checks look to be active checks you would not use that functionality for this, please see here for more info:

https://assets.nagios.com/downloads/nag ... hness.html

Let us know the results.

Hi,

Thanks for your investigations.
I have applied the changes. I will let you know the results.

B.Regards

Sounds good, we'll keep an eye out.