Monitors Stuck Pending

[chicjo01]

Support,
On one of our Nagios servers, we have 8000+ monitors which remain in a pending state. If I manually force a check, it will respond and update the monitor. What can I do to get the pending monitors to start pulling and updating there status?

Nagios Version: 5.4.2

Capture.PNG

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.2.4
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 12-07-2016
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Warning: Duplicate definition found for service 'Windows Mssqldb Disk J' on host 'bctwdpocsql002.risk.regn.net' (config file '/usr/local/nagios/etc/services/windows+mssqldb+j:.cfg', starting on line 16)
Warning: Duplicate definition found for service 'Windows Mssqldb Disk Q' on host 'bctwdpocsql002.risk.regn.net' (config file '/usr/local/nagios/etc/services/windows+mssqldb+q:.cfg', starting on line 16)
Warning: Duplicate definition found for service 'Windows Mssqldb Disk K' on host 'bctwdpocsql002.risk.regn.net' (config file '/usr/local/nagios/etc/services/windows+mssqldb+k:+cluster_bctwdpocsql001_002.cfg', starting on line 16)
Warning: Duplicate definition found for service 'Windows Mssqldb Disk M' on host 'bctwdpocsql002.risk.regn.net' (config file '/usr/local/nagios/etc/services/windows+mssqldb+m:+cluster_bctwdpocsql001_002.cfg', starting on line 16)
Warning: Duplicate definition found for service 'Windows Mssqldb Disk I' on host 'bctwdpocsql002.risk.regn.net' (config file '/usr/local/nagios/etc/services/windows+mssqldb+i:.cfg', starting on line 16)
Warning: Duplicate definition found for service 'Windows Mssqldb Disk E' on host 'bctwdpocsql002.risk.regn.net' (config file '/usr/local/nagios/etc/services/windows+mssqldb+e:.cfg', starting on line 16)
Warning: Duplicate definition found for service 'Windows Mssqldb Disk E' on host 'bctwdpocsql001.risk.regn.net' (config file '/usr/local/nagios/etc/services/windows+mssqldb+e:+cluster_bctwdpocsql001_002.cfg', starting on line 16)
Warning: Duplicate definition found for service 'Windows Mssqldb Disk I' on host 'bctwdpocsql001.risk.regn.net' (config file '/usr/local/nagios/etc/services/windows+mssqldb+i:+cluster_bctwdpocsql001_002.cfg', starting on line 16)
Warning: Duplicate definition found for service 'Windows Mssqldb Disk M' on host 'bctwdpocsql001.risk.regn.net' (config file '/usr/local/nagios/etc/services/windows+mssqldb+m:.cfg', starting on line 16)
Warning: Duplicate definition found for service 'Windows Mssqldb Disk K' on host 'bctwdpocsql001.risk.regn.net' (config file '/usr/local/nagios/etc/services/windows+mssqldb+k:.cfg', starting on line 16)
Warning: Duplicate definition found for service 'Windows Mssqldb Disk Q' on host 'bctwdpocsql001.risk.regn.net' (config file '/usr/local/nagios/etc/services/windows+mssqldb+q:+cluster_bctwdpocsql001_002.cfg', starting on line 16)
Warning: Duplicate definition found for service 'Windows Mssqldb Disk J' on host 'bctwdpocsql001.risk.regn.net' (config file '/usr/local/nagios/etc/services/windows+mssqldb+j:+cluster_bctwdpocsql001_002.cfg', starting on line 16)
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 3960 services.
Checked 381 hosts.
Checked 6541 host groups.
Checked 2066 service groups.
Checked 239 contacts.
Checked 6545 contact groups.
Checked 149 commands.
Checked 9 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 381 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 9 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check

[npolovenko]

Hello, @chicjo01.

Can you run the following commands and let us know if that fixes the issue:

Code: Select all

service nagios restart
service crond restart

Also, are those pending services newly added? Are they using some particular check or different kind of checks?
Could you send in your Nagios XI System Profile so I can review it?
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and attach it to your next post. Or you may upload it to a cloud storage of your choice and share a link with me in pm.



*Profile was recieved

[chicjo01]

I have PM you the profile.zip file and I did a restart of the two services. The queue continues to show the same amount.

[npolovenko]

Let's increase a few limits in /etc/php.ini:

Code: Select all

max_execution_time = 90 ; Maximum execution time of each script, in seconds
max_input_time = 60 ; Maximum amount of time each script may spend parsing request data
memory_limit = 512M ; Maximum amount of memory a script may consume

After that please restart Nagios with:

Code: Select all

service nagios restart
service httpd restart

Also, can you let me know what kind of services are in a pending state? I know there are thousands of them, but maybe you could see some pattern?

[chicjo01]

I have made the adjustment to the /etc/php.ini file and restarted the services requested. The only pattern that I see is that every monitor has a duplicate, not sure if that helps.

nagios@~ ~]$ sudo service nagios restart
Restarting nagios (via systemctl): [ OK ]

[nagios@~ ~]$ sudo service httpd restart
Redirecting to /bin/systemctl restart httpd.service

[nagios@~ ~]$ grep -A 50 -B 5 max_execution /etc/php.ini
;;;;;;;;;;;;;;;;;;;

; Maximum execution time of each script, in seconds
; http://php.net/max-execution-time
; Note: This directive is hardcoded to 0 for the CLI SAPI
max_execution_time = 90

; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate unexpectedly
; long running scripts.
; Note: This directive is hardcoded to -1 for the CLI SAPI
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://php.net/max-input-time
max_input_time = 60

; Maximum input variable nesting level
; http://php.net/max-input-nesting-level
;max_input_nesting_level = 64

; How many GET/POST/COOKIE input variables may be accepted
; max_input_vars = 1000

; Maximum amount of memory a script may consume (128MB)
; http://php.net/memory-limit
memory_limit = 512M

Capture.PNG

[npolovenko]

@chicjo01, Can you see if the checks are also pending/not updating in nagios core interface? Change the link inside the web browser from http://ip_address/nagiosxi to http://ip_address/nagios/ You'll be prompted to sign in with admin credentials.
This article might help: https://support.nagios.com/kb/article.php?id=19

[chicjo01]

I think I found the problem, the duplicates monitors gave me something to look at. I checked the database for the nagios_hosts and nagios_services and wouldn't you know they had duplicate entries, but different instance ids. I removed all the entries that were not part of the first id and did a full apply configuration. Removed the configuration files, re-built them, verified the configuration file and restarted Nagios.

Need to wait a little bit to see if the issue continues and if the monitors start to act normally. Will let you know.

As for checking in Nagios Core, the problem was there as well.

[npolovenko]

@chicjo01, Sure, keep us updated. You may also run a ./repair_databases.sh script in /usr/local/nagiosxi/scripts/ This one might take a while to run due to amount of services on your XI.

[chicjo01]

The correction I made, looks to have solved the issue. Thanks, this can be closed.

[npolovenko]

Closing this thread.