Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

All host(s) & service(s) stuck in 'pending' status in v5.7.1

https://support.nagios.com/forum/viewtopic.php?t=59095

Page 1 of 1

All host(s) & service(s) stuck in 'pending' status in v5.7.1

Posted: Mon Jun 22, 2020 12:38 pm
by rrustamSAI
I am having an issue, similar to the one described in https://support.nagios.com/forum/viewto ... 16&t=58967

Re: All host(s) & service(s) stuck in 'pending' status in v5

Posted: Mon Jun 22, 2020 2:02 pm
by lmiltchev
Can you run the following commands from the command line and show the output in code wraps?

Code: Select all

service nagios restart
service nagios status
tail -50 /usr/local/nagios/var/nagios.log
grep broker /usr/local/nagios/etc/nagios.cfg
cat /usr/local/nagios/etc/ndo.cfg

Re: All host(s) & service(s) stuck in 'pending' status in v5

Posted: Mon Jun 22, 2020 2:31 pm
by rrustamSAI

Code: Select all

# date; service nagios restart
Mon Jun 22 14:27:14 CDT 2020
Stopping nagios: ......done.
Starting nagios: done.

Code: Select all

# date; service nagios status
Mon Jun 22 14:27:27 CDT 2020
nagios (pid 49437) is running...

Code: Select all

# date; tail -50 /usr/local/nagios/var/nagios.log
Mon Jun 22 14:27:49 CDT 2020
[1592854048] SERVICE DOWNTIME ALERT: faxmci01aprd;proc:uptime;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: faxmci01aprd;service:all_started;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: faxmci01aprd;service:nrpe;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd-vhost-sai;service:http;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd-vhost-sai;service:https;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd-vhost-sai;service:https-sslcert;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd-vhost;service:http;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd-vhost;service:https;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd-vhost;service:https-sslcert;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/boot;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/dev/shm;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/home;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/opt;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/opt/openv;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/srv/backup;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/tmp;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/usr;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/var;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/var/www;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/var/www/drupal7.saionline.com;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/var/www/eoffice.saionline.com;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/var/www/icleap.saionline.com;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/var/www/rightbridgeredirect.saionline.com;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/var/www/ssn.saionline.com;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/var/www/triad.saionline.com;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/var/www/websvc.saionline.com;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/var/www/www.welcome-fintegra.com;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:/var/www/www.welcome-osja.com;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;disk:fsck;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;file:/srv/home/zzz_heartbeat;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;file:/var/www/zzz_heartbeat;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;file:sync-cluster.err;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;memory:swap;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;proc:load;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;proc:time;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;proc:zombies;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;service:apache-status;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;service:cron;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;service:http;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;service:https;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;service:https-sslcert;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;service:local_smtp;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;service:nrpe;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewcmci01aprd;service:ssh;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewamci10aprd.sarep.dr;disk:c:;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewamci10aprd.sarep.dr;disk:other;STARTED; Service has entered a period of scheduled downtime
[1592854048] SERVICE DOWNTIME ALERT: ewamci10aprd.sarep.dr;memory:available;STARTED; Service has entered a period of scheduled downtime
[1592854055] SERVICE FLAPPING ALERT: dcmoma02bprd;proc:zombies;STOPPED; Service appears to have stopped flapping (3.9% change < 5.0% threshold)
[1592854057] SERVICE FLAPPING ALERT: emboma01prd;proc:zombies;STOPPED; Service appears to have stopped flapping (3.9% change < 5.0% threshold)

Code: Select all

# date; grep broker /usr/local/nagios/etc/nagios.cfg
Mon Jun 22 14:28:00 CDT 2020
# Commented out by NDO 'make install-broker-line' on Fri Jun 19 21:45:43 CDT 2020
# Commented out by NDO 'make install-broker-line' on Mon Jun 22 09:47:29 CDT 2020
# Commented out by NDO 'make install-broker-line' on Mon Jun 22 11:20:15 CDT 2020
###broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg
event_broker_options=-1
# Added by NDO 'make install-broker-line' on Fri Jun 19 21:45:43 CDT 2020
# Commented out by NDO 'make install-broker-line' on Mon Jun 22 09:47:29 CDT 2020
# Commented out by NDO 'make install-broker-line' on Mon Jun 22 11:20:15 CDT 2020
###broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
# Added by NDO 'make install-broker-line' on Mon Jun 22 09:47:29 CDT 2020
# Commented out by NDO 'make install-broker-line' on Mon Jun 22 11:20:15 CDT 2020
#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
# Added by NDO 'make install-broker-line' on Mon Jun 22 11:20:15 CDT 2020
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Code: Select all

# date;cat /usr/local/nagios/etc/ndo.cfg
Mon Jun 22 14:28:11 CDT 2020
# Default NDO config for Nagios XI

db_user=ndoutils
db_pass=n@gweb
db_name=nagios
db_host=localhost
db_port=3306
#db_socket=/var/lib/mysql.sock
db_max_reconnect_attempts=5

acknowledgement_data=1
comment_data=1
contact_status_data=1
downtime_data=1
event_handler_data=1
external_command_data=1
flapping_data=1
host_check_data=1
host_status_data=1
log_data=1
main_config_data=1
notification_data=1
object_config_data=1
process_data=1
program_status_data=1
retention_data=1
service_check_data=1
service_status_data=1
state_change_data=1
system_command_data=1
timed_event_data=1

config_output_options=2

max_object_insert_count=250
lmiltchev wrote:Can you run the following commands from the command line and show the output in code wraps?

Code: Select all

service nagios restart
service nagios status
tail -50 /usr/local/nagios/var/nagios.log
grep broker /usr/local/nagios/etc/nagios.cfg
cat /usr/local/nagios/etc/ndo.cfg

Re: All host(s) & service(s) stuck in 'pending' status in v5

Posted: Mon Jun 22, 2020 2:43 pm
by lmiltchev
The output looks normal - there is nothing weird that stands out. Do you see hosts/services in the GUI at all? If you do, are they still in a pending state (some of them or all of them)?

Can you PM me your latest profile?

Admin > System Profile > Download Profile

Re: All host(s) & service(s) stuck in 'pending' status in v5

Posted: Mon Jun 22, 2020 3:06 pm
by rrustamSAI
All hosts & services are still in pending status ... I attached the screenshoot.

I'll PM you the profile after this. -->>> I tried but my profile.zip is 50.9MB exceeded the 50MB limit <<<---
lmiltchev wrote:The output looks normal - there is nothing weird that stands out. Do you see hosts/services in the GUI at all? If you do, are they still in a pending state (some of them or all of them)?

Can you PM me your latest profile?

Admin > System Profile > Download Profile

Re: All host(s) & service(s) stuck in 'pending' status in v5

Posted: Mon Jun 22, 2020 3:42 pm
by rrustamSAI
Oddly now "/usr/local/nagios/var/nagios.log" stop growing (no additional entry) as of 3:01PM CDT:

Code: Select all

date; ls -l /usr/local/nagios/var/nagios.log 
Mon Jun 22 15:40:04 CDT 2020
-rw-r--r-- 1 nagios nagios 12965665 Jun 22 15:01 /usr/local/nagios/var/nagios.log

Code: Select all

$ date; tail -f nagios.log
Mon Jun 22 15:41:54 CDT 2020
[1592855906] SERVICE ALERT: rlpoma01gprd;file:sync-cluster.err;CRITICAL;SOFT;1;FILE_AGE CRITICAL: /var/log/sync-cluster.err is 5 seconds old and 60 bytes
[1592855948] SERVICE FLAPPING ALERT: mbxoma01aprd;service:all_started;STOPPED; Service appears to have stopped flapping (4.9% change < 5.0% threshold)
[1592855983] SERVICE ALERT: russiaapp.mci;file:/tmp/zzz_heartbeat;WARNING;SOFT;1;FILE_AGE WARNING: /tmp/zzz_heartbeat is 942 seconds old and 0 bytes
[1592855985] SERVICE ALERT: rptoma01dev;memory:available;WARNING;SOFT;1;CHECK_NRPE: Invalid packet version received from server.
[1592856008] SERVICE ALERT: rptoma02bprd;memory:available;OK;SOFT;3;CHECK_NRPE: Invalid packet version received from server.
[1592856027] SERVICE FLAPPING ALERT: wwwoma01bprd;service:smb;STOPPED; Service appears to have stopped flapping (4.0% change < 5.0% threshold)
[1592856027] SERVICE ALERT: rlpoma01gprd;file:sync-cluster.err;CRITICAL;SOFT;2;FILE_AGE CRITICAL: /var/log/sync-cluster.err is 4 seconds old and 60 bytes
[1592856027] SERVICE FLAPPING ALERT: wwwoma01bprd;disk:/var/www/www.saionline.com-443;STOPPED; Service appears to have stopped flapping (4.0% change < 5.0% threshold)
[1592856059] SERVICE ALERT: wdsoma01prd;memory:swap;WARNING;SOFT;1;CHECK_NRPE: Invalid packet version received from server.
[1592856061] SERVICE ALERT: sgdoma01dev;memory:available;CRITICAL;HARD;4;CHECK_NRPE: Invalid packet version received from server.
My guess, if I restart nagios process it'll proceed again.

Re: All host(s) & service(s) stuck in 'pending' status in v5

Posted: Mon Jun 22, 2020 4:02 pm
by lmiltchev
We rarely see such large profiles. Sorry for the inconvenience! I believe we need to move this out of the public forum. Please open a new support ticket via our support center here:

https://support.nagios.com/tickets/

Provide a shared download link to the profile.zip file in the ticket if possible. If this is not possible, you could split the profile.zip into smaller zip files, and send them in separate emails.

Thank you!
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited