automatic repair unavailable

[gwakem]

I logged into our Fusion 4.0.1 on rhel 7.4, and none of the data was coming up. The menus undfer several different areas said nothing had reported in yet, which was absolutely not the case. I should be seeing over 200 servers checking in.

I attempted to run /usr/local/nagiosfusion/scripts/repair_databases.sh and did not expect the results:

Code: Select all

13:12:16 [root@fusion /usr/local/nagiosfusion/scripts]# ./repair_databases.sh 

Nagios Fusion
Repair Databases
****************


AUTOMATIC REPAIR UNAVAILABLE!

PLEASE CONTACT NAGIOS SUPPORT STAFF IMMEDIATELY!

Stopping the database...

Stopping the database succeeded.

It is suggested you leave the database in this state until you contact Nagios Support

I stopped right there and here I am. What information can I provide to help you help me?

[tgriep]

Running the repair script on a Fusion server doesn't really need to be done as the database engine doesn't need to be repaired and that message is printed out.

There is a cron job that runs to poll the data from the remote server and that may not be running so login to the server as root and run the following

Code: Select all

service mysqld start
service crond restart

Give the system 10 to 15 minutes to poll the data and it it still is not working, run the following as root and post the output.

Code: Select all

ps -ef
tail -100 /usr/local/nagiosfusion/var/log/poll_subsys.log

Thanks

[gwakem]

I was able to verify that crond was running prior to me logging in and seeing the issue. I was also able to confirm that cron did not stop. After verifying the cron job was running, I am still seeing the "No servers have checked in with data yet. "

Code: Select all

15:13:45 [root@fusion ~]# ps -ef && tail -100 /usr/local/nagiosfusion/var/log/poll_subsys.log
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Nov04 ?        00:01:12 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root         2     0  0 Nov04 ?        00:00:00 [kthreadd]
root         3     2  0 Nov04 ?        00:00:03 [ksoftirqd/0]
root         5     2  0 Nov04 ?        00:00:00 [kworker/0:0H]
root         7     2  0 Nov04 ?        00:00:01 [migration/0]
root         8     2  0 Nov04 ?        00:00:00 [rcu_bh]
root         9     2  0 Nov04 ?        00:00:42 [rcu_sched]
root        10     2  0 Nov04 ?        00:00:01 [watchdog/0]
root        11     2  0 Nov04 ?        00:00:00 [watchdog/1]
root        12     2  0 Nov04 ?        00:00:01 [migration/1]
root        13     2  0 Nov04 ?        00:00:01 [ksoftirqd/1]
root        15     2  0 Nov04 ?        00:00:00 [kworker/1:0H]
root        16     2  0 Nov04 ?        00:00:00 [watchdog/2]
root        17     2  0 Nov04 ?        00:00:01 [migration/2]
root        18     2  0 Nov04 ?        00:00:01 [ksoftirqd/2]
root        20     2  0 Nov04 ?        00:00:00 [kworker/2:0H]
root        21     2  0 Nov04 ?        00:00:00 [watchdog/3]
root        22     2  0 Nov04 ?        00:00:01 [migration/3]
root        23     2  0 Nov04 ?        00:00:01 [ksoftirqd/3]
root        25     2  0 Nov04 ?        00:00:00 [kworker/3:0H]
root        27     2  0 Nov04 ?        00:00:00 [kdevtmpfs]
root        28     2  0 Nov04 ?        00:00:00 [netns]
root        29     2  0 Nov04 ?        00:00:00 [khungtaskd]
root        30     2  0 Nov04 ?        00:00:00 [writeback]
root        31     2  0 Nov04 ?        00:00:00 [kintegrityd]
root        32     2  0 Nov04 ?        00:00:00 [bioset]
root        33     2  0 Nov04 ?        00:00:00 [kblockd]
root        34     2  0 Nov04 ?        00:00:00 [md]
root        40     2  0 Nov04 ?        00:00:29 [kswapd0]
root        41     2  0 Nov04 ?        00:00:00 [ksmd]
root        42     2  0 Nov04 ?        00:00:01 [khugepaged]
root        43     2  0 Nov04 ?        00:00:00 [crypto]
root        51     2  0 Nov04 ?        00:00:00 [kthrotld]
root        53     2  0 Nov04 ?        00:00:00 [kmpath_rdacd]
root        54     2  0 Nov04 ?        00:00:00 [kpsmoused]
root        55     2  0 Nov04 ?        00:00:00 [ipv6_addrconf]
root        74     2  0 Nov04 ?        00:00:00 [deferwq]
root       110     2  0 Nov04 ?        00:00:01 [kauditd]
root       305     2  0 Nov04 ?        00:00:00 [ata_sff]
root       312     2  0 Nov04 ?        00:00:00 [ttm_swap]
root       318     2  0 Nov04 ?        00:00:00 [scsi_eh_0]
root       319     2  0 Nov04 ?        00:00:00 [scsi_tmf_0]
root       320     2  0 Nov04 ?        00:00:00 [scsi_eh_1]
root       321     2  0 Nov04 ?        00:00:00 [scsi_tmf_1]
root       324     2  0 Nov04 ?        00:00:00 [virtscsi-scan]
root       325     2  0 Nov04 ?        00:00:00 [scsi_eh_2]
root       326     2  0 Nov04 ?        00:00:00 [scsi_tmf_2]
root       353     2  0 Nov04 ?        00:00:02 [kworker/2:1H]
root       394     2  0 Nov04 ?        00:00:00 [kdmflush]
root       395     2  0 Nov04 ?        00:00:00 [bioset]
root       402     2  0 Nov04 ?        00:00:00 [kdmflush]
root       403     2  0 Nov04 ?        00:00:00 [bioset]
root       410     2  0 Nov04 ?        00:00:00 [kdmflush]
root       411     2  0 Nov04 ?        00:00:00 [bioset]
root       429     2  0 Nov04 ?        00:00:00 [bioset]
root       430     2  0 Nov04 ?        00:00:00 [xfsalloc]
root       431     2  0 Nov04 ?        00:00:00 [xfs_mru_cache]
root       432     2  0 Nov04 ?        00:00:00 [xfs-buf/dm-2]
root       433     2  0 Nov04 ?        00:00:00 [xfs-data/dm-2]
root       434     2  0 Nov04 ?        00:00:00 [xfs-conv/dm-2]
root       435     2  0 Nov04 ?        00:00:00 [xfs-cil/dm-2]
root       436     2  0 Nov04 ?        00:00:00 [xfs-reclaim/dm-]
root       437     2  0 Nov04 ?        00:00:00 [xfs-log/dm-2]
root       438     2  0 Nov04 ?        00:00:00 [xfs-eofblocks/d]
root       439     2  0 Nov04 ?        00:00:00 [xfsaild/dm-2]
root       468     2  0 Nov04 ?        00:00:00 [xfs-buf/dm-0]
root       469     2  0 Nov04 ?        00:00:00 [xfs-data/dm-0]
root       470     2  0 Nov04 ?        00:00:00 [xfs-conv/dm-0]
root       471     2  0 Nov04 ?        00:00:00 [xfs-cil/dm-0]
root       472     2  0 Nov04 ?        00:00:00 [xfs-reclaim/dm-]
root       473     2  0 Nov04 ?        00:00:00 [xfs-log/dm-0]
root       474     2  0 Nov04 ?        00:00:00 [xfs-eofblocks/d]
root       475     2  0 Nov04 ?        00:00:53 [xfsaild/dm-0]
root       525     1  0 Nov04 ?        00:00:20 /usr/lib/systemd/systemd-journald
root       542     2  0 Nov04 ?        00:00:03 [kworker/0:1H]
root       548     2  0 Nov04 ?        00:00:00 [rpciod]
root       549     2  0 Nov04 ?        00:00:00 [xprtiod]
root       552     1  0 Nov04 ?        00:00:00 /usr/sbin/lvmetad -f
root       565     1  0 Nov04 ?        00:00:00 /usr/lib/systemd/systemd-udevd
root       649     2  0 Nov04 ?        00:00:00 [xfs-buf/vda1]
root       650     2  0 Nov04 ?        00:00:00 [xfs-data/vda1]
root       651     2  0 Nov04 ?        00:00:00 [xfs-conv/vda1]
root       652     2  0 Nov04 ?        00:00:00 [xfs-cil/vda1]
root       654     2  0 Nov04 ?        00:00:00 [xfs-reclaim/vda]
root       655     2  0 Nov04 ?        00:00:00 [xfs-log/vda1]
root       656     2  0 Nov04 ?        00:00:00 [xfs-eofblocks/v]
root       657     2  0 Nov04 ?        00:00:00 [xfsaild/vda1]
root       665     2  0 Nov04 ?        00:00:00 [kdmflush]
root       666     2  0 Nov04 ?        00:00:00 [bioset]
root       668     2  0 Nov04 ?        00:00:00 [kdmflush]
root       669     2  0 Nov04 ?        00:00:00 [bioset]
root       671     2  0 Nov04 ?        00:00:00 [kdmflush]
root       672     2  0 Nov04 ?        00:00:00 [bioset]
root       673     2  0 Nov04 ?        00:00:00 [kdmflush]
root       675     2  0 Nov04 ?        00:00:00 [bioset]
root       676     2  0 Nov04 ?        00:00:00 [kdmflush]
root       677     2  0 Nov04 ?        00:00:00 [bioset]
root       701     2  0 Nov04 ?        00:00:00 [xfs-buf/dm-3]
root       702     2  0 Nov04 ?        00:00:00 [xfs-data/dm-3]
root       703     2  0 Nov04 ?        00:00:00 [xfs-conv/dm-3]
root       704     2  0 Nov04 ?        00:00:00 [xfs-cil/dm-3]
root       706     2  0 Nov04 ?        00:00:00 [xfs-reclaim/dm-]
root       708     2  0 Nov04 ?        00:00:00 [xfs-log/dm-3]
root       709     2  0 Nov04 ?        00:00:00 [xfs-eofblocks/d]
root       713     2  0 Nov04 ?        00:00:00 [xfs-buf/dm-5]
root       714     2  0 Nov04 ?        00:00:00 [xfs-data/dm-5]
root       716     2  0 Nov04 ?        00:00:00 [xfs-conv/dm-5]
root       717     2  0 Nov04 ?        00:00:00 [xfs-cil/dm-5]
root       718     2  0 Nov04 ?        00:00:00 [xfs-reclaim/dm-]
root       719     2  0 Nov04 ?        00:00:00 [xfs-log/dm-5]
root       720     2  0 Nov04 ?        00:00:00 [xfs-eofblocks/d]
root       721     2  0 Nov04 ?        00:00:00 [xfsaild/dm-3]
root       722     2  0 Nov04 ?        00:00:00 [xfs-buf/dm-6]
root       723     2  0 Nov04 ?        00:00:00 [xfs-data/dm-6]
root       724     2  0 Nov04 ?        00:00:00 [xfs-conv/dm-6]
root       725     2  0 Nov04 ?        00:00:00 [xfs-cil/dm-6]
root       726     2  0 Nov04 ?        00:00:00 [xfs-reclaim/dm-]
root       727     2  0 Nov04 ?        00:00:00 [xfs-log/dm-6]
root       728     2  0 Nov04 ?        00:00:00 [xfs-eofblocks/d]
root       730     2  0 Nov04 ?        00:00:00 [xfsaild/dm-5]
root       732     2  0 Nov04 ?        00:00:53 [xfsaild/dm-6]
root       734     2  0 Nov04 ?        00:00:00 [xfs-buf/dm-7]
root       735     2  0 Nov04 ?        00:00:00 [xfs-data/dm-7]
root       736     2  0 Nov04 ?        00:00:00 [xfs-conv/dm-7]
root       737     2  0 Nov04 ?        00:00:00 [xfs-cil/dm-7]
root       738     2  0 Nov04 ?        00:00:00 [xfs-reclaim/dm-]
root       739     2  0 Nov04 ?        00:00:00 [xfs-log/dm-7]
root       740     2  0 Nov04 ?        00:00:00 [xfs-eofblocks/d]
root       741     2  0 Nov04 ?        00:00:00 [xfsaild/dm-7]
root       749     2  0 Nov04 ?        00:00:00 [xfs-buf/dm-4]
root       750     2  0 Nov04 ?        00:00:00 [xfs-data/dm-4]
root       751     2  0 Nov04 ?        00:00:00 [xfs-conv/dm-4]
root       752     2  0 Nov04 ?        00:00:00 [xfs-cil/dm-4]
root       753     2  0 Nov04 ?        00:00:00 [xfs-reclaim/dm-]
root       754     2  0 Nov04 ?        00:00:00 [xfs-log/dm-4]
root       755     2  0 Nov04 ?        00:00:00 [xfs-eofblocks/d]
root       756     2  0 Nov04 ?        00:00:00 [xfsaild/dm-4]
root       787     1  0 Nov04 ?        00:00:00 /usr/sbin/rpc.idmapd
root       788     1  0 Nov04 ?        00:00:04 /sbin/auditd
root       810     1  0 Nov04 ?        00:00:00 /usr/bin/qemu-ga --method=virtio-serial --path=/dev/virtio-ports/org.qemu.guest_agent.0 --blacklist=g
root       811     1  0 Nov04 ?        00:00:00 /usr/sbin/smartd -n -q never
libstor+   819     1  0 Nov04 ?        00:00:00 /usr/bin/lsmd -d
dbus       822     1  0 Nov04 ?        00:01:10 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
rpc        826     1  0 Nov04 ?        00:00:00 /sbin/rpcbind -w
root       827     2  0 Nov04 ?        00:00:01 [kworker/3:1H]
chrony     829     1  0 Nov04 ?        00:00:19 /usr/sbin/chronyd
root       831     1  0 Nov04 ?        00:00:00 /usr/sbin/gssproxy -D
root       837     1  0 Nov04 ?        00:00:16 /usr/sbin/NetworkManager --no-daemon
root       839     1  0 Nov04 ?        00:00:17 /usr/sbin/rsyslogd -n
polkitd    840     1  0 Nov04 ?        00:00:22 /usr/lib/polkit-1/polkitd --no-debug
root       843     1  0 Nov04 ?        00:00:00 /usr/sbin/abrtd -d -s
root       845     1  0 Nov04 ?        00:00:01 /usr/bin/abrt-watch-log -F BUG: WARNING: at WARNING: CPU: INFO: possible recursive locking detected e
root       849     1  0 Nov04 ?        00:00:07 /usr/sbin/irqbalance --foreground
root       851     1  0 Nov04 ?        00:00:00 /usr/sbin/sssd -i -f
ovirtag+   863     1  0 Nov04 ?        00:11:53 /usr/bin/python /usr/share/ovirt-guest-agent/ovirt-guest-agent.py
root       864     1  0 Nov04 ?        00:00:00 /usr/sbin/rpc.gssd
nrpe       869     1  0 Nov04 ?        00:00:11 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
root       877   851  0 Nov04 ?        00:00:24 /usr/libexec/sssd/sssd_be --domain ingles-markets.com --uid 0 --gid 0 --debug-to-files
root       984     2  0 Nov04 ?        00:00:01 [kworker/1:1H]
root      1034   851  0 Nov04 ?        00:00:08 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --debug-to-files
root      1036   851  0 Nov04 ?        00:00:11 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --debug-to-files
root      1057     1  0 Nov04 ?        00:00:33 /usr/lib/systemd/systemd-logind
root      1111     1  0 Nov04 ?        00:00:00 /usr/sbin/oddjobd -n -p /var/run/oddjobd.pid -t 300
root      1112     1  0 Nov04 ?        00:00:26 /usr/bin/python -Es /usr/sbin/tuned -l -P
root      1114     1  0 Nov04 ?        00:00:00 /usr/sbin/sshd -D
root      1118     1  0 Nov04 ?        00:00:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid
rpcuser   1120     1  0 Nov04 ?        00:00:00 /usr/sbin/rpc.statd
root      1134     1  0 Nov04 ?        00:00:11 /usr/sbin/httpd -DFOREGROUND
root      1135     1  0 Nov04 ?        00:00:00 /usr/bin/rhsmcertd
root      1150     1  0 Nov04 ?        00:00:05 /usr/sbin/crond -n
root      1165     1  0 Nov04 ?        00:00:00 rhnsd
root      1179     1  0 Nov04 ?        00:00:00 /usr/sbin/rpc.mountd
root      1188     1  0 Nov04 ?        00:00:06 sendmail: accepting connections
root      1197     1  0 Nov04 tty1     00:00:00 /sbin/agetty --noclear tty1 linux
root      1237     2  0 Nov04 ?        00:00:00 [nfsd4_callbacks]
root      1238     2  0 Nov04 ?        00:00:00 [lockd]
smmsp     1249     1  0 Nov04 ?        00:00:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root      1269     2  0 Nov04 ?        00:00:00 [nfsd]
root      1270     2  0 Nov04 ?        00:00:00 [nfsd]
root      1271     2  0 Nov04 ?        00:00:00 [nfsd]
root      1272     2  0 Nov04 ?        00:00:00 [nfsd]
root      1273     2  0 Nov04 ?        00:00:00 [nfsd]
root      1274     2  0 Nov04 ?        00:00:00 [nfsd]
root      1275     2  0 Nov04 ?        00:00:00 [nfsd]
root      1276     2  0 Nov04 ?        00:00:00 [nfsd]
root      1414     1  0 Nov04 ?        00:00:00 /usr/bin/rubrik/bootstrap_agent_main --port 12800 --ssl_priv_key /etc/rubrik/keys/agent.pem --ssl_pub
root      1574     1  0 Nov04 ?        00:00:24 /usr/bin/rubrik/backup_agent_main --port 12801 --ssl_priv_key /etc/rubrik/keys/agent.pem --ssl_pub_ke
apache    5127  1134  0 08:55 ?        00:00:05 /usr/sbin/httpd -DFOREGROUND
apache    8444  1134  0 13:08 ?        00:00:02 /usr/sbin/httpd -DFOREGROUND
apache    8448  1134  0 13:08 ?        00:00:02 /usr/sbin/httpd -DFOREGROUND
apache    8451  1134  0 13:08 ?        00:00:02 /usr/sbin/httpd -DFOREGROUND
apache    8539  1134  0 13:08 ?        00:00:01 /usr/sbin/httpd -DFOREGROUND
apache    8766  1134  0 13:09 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
root     13752  1114  0 13:42 ?        00:00:00 sshd: gwakem [priv]
gwakem   13754 13752  0 13:42 ?        00:00:00 sshd: gwakem@pts/0
gwakem   13755 13754  0 13:42 pts/0    00:00:00 -bash
root     13807 13755  0 13:42 pts/0    00:00:00 su -
root     13832 13807  0 13:43 pts/0    00:00:00 -bash
apache   16731  1134  0 03:31 ?        00:00:05 /usr/sbin/httpd -DFOREGROUND
apache   16733  1134  0 03:31 ?        00:00:05 /usr/sbin/httpd -DFOREGROUND
apache   16734  1134  0 03:31 ?        00:00:04 /usr/sbin/httpd -DFOREGROUND
root     22705     2  0 14:49 ?        00:00:00 [kworker/u32:2]
root     23006     2  0 14:52 ?        00:00:00 [kworker/1:2]
root     23551     2  0 14:55 ?        00:00:00 [kworker/u32:1]
root     24231     2  0 15:00 ?        00:00:00 [kworker/1:0]
root     24251     2  0 15:00 ?        00:00:00 [kworker/0:0]
root     24347     2  0 15:01 ?        00:00:00 [kworker/3:1]
root     24744     2  0 15:03 ?        00:00:00 [kworker/2:1]
root     24902     2  0 15:05 ?        00:00:00 [kworker/0:2]
root     25207     2  0 15:07 ?        00:00:00 [kworker/3:0]
root     25375     2  0 15:08 ?        00:00:00 [kworker/2:0]
mysql    25608     1  0 15:10 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
mysql    25854 25608  0 15:10 ?        00:00:01 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --lo
root     26054     2  0 15:11 ?        00:00:00 [kworker/2:3]
apache   26879  1134  0 15:14 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
root     27048     2  0 15:14 ?        00:00:00 [kworker/0:1]
root     27152     2  0 15:15 ?        00:00:00 [kworker/3:2]
root     27330  1150  0 15:16 ?        00:00:00 /usr/sbin/CROND -n
root     27331  1150  0 15:16 ?        00:00:00 /usr/sbin/CROND -n
root     27332  1150  0 15:16 ?        00:00:00 /usr/sbin/CROND -n
root     27333  1150  0 15:16 ?        00:00:00 /usr/sbin/CROND -n
root     27334  1150  0 15:16 ?        00:00:00 /usr/sbin/CROND -n
nagios   27335 27331  0 15:16 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/poll_subsys.php --max-time=60 --master-poll >
nagios   27336 27330  0 15:16 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/sysstat_subsys.php --max-time=60 >>/usr/local
nagios   27337 27334  0 15:16 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/auth_subsys.php --max-time=60 >>/usr/local/na
nagios   27338 27333  0 15:16 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/cmd_subsys.php --max-time=60 >>/usr/local/nag
nagios   27339 27332  0 15:16 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/log_subsys.php --max-time=60 >>/usr/local/nag
nagios   27340 27335  0 15:16 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/poll_subsys.php --max-time=60 --master-poll
nagios   27341 27336  0 15:16 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/sysstat_subsys.php --max-time=60
nagios   27342 27337  0 15:16 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/auth_subsys.php --max-time=60
nagios   27343 27339  0 15:16 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/log_subsys.php --max-time=60
nagios   27344 27338  0 15:16 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/cmd_subsys.php --max-time=60
root     27465 13832  0 15:16 pts/0    00:00:00 ps -ef
2017-11-08 15:15:02[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:03[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:04[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:05[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:06[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:07[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:08[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:09[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:10[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:11[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:12[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:13[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:14[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:15[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:16[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:17[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:18[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:19[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:20[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:21[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:22[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:23[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:24[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:25[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:26[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:27[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:28[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:29[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:30[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:31[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:32[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:33[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:34[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:35[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:36[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:37[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:38[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:39[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:40[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:41[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:42[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:43[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:44[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:45[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:46[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:47[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:48[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:49[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:50[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:51[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:52[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:53[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:54[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:55[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:56[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:57[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:58[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:15:59[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:00[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:01[s: 0, u: 0] poll_master() called at 1510172161
2017-11-08 15:16:01[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:01[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:02[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:03[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:04[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:05[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:06[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:07[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:08[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:09[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:10[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:11[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:12[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:13[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:14[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:15[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:16[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:17[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:18[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:19[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:20[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:21[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:22[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:23[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:24[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:25[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:26[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:27[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:28[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:29[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:30[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:31[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:32[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:33[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:34[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:35[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:36[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:37[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:38[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-08 15:16:39[s: 0, u: 0] poll_master(): 2 pollers registered

[tgriep]

It log file, is shows 2 pollers are registered but I don't see they messages for when the poller is trying to connect.
Try restarting cron and see if the polling starts.

If not, run the following and post the /tmp/info.txt file.

Code: Select all

tail -n 50 /usr/local/nagiosfusion/var/log/*.log >/tmp/info.txt
echo 'select * from  servers;' |mysql -u fusion -pfusion fusion -t >>/tmp/info.txt
chage -l nagios >>/tmp/info.txt
df -h  >>/tmp/info.txt

Thanks

[gwakem]

I have PM'd the information. Sorry for the delay, I had to wait until my post timer reset.

[gwakem]

Quick (unrelated) side note: The recent 4.0.1 patch reverted the fix we had in place for local logging. I will fix on my side, but ouch:

Code: Select all

16:14:57 [root@fusion /usr/local/nagiosfusion/var/log]# ll
total 696M
drwxr-xr-x 2 nagios nagios  96K Nov  8 03:31 .
drwxr-xr-x 4 nagios nagios  114 May 18 10:25 ..
-rw-r--r-- 1 nagios nagios  74M Nov  8 16:14 auth_subsys.log
:::SNIP::::
-rw-r--r-- 1 nagios nagios    0 Oct 16 03:49 auth_subsys.log-20171014.gz-20171015.gz
-rw-r--r-- 1 nagios nagios    0 Oct 17 03:16 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz
-rw-r--r-- 1 nagios nagios    0 Oct 18 03:25 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz
-rw-r--r-- 1 nagios nagios    0 Oct 19 03:40 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz
-rw-r--r-- 1 nagios nagios    0 Oct 20 03:11 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz
-rw-r--r-- 1 nagios nagios    0 Oct 21 03:48 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz
-rw-r--r-- 1 nagios nagios    0 Oct 22 03:46 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz
-rw-r--r-- 1 nagios nagios    0 Oct 23 03:06 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz
-rw-r--r-- 1 nagios nagios    0 Oct 24 03:19 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz
-rw-r--r-- 1 nagios nagios    0 Oct 25 03:08 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz
-rw-r--r-- 1 nagios nagios    0 Oct 26 03:18 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz-20171025.gz
-rw-r--r-- 1 nagios nagios    0 Oct 27 03:49 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz-20171025.gz-20171026.gz
-rw-r--r-- 1 nagios nagios    0 Oct 28 03:26 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz-20171025.gz-20171026.gz-20171027.gz
-rw-r--r-- 1 nagios nagios    0 Oct 29 03:43 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz-20171025.gz-20171026.gz-20171027.gz-20171028.gz
-rw-r--r-- 1 nagios nagios    0 Oct 30 03:36 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz-20171025.gz-20171026.gz-20171027.gz-20171028.gz-20171029.gz
-rw-r--r-- 1 nagios nagios    0 Oct 31 03:26 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz-20171025.gz-20171026.gz-20171027.gz-20171028.gz-20171029.gz-20171030.gz
-rw-r--r-- 1 nagios nagios    0 Nov  1 03:15 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz-20171025.gz-20171026.gz-20171027.gz-20171028.gz-20171029.gz-20171030.gz-20171031.gz
-rw-r--r-- 1 nagios nagios    0 Nov  2 03:23 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz-20171025.gz-20171026.gz-20171027.gz-20171028.gz-20171029.gz-20171030.gz-20171031.gz-20171101.gz
-rw-r--r-- 1 nagios nagios 6.0M Oct 14 03:06 auth_subsys.log-20171014.gz-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz-20171025.gz-20171026.gz-20171027.gz-20171028.gz-20171029.gz-20171030.gz-20171031.gz-20171101.gz-20171102.gz
-rw-r--r-- 1 nagios nagios    0 Oct 17 03:16 auth_subsys.log-20171015.gz-20171016.gz
-rw-r--r-- 1 nagios nagios    0 Oct 18 03:25 auth_subsys.log-20171015.gz-20171016.gz-20171017.gz
-rw-r--r-- 1 nagios nagios    0 Oct 19 03:40 auth_subsys.log-20171015.gz-20171016.gz-20171017.gz-20171018.gz
-rw-r--r-- 1 nagios nagios    0 Oct 20 03:11 auth_subsys.log-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz
-rw-r--r-- 1 nagios nagios    0 Oct 21 03:48 auth_subsys.log-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz
-rw-r--r-- 1 nagios nagios    0 Oct 22 03:46 auth_subsys.log-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz
-rw-r--r-- 1 nagios nagios    0 Oct 23 03:06 auth_subsys.log-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz
-rw-r--r-- 1 nagios nagios    0 Oct 24 03:19 auth_subsys.log-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz
-rw-r--r-- 1 nagios nagios    0 Oct 25 03:08 auth_subsys.log-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz
-rw-r--r-- 1 nagios nagios    0 Oct 26 03:18 auth_subsys.log-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz-20171025.gz
-rw-r--r-- 1 nagios nagios    0 Oct 27 03:49 auth_subsys.log-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz-20171025.gz-20171026.gz
-rw-r--r-- 1 nagios nagios    0 Oct 28 03:26 auth_subsys.log-20171015.gz-20171016.gz-20171017.gz-20171018.gz-20171019.gz-20171020.gz-20171021.gz-20171022.gz-20171023.gz-20171024.gz-20171025.gz-20171026.gz-20171027.gz
:::SNIP:::
16:15:00 [root@fusion /usr/local/nagiosfusion/var/log]# ll |wc -l
823

[tgriep]

Can you login to the Fusion GUI and go to the Admin > System Settings menu.
Increase the Simultaneous Pollers: to 5 and update the settings.

With your custom log rotation, you wouold have to look at all of the following files and the archives to see when the system stopped polling the servers.
/usr/local/nagiosfusion/var/log/poll_subsys.log
The log entries were good at this point
2017-11-04 20:05:25

The log files would have to be looked at from that point to see why they stopped.

Also, did you restart cron?

[gwakem]

Sorry, yes I did restart cron. I will dive into the logs either tomorrow or tonight later and update accordingly.

[tgriep]

Hey, no problem. I was hoping to catch something in the other log files but only viewing the last 50 lines, nothing came to view.
The only thing I did see is some of the servers had authentication issues. It didn't look like all of them failed but you may want to check that as well.

[gwakem]

It's working now!
I don't understand what happened here, but I updated the server via yum, rebooted, same issue. I changed the pollers from 2 to 5, applied, and suddenly the poll_subsys.log exploded with connection attempts, failures (against all servers), and then it just started working... all failures cleared up for everything that wasn't offline. Not only did I get the interface back with all server data showing up, but it loads in 2 seconds instead of the previous 10+ seconds. I guess with 200+ servers being fused, upping the pollers was a good idea, since the performance increase was huge.

One thing though, finding what I needed in the logs was a nightmare. I had to gunzip a gzip within a gzip within a gzip before I could get to the actual log. and that was after I cleared out several hundred 0 byte gzipped log files. Please please please fix that logging issue so this doesnt happen on next update.

Thank you for your help with this, this post can be locked.