No data coming from the Fused CORE servers

[dlukinski]

What version of Nagios Core are the servers running?

4.3.4 (no data incoming from 2 x XI servers either)

[tgriep]

Can you run the following commands as root on the Fusion server and post the output?

Code: Select all

echo 'select * from polling_lock ;' |mysql -u fusion -pfusion fusion
df -h
df -i
ps -ef --cols=300

Then, run the following as root on the Fusion server and post the /tmp/varlog.tgz file so we can view the log files to see why the servers are not getting polled?

Code: Select all

tar cvfz /tmp/varlog.tgz /usr/local/nagiosfusion/var/log/*.log

Thanks

[dlukinski]

Can you run the following commands as root on the Fusion server and post the output?

Code: Select all

echo 'select * from polling_lock ;' |mysql -u fusion -pfusion fusion
df -h
df -i
ps -ef --cols=300

Then, run the following as root on the Fusion server and post the /tmp/varlog.tgz file so we can view the log files to see why the servers are not getting polled?

Code: Select all

tar cvfz /tmp/varlog.tgz /usr/local/nagiosfusion/var/log/*.log

Thanks

Here is the output and the archive attached

Code: Select all

[root@fikc-nagfsprod01 ~]# echo 'select * from polling_lock ;' |mysql -u fusion -pfusion fusion                                                                                                                polling_lock_id server_id       username        polling_started
1       4       nagiosadmin     2018-01-10 20:15:58
2       5       nagiosadmin     2018-01-10 20:15:58
3       8       nagiosadmin     2018-01-10 20:15:58
4       6       nagiosadmin     2018-01-10 20:15:58
5       3       nagiosadmin     2018-01-10 20:15:58
6       9       nagiosadmin     2018-01-10 20:15:58
7       7       nagiosadmin     2018-01-10 20:15:58
8       10      nagiosadmin     2018-01-10 20:15:58
9       2       admin   2018-01-10 20:15:58
10      1       admin   2018-01-10 20:15:58
[root@fikc-nagfsprod01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       57G  2.1G   52G   4% /
tmpfs                 939M     0  939M   0% /dev/shm
/dev/sda1             477M  122M  330M  27% /boot
[root@fikc-nagfsprod01 ~]# df -i
Filesystem            Inodes IUsed   IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
                     3739632 53985 3685647    2% /
tmpfs                 240134     1  240133    1% /dev/shm
/dev/sda1             128016    62  127954    1% /boot
[root@fikc-nagfsprod01 ~]# ps -ef --cols=300
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jan10 ?        00:00:00 /sbin/init
root         2     0  0 Jan10 ?        00:00:00 [kthreadd]
root         3     2  0 Jan10 ?        00:00:00 [migration/0]
root         4     2  0 Jan10 ?        00:00:00 [ksoftirqd/0]
root         5     2  0 Jan10 ?        00:00:00 [stopper/0]
root         6     2  0 Jan10 ?        00:00:00 [watchdog/0]
root         7     2  0 Jan10 ?        00:00:33 [events/0]
root         8     2  0 Jan10 ?        00:00:00 [events/0]
root         9     2  0 Jan10 ?        00:00:00 [events_long/0]
root        10     2  0 Jan10 ?        00:00:00 [events_power_ef]
root        11     2  0 Jan10 ?        00:00:00 [cgroup]
root        12     2  0 Jan10 ?        00:00:00 [khelper]
root        13     2  0 Jan10 ?        00:00:00 [netns]
root        14     2  0 Jan10 ?        00:00:00 [async/mgr]
root        15     2  0 Jan10 ?        00:00:00 [pm]
root        16     2  0 Jan10 ?        00:00:00 [sync_supers]
root        17     2  0 Jan10 ?        00:00:00 [bdi-default]
root        18     2  0 Jan10 ?        00:00:00 [kintegrityd/0]
root        19     2  0 Jan10 ?        00:00:03 [kblockd/0]
root        20     2  0 Jan10 ?        00:00:00 [kacpid]
root        21     2  0 Jan10 ?        00:00:00 [kacpi_notify]
root        22     2  0 Jan10 ?        00:00:00 [kacpi_hotplug]
root        23     2  0 Jan10 ?        00:00:00 [ata_aux]
root        24     2  0 Jan10 ?        00:00:00 [ata_sff/0]
root        25     2  0 Jan10 ?        00:00:00 [ksuspend_usbd]
root        26     2  0 Jan10 ?        00:00:00 [khubd]
root        27     2  0 Jan10 ?        00:00:00 [kseriod]
root        28     2  0 Jan10 ?        00:00:00 [md/0]
root        29     2  0 Jan10 ?        00:00:00 [md_misc/0]
root        30     2  0 Jan10 ?        00:00:00 [linkwatch]
root        33     2  0 Jan10 ?        00:00:00 [khungtaskd]
root        34     2  0 Jan10 ?        00:00:00 [kswapd0]
root        35     2  0 Jan10 ?        00:00:00 [ksmd]
root        36     2  0 Jan10 ?        00:00:06 [khugepaged]
root        37     2  0 Jan10 ?        00:00:00 [aio/0]
root        38     2  0 Jan10 ?        00:00:00 [crypto/0]
root        45     2  0 Jan10 ?        00:00:00 [kthrotld/0]
root        46     2  0 Jan10 ?        00:00:00 [pciehpd]
root        48     2  0 Jan10 ?        00:00:00 [kpsmoused]
root        49     2  0 Jan10 ?        00:00:00 [usbhid_resumer]
root        50     2  0 Jan10 ?        00:00:00 [deferwq]
root        82     2  0 Jan10 ?        00:00:00 [kdmremove]
root        83     2  0 Jan10 ?        00:00:00 [kstriped]
root       170     2  0 Jan10 ?        00:00:00 [ttm_swap]
root       281     2  0 Jan10 ?        00:00:00 [scsi_eh_0]
root       282     2  0 Jan10 ?        00:00:00 [scsi_eh_1]
root       287     2  0 Jan10 ?        00:00:01 [mpt_poll_0]
root       288     2  0 Jan10 ?        00:00:00 [mpt/0]
root       289     2  0 Jan10 ?        00:00:00 [scsi_eh_2]
root       393     2  0 Jan10 ?        00:00:00 [kdmflush]
root       395     2  0 Jan10 ?        00:00:00 [kdmflush]
root       413     2  0 Jan10 ?        00:00:07 [jbd2/dm-0-8]
root       414     2  0 Jan10 ?        00:00:00 [ext4-dio-unwrit]
root       489     1  0 Jan10 ?        00:00:00 /sbin/udevd -d
root       701     2  0 Jan10 ?        00:00:01 [vmmemctl]
root       807   489  0 Jan10 ?        00:00:00 /sbin/udevd -d
root       838     2  0 Jan10 ?        00:00:00 [jbd2/sda1-8]
root       839     2  0 Jan10 ?        00:00:00 [ext4-dio-unwrit]
root       872     2  0 Jan10 ?        00:00:00 [kauditd]
root      1071     2  0 Jan10 ?        00:00:01 [flush-253:0]
root      1147     1  0 Jan10 ?        00:00:01 auditd
root      1169     1  0 Jan10 ?        00:00:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
dbus      1184     1  0 Jan10 ?        00:00:00 dbus-daemon --system
root      1234     1  0 Jan10 ?        00:00:00 /usr/sbin/sshd
root      1245     1  0 Jan10 ?        00:00:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp       1256     1  0 Jan10 ?        00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root      1293     1  0 Jan10 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pi                                                                  d-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql     1416  1293  0 Jan10 ?        00:03:58 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log                                                                  /mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
root      1512     1  0 Jan10 ?        00:00:00 /usr/libexec/postfix/master
postfix   1521  1512  0 Jan10 ?        00:00:00 qmgr -l -t fifo -u
root      1525     1  0 Jan10 ?        00:00:02 /usr/sbin/httpd
root      1537     1  0 Jan10 ?        00:00:01 crond
root      1557     1  0 Jan10 tty1     00:00:00 /sbin/mingetty /dev/tty1
root      1559     1  0 Jan10 tty2     00:00:00 /sbin/mingetty /dev/tty2
root      1561     1  0 Jan10 tty3     00:00:00 /sbin/mingetty /dev/tty3
root      1563     1  0 Jan10 tty4     00:00:00 /sbin/mingetty /dev/tty4
root      1565     1  0 Jan10 tty5     00:00:00 /sbin/mingetty /dev/tty5
root      1567     1  0 Jan10 tty6     00:00:00 /sbin/mingetty /dev/tty6
apache    1568  1525  0 Jan10 ?        00:00:03 /usr/sbin/httpd
apache    1569  1525  0 Jan10 ?        00:00:03 /usr/sbin/httpd
apache    1570  1525  0 Jan10 ?        00:00:03 /usr/sbin/httpd
apache    1571  1525  0 Jan10 ?        00:00:03 /usr/sbin/httpd
apache    1572  1525  0 Jan10 ?        00:00:03 /usr/sbin/httpd
apache    1573  1525  0 Jan10 ?        00:00:03 /usr/sbin/httpd
apache    1574  1525  0 Jan10 ?        00:00:03 /usr/sbin/httpd
apache    1575  1525  0 Jan10 ?        00:00:03 /usr/sbin/httpd
apache    1657  1525  0 Jan10 ?        00:00:03 /usr/sbin/httpd
apache    1660  1525  0 Jan10 ?        00:00:03 /usr/sbin/httpd
apache    7391  1525  0 13:50 ?        00:00:02 /usr/sbin/httpd
postfix  11826  1512  0 18:38 ?        00:00:00 pickup -l -t fifo -u
postfix  18466  1512  0 19:30 ?        00:00:00 cleanup -z -t unix -u
postfix  18468  1512  0 19:30 ?        00:00:00 local -t unix
root     18492  1234  0 19:30 ?        00:00:00 sshd: root@pts/0
root     18512 18492  0 19:30 pts/0    00:00:00 -bash
root     18565  1537  0 19:31 ?        00:00:00 CROND
root     18566  1537  0 19:31 ?        00:00:00 CROND
root     18567  1537  0 19:31 ?        00:00:00 CROND
root     18568  1537  0 19:31 ?        00:00:00 CROND
root     18569  1537  0 19:31 ?        00:00:00 CROND
nagios   18570 18569  0 19:31 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/auth_subsys.php --max-time=60 >>/usr/                                                                  local/nagiosfusion/var/log/auth_subsys.log 2>&1
nagios   18571 18567  0 19:31 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/log_subsys.php --max-time=60 >>/usr/l                                                                  ocal/nagiosfusion/var/log/log_subsys.log 2>&1
nagios   18572 18566  0 19:31 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/poll_subsys.php --max-time=60 --maste                                                                  r-poll >>/usr/local/nagiosfusion/var/log/poll_subsys.log 2>&1
nagios   18573 18568  0 19:31 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/cmd_subsys.php --max-time=60 >>/usr/l                                                                  ocal/nagiosfusion/var/log/cmd_subsys.log 2>&1
nagios   18574 18565  0 19:31 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/sysstat_subsys.php --max-time=60 >>/u                                                                  sr/local/nagiosfusion/var/log/sysstat_subsys.log 2>&1
nagios   18575 18571  0 19:31 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/log_subsys.php --max-time=60
nagios   18576 18572  0 19:31 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/poll_subsys.php --max-time=60 --master-poll
nagios   18577 18573  0 19:31 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/cmd_subsys.php --max-time=60
nagios   18578 18574  0 19:31 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/sysstat_subsys.php --max-time=60
nagios   18579 18570  0 19:31 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosfusion/cron/auth_subsys.php --max-time=60
root     18641 18512  0 19:31 pts/0    00:00:00 ps -ef --cols=300
[root@fikc-nagfsprod01 ~]# tar cvfz /tmp/varlog.tgz /usr/local/nagiosfusion/var/log/*.log
tar: Removing leading `/' from member names
/usr/local/nagiosfusion/var/log/auth_subsys.log
tar: /usr/local/nagiosfusion/var/log/auth_subsys.log: file changed as we read it
/usr/local/nagiosfusion/var/log/cmd_subsys.log
/usr/local/nagiosfusion/var/log/dbmaint_subsys.log
/usr/local/nagiosfusion/var/log/log_subsys.log
/usr/local/nagiosfusion/var/log/poll_subsys.10.nagiosadmin.log
/usr/local/nagiosfusion/var/log/poll_subsys.1.admin.log
/usr/local/nagiosfusion/var/log/poll_subsys.2.admin.log
/usr/local/nagiosfusion/var/log/poll_subsys.3.nagiosadmin.log
/usr/local/nagiosfusion/var/log/poll_subsys.4.nagiosadmin.log
/usr/local/nagiosfusion/var/log/poll_subsys.5.nagiosadmin.log
/usr/local/nagiosfusion/var/log/poll_subsys.6.nagiosadmin.log
/usr/local/nagiosfusion/var/log/poll_subsys.7.nagiosadmin.log
/usr/local/nagiosfusion/var/log/poll_subsys.8.nagiosadmin.log
/usr/local/nagiosfusion/var/log/poll_subsys.9.nagiosadmin.log
/usr/local/nagiosfusion/var/log/poll_subsys.log
/usr/local/nagiosfusion/var/log/sysstat_subsys.log
[root@fikc-nagfsprod01 ~]#

[tgriep]

I found this message in one of the log files.
PHP Fatal error: Allowed memory size of 262144 bytes exhausted (tried to allocate 122880 bytes) in /usr/local/nagiosfusion/html/includes/classes/jsonservice.inc.php on line 278

To fix this, login the Fusion GUI and go to the Admin > System Settings menu and increase the Polling Subsystem Memory Limit: to 1024M and save it.
Then edit the /etc/php.ini file and find the memory_limit option and change it to the following

Code: Select all

memory_limit = 1024M

Save the file and restart apache by running

Code: Select all

service httpd restart

See if that fixes the issue.

[dlukinski]

I found this message in one of the log files.
PHP Fatal error: Allowed memory size of 262144 bytes exhausted (tried to allocate 122880 bytes) in /usr/local/nagiosfusion/html/includes/classes/jsonservice.inc.php on line 278

To fix this, login the Fusion GUI and go to the Admin > System Settings menu and increase the Polling Subsystem Memory Limit: to 1024M and save it.
Then edit the /etc/php.ini file and find the memory_limit option and change it to the following

Code: Select all

memory_limit = 1024M

Save the file and restart apache by running

Code: Select all

service httpd restart

See if that fixes the issue.

Done, still no data incoming. Fresh logs attached

[tgriep]

Edit the Core server settins and in the CGI Path: field, fill in the following

Code: Select all

/cgi-bin/

Save the settings and try testing the fusion settings by clicking on the Test Fusion Settings button.

If it tests OK, go to the System Settings menu and imcrease the Polling interval to 3600 seconds, save it and see if that works.

The GGI Path: field has to be filled in for all Core servers.

[dlukinski]

Edit the Core server settins and in the CGI Path: field, fill in the following

Code: Select all

/cgi-bin/

Save the settings and try testing the fusion settings by clicking on the Test Fusion Settings button.

If it tests OK, go to the System Settings menu and imcrease the Polling interval to 3600 seconds, save it and see if that works.

The GGI Path: field has to be filled in for all Core servers.

I do not understand where to edit and what to edit?

[npolovenko]

I think @tgriep meant the fused server settings page. In Fusion web interface please click on the Servers menu, and then on the edit button next to the fused Nagios Core server.

edit fused.png

And in the CGI-Path field paste the following:

Code: Select all

/cgi-bin/

Next, in the same menu click on Test Fusion Settings Button. Do the same for all other Nagios Core servers.

The second part would be to go to the Admin Menu in the web interface, then System Settings. Click on Data&Polling tab. In the Polling interval field put 3600. Click on Update Settings.
Also, if you could send us the screenshot of the whole Data&Polling page that would be very helpful.

[dlukinski]

I think @tgriep meant the fused server settings page. In Fusion web interface please click on the Servers menu, and then on the edit button next to the fused Nagios Core server.

edit fused.png

And in the CGI-Path field paste the following:

Code: Select all

/cgi-bin/

Next, in the same menu click on Test Fusion Settings Button. Do the same for all other Nagios Core servers.

The second part would be to go to the Admin Menu in the web interface, then System Settings. Click on Data&Polling tab. In the Polling interval field put 3600. Click on Update Settings.
Also, if you could send us the screenshot of the whole Data&Polling page that would be very helpful.

/cgi-bin/ was already there but updated anyways. All tested OK. NOTHING WORKS

! - I repeat, two XI servers polling does not work either - !
This is a terrible product we cannot start sing 2 month after purchasing !!!

[npolovenko]

@dlukinski, I think we should escalate this to a support ticket. Would you be able to open a ticket so that our techs could start a remote session? https://support.nagios.com/tickets/