Cron Jobs arent running

[bosecorp]

Rebooted the server and after that Cron jobs are running

seems to be a mistmatch with what is configured in cron vs the actual file names

* * * * * /usr/bin/php -q /usr/local/nagiosfusion/cron/cmdsubsys.php > /usr/local/nagiosfusion/var/cmdsubsys.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosfusion/cron/eventman.php > /usr/local/nagiosfusion/var/eventman.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosfusion/cron/sysstat.php > /usr/local/nagiosfusion/var/sysstat.log 2>&1
*/5 * * * * /usr/bin/php -q /usr/local/nagiosfusion/cron/dbmaint.php > /usr/local/nagiosfusion/var/dbmaint.log 2>&1
*/5 * * * * /usr/bin/php -q /usr/local/nagiosfusion/cron/poller.php > /usr/local/nagiosfusion/var/poller.log 2>&1


# ls -l
total 168
-rwxr-xr-x. 1 nagios nagios 32955 Oct 6 09:49 auth_subsys.php
-rwxr-xr-x. 1 nagios nagios 19108 Oct 6 09:49 cmd_subsys.php
-rwxr-xr-x. 1 nagios nagios 5442 Oct 6 09:49 dbmaint_subsys.php
-rwxr-xr-x. 1 nagios nagios 2946 Oct 6 09:49 log_subsys.php
-rwxr-xr-x. 1 nagios nagios 97979 Oct 6 09:49 poll_subsys.php
-rwxr-xr-x. 1 nagios nagios 3073 Oct 6 09:49 sysstat_subsys.php

[npolovenko]

@bosecorp, Seems like the cron jobs did not get overridden during the upgrade process.

Code: Select all

#
# /etc/cron.d/nagiosfusion: crontab fragment for nagiosfusion
#

*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosfusion/cron/dbmaint_subsys.php >>/usr/local/nagiosfusion/var/log/dbmaint_subsys.log 2>&1

* * * * * nagios /usr/bin/php -q /usr/local/nagiosfusion/cron/auth_subsys.php --max-time=60 >>/usr/local/nagiosfusion/var/log/auth_subsys.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosfusion/cron/cmd_subsys.php --max-time=60 >>/usr/local/nagiosfusion/var/log/cmd_subsys.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosfusion/cron/log_subsys.php --max-time=60 >>/usr/local/nagiosfusion/var/log/log_subsys.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosfusion/cron/poll_subsys.php --max-time=60 --master-poll >>/usr/local/nagiosfusion/var/log/poll_subsys.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosfusion/cron/sysstat_subsys.php --max-time=60 >>/usr/local/nagiosfusion/var/log/sysstat_subsys.log 2>&1

But, before copying and pasting the correct paths, please execute the following command:

Code: Select all

curl https://assets.nagios.com/downloads/nagiosfusion/upgrade.sh | sh

This upgrade script should take care of the cron jobs for you. But if not, go ahead and replace the paths manually.
Let us know how it goes.

[bosecorp]

done that. the upgrade script worked.

but I still have no data

here are the logs

==> poll_subsys.log <==
2017-11-13 11:56:08[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-13 11:56:09[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-13 11:56:10[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-13 11:56:11[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-13 11:56:12[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-13 11:56:13[s: 0, u: 0] poll_master(): 2 pollers registered

==> log_subsys.log <==
PROCESSED SQL: UPDATE log SET processed = 1 WHERE log_id IN (117823961,117823962,117823963,117823964,117823965,117823966,117823967,117823968,117823969,117823970,117823971,117823972,117823973,117823974,117823975,117823976,117823977,117823978,117823979,117823980,117823981,117823982,117823983,117823984,117823985,117823986,117823987,117823988,117823989,117823990)

==> poll_subsys.log <==
2017-11-13 11:56:14[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-13 11:56:15[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-13 11:56:16[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-13 11:56:17[s: 0, u: 0] poll_master(): 2 pollers registered

==> sysstat_subsys.log <==
2017-11-13 11:56:17[s: 0, u: 0] LOAD: ( [load1] => 1.30, [load5] => 1.25, [load15] => 1.45 )
2017-11-13 11:56:17[s: 0, u: 0] MEMORY: ( [total] => 32012, [used] => 692, [free] => 2561, [shared] => 16, [buffers] => 28758, [cached] => 30882 )
2017-11-13 11:56:17[s: 0, u: 0] SWAP: ( [total] => 0, [used] => 0, [free] => 0 )

==> poll_subsys.log <==
2017-11-13 11:56:18[s: 0, u: 0] poll_master(): 2 pollers registered

==> log_subsys.log <==
PROCESSED SQL: UPDATE log SET processed = 1 WHERE log_id IN (117823991,117824000,117824001,117824002,117824003,117824004,117824005,117824006,117824007,117824008,117824009,117824010,117824011,117824012,117824013,117824014,117824015,117824016,117823992,117823993,117823994,117823995,117823996,117823997,117823998,117823999,117824017,117824018,117824019,117824020)

==> poll_subsys.log <==
2017-11-13 11:56:19[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-13 11:56:20[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-13 11:56:21[s: 0, u: 0] poll_master(): 2 pollers registered
2017-11-13 11:56:22[s: 0, u: 0] poll_master(): 2 pollers registered

==> sysstat_subsys.log <==
2017-11-13 11:56:22[s: 0, u: 0] IOSTAT: ( [user] => 16.12, [nice] =>


I also see this in the errro_log file

==> /var/log/httpd/error_log <==
[Mon Nov 13 12:13:20.038320 2017] [:error] [pid 980] [client 10.205.129.110:53709] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 78 bytes) in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 367, referer: http://usvanagiospfu2/nagiosfusion/admin/

[npolovenko]

@bosecorp, Did you restart crond after changing the cron file?

Code: Select all

service crond restart

after that please run:

Code: Select all

service crond status

*Wait a few minutes for data to start updating.

Also, please upload your httpd log from:

Code: Select all

 /var/log/httpd/error_log

Thanks.

[bosecorp]

here you go

root@usvanagiospxi3:(11-13 12:53): /root
# service crond restart
Redirecting to /bin/systemctl restart crond.service
root@usvanagiospxi3:(11-13 12:53): /root
# systemctl restart crond
root@usvanagiospxi3:(11-13 12:53): /root
# service crond status
Redirecting to /bin/systemctl status crond.service
â— crond.service - Command Scheduler
Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2017-11-13 12:54:11 EST; 6s ago
Main PID: 78740 (crond)
CGroup: /system.slice/crond.service
└─78740 /usr/sbin/crond -n

Nov 13 12:54:11 usvanagiospxi3 systemd[1]: Started Command Scheduler.
Nov 13 12:54:11 usvanagiospxi3 systemd[1]: Starting Command Scheduler...
Nov 13 12:54:11 usvanagiospxi3 crond[78740]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 87% if used.)
Nov 13 12:54:11 usvanagiospxi3 crond[78740]: (CRON) INFO (running with inotify support)
Nov 13 12:54:11 usvanagiospxi3 crond[78740]: (CRON) INFO (@reboot jobs will be run at computer's startup.)
root@usvanagiospxi3:(11-13 12:53): /root
# tail -f /var/log/httpd/error_log
[Mon Nov 13 03:36:02.113856 2017] [lbmethod_heartbeat:notice] [pid 32781] AH02282: No slotmem from mod_heartmonitor
[Mon Nov 13 03:36:02.144855 2017] [mpm_prefork:notice] [pid 32781] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Mon Nov 13 03:36:02.144867 2017] [core:notice] [pid 32781] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
YPBINDPROC_DOMAIN: Domain not bound
[Mon Nov 13 07:53:03.073114 2017] [:error] [pid 71901] [client 10.20.2.101:64374] PHP Warning: ldap_bind(): Unable to bind to server: Invalid credentials in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 714, referer: http://usvanagiospxi3/nagiosxi/login.ph ... f&noauth=1
YPBINDPROC_DOMAIN: Domain not bound
YPBINDPROC_DOMAIN: Domain not bound
[Mon Nov 13 12:16:37.203919 2017] [:error] [pid 56010] [client 10.144.3.120:4173] PHP Warning: base64_decode() expects parameter 1 to be string, array given in /usr/local/nagiosxi/html/includes/utilsl.inc.php on line 3429, referer: http://nagiosxi/nagiosxi/includes/compo ... /index.php
YPBINDPROC_DOMAIN: Domain not bound
YPBINDPROC_DOMAIN: Domain not bound
^C

[npolovenko]

@bosecorp Can you try to reconfigure one of the servers. In Fusion web interface go to configuration menu, check all the server settings, click on Test Fusion Settings as well. Save changes if necessary. The goal is to see whether something is wrong with the old server's configuration or with the Fusion itself. Perhaps you could even add a new server just for testing purposes.

[bosecorp]

Nothing...

tried remove one the fused servers, then adding the server again but the same issue

what I did notice is that when I try to see the logs from the GUI, i get this error in the error_log file


[Mon Nov 13 15:05:14.627797 2017] [:error] [pid 24840] [client 10.144.3.120:16025] PHP Warning: file_get_contents(/usr/local/nagiosxi/var/components/bpi.xml): failed to open stream: No such file or directory in /usr/local/nagiosxi/html/includes/components/nagiosbpi/functions/bpi_functions.php on line 370

[npolovenko]

@bosecorp, Are you running Fusion on the same server as Nagios XI? Can you run ps -ef?

[bosecorp]

No I am not

however, I re-run the full install script minus the database part and I am in a much better place now.

I am still unable to see logs when I go to admin ==> fusion logs.

on the error_log.log file I see this error

[Mon Nov 13 16:30:56.881242 2017] [:error] [pid 21399] [client 10.205.129.110:64203] PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 82 bytes) in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 367, referer: http://usvanagiospfu2/nagiosfusion//adm ... ervers.php

[npolovenko]

That's definitely better. What's your memory limit in:

Code: Select all

/etc/php.ini

Look for the line that saysmemory_limit = and double the number. if you have 128M change it to 256M, if you have 256M change it to 512M.
After that restart apache

Code: Select all

service httpd restart

If you still have problems with the interface, navigate to /usr/local/nagiosfusion/scripts/
and run the script called

Code: Select all

./repair_databases.sh

Let us know how it goes.