Can not apply configuration changes

[hqin]

I just installed latest Nagios XI in our Oracle Linux 7.6 system(Oracle VM). When I tried to make some changes in XI web interface, for example, adding some new contacts, and click on "apply configuration", it shows "Waiting for configuration verification..." forever. Not sure what is going on? I can not see anything in Nagios log file. Also if I restart server, http doesn't seem to automatically start up and I have to run command to start it up: /usr/local/nagiosxi/scripts/manage_services.sh start httpd.

BTW, where can I find detail management document for Nagios XI? Nagios XI administration guide or user guide doesn't have information relating to any commands to manage Nagios XI

[ssax]

Please run this tail command (and leave it running):

Code: Select all

tail -F /usr/local/nagiosxi/var/cmdsubsys.log

Then try to apply configuration through the web interface and once it completes/errors/goes for more than 5 minutes send me the full output from the tail command above.

[hqin]

Please run this tail command (and leave it running):

Code: Select all

tail -F /usr/local/nagiosxi/var/cmdsubsys.log

Then try to apply configuration through the web interface and once it completes/errors/goes for more than 5 minutes send me the full output from the tail command above.

There is not such log file.

tail: cannot open /usr/local/nagiosxi/var/cmdsubsys.log for reading: No such file or directory

[lmiltchev]

There is not such log file.

This is very strange... Did you install Nagios XI, using some custom (non-default) paths? The "cmdsubsys.log" file should be in the "/usr/local/nagiosxi/var" directory...

Can you run the following commands, and show the output?

Code: Select all

ls -la /usr/local/nagiosxi/var
grep 'memory_limit =\|max_execution_time =\|max_input_time =\|max_input_vars =\|memory_limit =' /etc/php.ini
tail /var/log/mariadb/mariadb.log

[hqin]

There is not such log file.

This is very strange... Did you install Nagios XI, using some custom (non-default) paths? The "cmdsubsys.log" file should be in the "/usr/local/nagiosxi/var" directory...

Can you run the following commands, and show the output?

Code: Select all

ls -la /usr/local/nagiosxi/var
grep 'memory_limit =\|max_execution_time =\|max_input_time =\|max_input_vars =\|memory_limit =' /etc/php.ini
tail /var/log/mariadb/mariadb.log

I didn't use custom path. I installed Nagios XI by following the manual installation instruction - https://assets.nagios.com/downloads/nag ... 1483109039. I downloaded installation file xi-5_6_2.tar.gz.

Here is the output
[root@br-sysmntr01 local]# ls -la /usr/local/nagiosxi/var
total 20
drwxrwxr-x 7 nagios nagios 4096 Jun 10 17:58 .
drwxr-xr-x 10 root nagios 102 Jun 10 17:53 ..
drwxrwxr-x 2 nagios nagios 66 Jun 10 18:06 certs
drwsrwsr-x 2 apache nagios 34 Jun 10 17:58 components
drwxrwxr-x 2 nagios nagios 20 Jun 10 17:58 keys
-rw-rw-rw- 1 nagios nagios 0 Jun 10 17:58 NXTI_Write_Test
drwxrwxr-x 2 nagios nagios 22 Jun 17 14:57 subsys
drwxrwxr-x 2 nagios nagios 6 Jun 10 17:53 upgrades
-rw-r--r-- 1 apache apache 0 Jun 10 17:58 wkhtmltox.log
-rw-r--r-- 1 nagios nagios 7 Jun 10 17:58 xi-itype
-rw-r--r-- 1 nagios nagios 944 Jun 10 17:53 xi-sys.cfg
-rw-r--r-- 1 nagios nagios 37 Jun 10 17:58 xi-uuid
-rw-r--r-- 1 nagios nagios 196 Jun 10 17:53 xiversion
[root@br-sysmntr01 local]# grep 'memory_limit =\|max_execution_time =\|max_input_time =\|max_input_vars =\|memory_limit =' /etc/php.ini
max_execution_time = 60
max_input_time = 120
max_input_vars = 5000
memory_limit = 256M
[root@br-sysmntr01 local]# tail /var/log/mariadb/mariadb.log
190617 14:57:19 InnoDB: Initializing buffer pool, size = 128.0M
190617 14:57:19 InnoDB: Completed initialization of buffer pool
190617 14:57:19 InnoDB: highest supported file format is Barracuda.
190617 14:57:19 InnoDB: Waiting for the background threads to start
190617 14:57:20 Percona XtraDB (http://www.percona.com) 5.5.59-MariaDB-38.11 started; log sequence number 3180598
190617 14:57:20 [Note] Plugin 'FEEDBACK' is disabled.
190617 14:57:20 [Note] Server socket created on IP: '0.0.0.0'.
190617 14:57:20 [Note] Event Scheduler: Loaded 0 events
190617 14:57:20 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.60-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server

[lmiltchev]

Not of the "expected" logs are in that directory... There could be a few reasons for that:

1. crond is not running or it is not allowed to run as nagios user.

Run the following commands and show the output in code wraps:

Code: Select all

service crond status
service crond restart
ps -ef | grep cron
tail -50 /var/log/cron

2. nagios or apache user is expired

Show the output of the following commands:

Code: Select all

chage -l nagios
chage -l apache

3. cron cannot write to the var directory (permissions issues)

Show the output of the following command:

Code: Select all

ls -lad /usr/local/nagiosxi/var

4. Least likely - selinux is enabled

Code: Select all

sestatus

[hqin]

Not of the "expected" logs are in that directory... There could be a few reasons for that:

1. crond is not running or it is not allowed to run as nagios user.

Run the following commands and show the output in code wraps:

Code: Select all

service crond status
service crond restart
ps -ef | grep cron
tail -50 /var/log/cron

Code: Select all

[root@br-sysmntr01 local]# service crond restart
Redirecting to /bin/systemctl restart crond.service
[root@br-sysmntr01 local]# ps -ef | grep cron
root     20846     1  0 17:13 ?        00:00:00 /usr/sbin/crond -n
root     20848 15294  0 17:13 pts/0    00:00:00 grep --color=auto cron
[root@br-sysmntr01 local]# tail -50 /var/log/cron
Jun 18 17:11:01 br-sysmntr01 CROND[20618]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Jun 18 17:11:01 br-sysmntr01 CROND[20618]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:12:01 br-sysmntr01 CROND[20661]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1)
Jun 18 17:12:01 br-sysmntr01 CROND[20663]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/snmptt_service_results.php >> /usr/local/nagiosxi/var/snmptt_service_results.log 2>&1)
Jun 18 17:12:01 br-sysmntr01 CROND[20664]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Jun 18 17:12:01 br-sysmntr01 CROND[20665]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Jun 18 17:12:01 br-sysmntr01 CROND[20666]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Jun 18 17:12:01 br-sysmntr01 CROND[20667]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Jun 18 17:12:01 br-sysmntr01 CROND[20668]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Jun 18 17:12:01 br-sysmntr01 CROND[20669]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Jun 18 17:12:01 br-sysmntr01 CROND[20670]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Jun 18 17:12:01 br-sysmntr01 CROND[20671]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Jun 18 17:12:01 br-sysmntr01 CROND[20672]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Jun 18 17:12:01 br-sysmntr01 CROND[20670]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:12:01 br-sysmntr01 CROND[20671]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:12:01 br-sysmntr01 CROND[20667]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:12:01 br-sysmntr01 CROND[20672]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:12:01 br-sysmntr01 CROND[20669]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:12:01 br-sysmntr01 CROND[20664]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:12:01 br-sysmntr01 CROND[20663]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:12:01 br-sysmntr01 CROND[20661]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:12:01 br-sysmntr01 CROND[20668]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:12:01 br-sysmntr01 CROND[20666]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:12:01 br-sysmntr01 CROND[20665]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:13:01 br-sysmntr01 CROND[20780]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Jun 18 17:13:01 br-sysmntr01 CROND[20781]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Jun 18 17:13:01 br-sysmntr01 CROND[20781]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:13:01 br-sysmntr01 CROND[20780]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:13:01 br-sysmntr01 CROND[20782]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Jun 18 17:13:01 br-sysmntr01 CROND[20782]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:13:01 br-sysmntr01 CROND[20783]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1)
Jun 18 17:13:01 br-sysmntr01 CROND[20783]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:13:01 br-sysmntr01 CROND[20784]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/snmptt_service_results.php >> /usr/local/nagiosxi/var/snmptt_service_results.log 2>&1)
Jun 18 17:13:01 br-sysmntr01 CROND[20784]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:13:01 br-sysmntr01 CROND[20785]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Jun 18 17:13:01 br-sysmntr01 CROND[20785]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:13:01 br-sysmntr01 CROND[20786]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Jun 18 17:13:01 br-sysmntr01 CROND[20787]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Jun 18 17:13:01 br-sysmntr01 CROND[20786]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:13:01 br-sysmntr01 CROND[20789]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Jun 18 17:13:01 br-sysmntr01 CROND[20790]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Jun 18 17:13:01 br-sysmntr01 CROND[20789]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:13:01 br-sysmntr01 CROND[20787]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:13:01 br-sysmntr01 CROND[20790]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:13:01 br-sysmntr01 CROND[20791]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Jun 18 17:13:01 br-sysmntr01 CROND[20791]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory
Jun 18 17:13:49 br-sysmntr01 crond[4958]: (CRON) INFO (Shutting down)
Jun 18 17:13:49 br-sysmntr01 crond[20846]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 60% if used.)
Jun 18 17:13:49 br-sysmntr01 crond[20846]: (CRON) INFO (running with inotify support)
Jun 18 17:13:49 br-sysmntr01 crond[20846]: (CRON) INFO (@reboot jobs will be run at computer's startup.)

2. nagios or apache user is expired

Show the output of the following commands:

Code: Select all

chage -l nagios
chage -l apache

Code: Select all

[root@br-sysmntr01 local]# chage -l nagios
Last password change                                    : Jun 10, 2019
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7
[root@br-sysmntr01 local]# chage -l apache
chage: user 'apache' does not exist in /etc/passwd

3. cron cannot write to the var directory (permissions issues)

Show the output of the following command:

Code: Select all

ls -lad /usr/local/nagiosxi/var

Code: Select all

[root@br-sysmntr01 local]# ls -lad /usr/local/nagiosxi/var
drwxrwxr-x 7 nagios nagios 4096 Jun 10 17:58 /usr/local/nagiosxi/var

4. Least likely - selinux is enabled

Code: Select all

sestatus

Code: Select all

[root@br-sysmntr01 local]# sestatus
SELinux status:                 disabled

[hqin]

Looks like /home/nagios doesn't exist. Could that be the cause? Do I need to manually create it?

[lmiltchev]

Jun 18 17:12:01 br-sysmntr01 CROND[20670]: (CRON) ERROR chdir failed (/home/nagios): No such file or directory

Did you remove the nagios home directory on this system?

[root@br-sysmntr01 local]# chage -l apache
chage: user 'apache' does not exist in /etc/passwd

The /etc/passwd has also being tampered with. I am not sure what else has been changed on this server. This definitely doesn't look like a "vanilla" Nagios XI instance. Have you done any hardening of the server that we should know about? Can you PM me your latest profile?

Admin > System Profile > Download Profile

[hqin]

profile zip file has been PMed. I haven't done any hardening on the server. Not sure why something is missing. Should I reinstall Nagios XI? I didn't see any uninstall script under Nagios source installation files folder