Hosts and services disappears for a while

[ITOMB_IMT]

HI,

Recently we upgraded NagiosXI to 5.6.1 hosted on RHEL 7.5 and currently we are facing issues with CCM where Hosts and services disappear for a while and after running reconfigure script/multiple refreshes the Hosts/services appear. This happens every time when we make changes to Hosts/services in CCM. This is annoying to not see the hosts/services and running the script all the time.

[benjaminsmith]

Hi @ISSB_MAOST,

You might have crashed database tables, please try to run the database repair script by running the following command as root:

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

Then restart entire Nagios XI stack:

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mariadb
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

Let me know if the issue is corrected or not.

[ITOMB_IMT]

No, still issue exists.

If i make any change and apply configuration, then the hosts/services disappear. Later i have to run re-configure.sh script, then the hosts/services appears again.

[benjaminsmith]

Hi @ISSB_MAOST,

Let's tail the log file while you apply configuration to see what might be the source of the issue. Run the following command:

Code: Select all

tail -F /usr/local/nagiosxi/var/cmdsubsys.log

and then apply configuration and post/upload the entire output.

Next, please send us your system profile to review. Thanks.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket.

[ITOMB_IMT]

I sent you a Pm with the cmdsubsys.log, when i try to download system profile am seeing the following error.

PROFILE BUILD FAILED
Array
(
)
CODE: 1

[benjaminsmith]

Hi @ISSB_MAOST,

That error is likely the result of an incorrect sodoers file, so we'll want to fix that first and then test again. Please follow the steps in the knowledge base article below to correct.

Nagios XI - Profile Build Failed

[ITOMB_IMT]

I followed all the steps specified, instead of 5.5.9 i tried with 5.6.1 but still am seeing the same error. I have sent you a pm of our sudoers file, please have a look.

Thanks,

[benjaminsmith]

Hi @ISSB_MAOST,

I took a look the sudoers file and it does look ok, so I'd like to check the following.

1. Do you have SE Linux enabled?

Code: Select all

getenforce

2. Is the Nagios User account expired?

Code: Select all

chage -l nagios

If it is expired, run this command to enable the expired nagios user:

Code: Select all

chage -I -1 -m 0 -M 99999 -E -1 nagios

I noticed this in the command subsystem log:

touch: cannot touch ‘/usr/local/nagiosxi/scripts/reconfigure_nagios.lock’: Permission denied

Please PM the following file so we can review the permissions.

Code: Select all

ls -lR /usr/local/nagios/ > /tmp/support.txt

Thanks.

[ITOMB_IMT]

Please see the output of the commnads

# getenforce
Disabled
# chage -l nagios
Last password change : Oct 16, 2018
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : -1
Maximum number of days between password change : -1
Number of days of warning before password expires : -1

and also see the PM of the output of ls -lR /usr/local/nagios/ > /tmp/support.txt

[benjaminsmith]

Hi @ISSB_MAOST,

Well, it looks like there are multiple issues present. One is the inability to generate a system profile, and issue with hosts and services disappearing is likely a database issue.

1. Can you run the following tail command to get the Apache error logs:

Code: Select all

tail -f /var/log/httpd/*error_log

.. and then try to download the system profile from Admin > System Config > System Profile. Please PM the error output from the tail command.

2. What version of mysql are you using?

Code: Select all

mysql -V

3. Can you post the output of the following database query and the db logs?
A. Login

Code: Select all

mysql -u root -pnagiosxi

B. Change database, and run the query

Code: Select all

use nagiosql;
SELECT @@SQL_MODE, @@GLOBAL.SQL_MODE\G;

The database logs are in /var/log/mysqld.log

Thanks.