Hosts and services temporarily unavailable

[drug]

Shortly following an Apply Configuration, hosts and services either entirely disappear from the web UI (not-CCM) and return "object does not exist" or they appear in a greyed out state. All hosts and services return after 30-60 seconds. Is this expected behavior?

[benjaminsmith]

Hello @drug,

It's likely a database issue. When did you notice this happening and have you made a changes to the server recently?

Let's try to capture some of the log files to help diagnose the issue.

1. Run the following tail command and then Apply Configuration and post the ful output.

Code: Select all

tail -F /usr/local/nagiosxi/var/cmdsubsys.log

2. Run one more tail command on the Apache errors logs, Apply Configuration and then generate the 'object does not exist' error, and post the full output.

Code: Select all

tail -f /var/log/httpd/*error_log

3. Finally, send over your system profile for us to review. Thanks.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message and then reply to this post to bring it up in the queue.

[drug]

No changes have been made recently. I believe this has been an issue since our deployment years ago; it's just become more apparent lately.

I should also note that MySQL is running on a separate server.

tail -F /usr/local/nagiosxi/var/cmdsubsys.log:

Code: Select all

...PROCESSING COMMAND ID 31590...
PROCESS COMMAND: CMD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
No entry for terminal type "unknown";
using dumb terminal settings.

--- reset_config_perms.sh ------------
> Setting script permissions
> Setting CCM script permissions
> Setting special script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting Nagios XI config permissions
> Setting NOM checkpoint user:group permissions
> + Setting Nagios Core corelog.newobjects user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------

--- ccm_import.php -------------------
> Setting import directory: /usr/local/nagios/etc/import/
> Importing config files into the CCM
  No files to import
--------------------------------------

--- ccm_export.php -------------------
> Writing CCM configuration to Nagios files
  Finished writing out configuraton
--------------------------------------

--------------------------------------
> Verifying configuration with Nagios Core
> Output: 
Nagios Core 4.4.3
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2019-01-15
License: GPL

Website: https://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
	Checked 7550 services.
	Checked 1139 hosts.
	Checked 118 host groups.
	Checked 7 service groups.
	Checked 105 contacts.
	Checked 15 contact groups.
	Checked 293 commands.
	Checked 142 time periods.
	Checked 0 host escalations.
	Checked 0 service escalations.
Checking for circular paths...
	Checked 1139 hosts
	Checked 23 service dependencies
	Checked 1 host dependencies
	Checked 142 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
> Return Code: 0
--------------------------------------
OUTPUT=--------------------------------------
RETURNCODE=0
PROCESSING COMMAND ID 31591...
PROCESS COMMAND: CMD=1150, DATA=remove
CMDLINE=php /usr/local/nagiosxi/html/includes/components/nagiosbpi/api_tool.php --cmd=syncall 
...CMD: syncall
MSG: Could not get data for objects. NDO or Core may not be running.
OUTPUT=MSG: Could not get data for objects. NDO or Core may not be running.
RETURNCODE=0

PROCESSED 2 COMMANDS
..................

No data was logged to the Apache error log at the time.

[benjaminsmith]

Hello @drug,

No changes have been made recently. I believe this has been an issue since our deployment years ago; it's just become more apparent lately.
I should also note that MySQL is running on a separate server.

1. Can you send me your system profile?

2. The profile won't contain the database logs, can your retrieve those from the database host.

3. Also, make sure the system times between the Nagios XI server and the offloaded database are in sync ( see Changing The System Time for directions on setting time or syncing to an NTP server).

Thanks.

[drug]

Oops, Profile sent. There are no MySQL errors or slow query logs in the last day. I did run mysqlcheck with

Code: Select all

--auto-repair --check --all-databases --use_frm

but the problem remains. The time on both servers is in sync with NTP sources and both are UTC.

[benjaminsmith]

Hi @drug,

Thanks for checking the database, it looks like there is more than one ndo2db process running on the server, please restart the services using the commands below or re-boot the server and then let me know if the issue persists.

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
killall -9 ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

[drug]

I ran the commands as requested (copy/paste) with no errors. However, I still receive an "Object Does Not Exist" error for a service (I've selected services at random) if that service's page is reloading while the configuration is being Applied.

[benjaminsmith]

Hi @Drug,

After looking into this issue further, it's normal for this to occur during apply configuration for users as the objects are being re-built and written to the database and this will take some time to complete.

That said, some improvements were made in 5.6.3. I would recommend upgrading to the latest version of Nagios XI, and let us know the results.

Backing Up Nagios XI
Upgrading Nagios XI

[drug]

We've upgraded to 5.6.5 but we're still experiencing the same symptoms whereby hosts display as gray (and then disappear altogether from the console) and if a URL is loaded for a specific host or service, "Object Does Not Exist" is returned. The latter is particularly problematic as that error "page" does not include a refresh so manual intervention is required to load the page after a configuration is applied.

We have a need to apply many configuration changes during business hours. Is there another option for making these changes or any other actions we might be able to take to minimize these UI disruptions while applying the configuration?

[lmiltchev]

Can you show us the mysql log from the remote server (with the offloaded db)?

Code: Select all

tail -50 /var/log/mysqld.log

or

Code: Select all

tail -50 /var/log/mariadb/mariadb.log

if you are using mariadb.

Also, can you run the following commands on the Nagios XI server, and show the output?

Code: Select all

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
su nagios
time /usr/local/nagiosxi/scripts/reconfigure_nagios.sh
ps -ef | grep [n]agios
grep 'max_execution_time\|max_input_time\|memory_limit\|max_input_vars' /etc/php.ini | grep -v ';'