Nagios XI crashing frequently

[progressive.nagiosXI]

HI Team,

As we have updated Nagios XI 5.5.7 our Nagios XI has been crashed 2 times due to this we are facing so much problem we have 48 customers with 1600 devices you can understand how hard it is for us.

I "ll tell you what we are facing whenever we are doing apply configuration its giving error like below article,

https://support.nagios.com/kb/article.php?id=34 ==> we have done all the steps in here, but unable to resolve this,

its giving problem showing in for any host or sevices or servicegroup file or commands file that somethis is not right after this we have tried restore the previous best snapshot but the problem is same even when we have deleted that device which was showing error still problem is showing ==> Looks like Nagios XI was stucked on that config file and unable to read the updated one (Which has been done by restoring on previous good snapshot)

As we have daily backup of server so we have started another server but this is very critical we require your help !!

[benjaminsmith]

Hello,

I would like to gather some information to help us troubleshoot your problem. There are two issues here: one is the server crashing frequently, and the other is that the apply configuration process never completes. Did you notice the apply configuration error before upgrading?

1. Can you send us a system profile to review.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and upload it to a cloud storage of your choice. You can share a link with me in a personal message.
After you upload the profile please post something in this thread to bring it up in the support queue.

2. Go to Admin > System Status. What is the status of the system components. Is everything up and running (i.e monitoring engine, database backend..etc)?

[progressive.nagiosXI]

Hi,

Its a bug in 5.5.7 update also i have gone through one of the posts of some user about that and i would like to switch on 5.5.5 can you please suggest how can i do the same

I have tried downloading the 5.5.5 tar.gz and then run upgrade script but it's not working is there any other way or steps.

[benjaminsmith]

Hi @progressive.nagiosXI,

Depending on the issue, downgrading alone may or may not resolve the problem. At this point, I believe it would be best to review your logs so we can try to find the root cause.

1. If you haven't already, can you PM me your system profile?

2. Log in as Nagios and run the re-configure script:

Code: Select all

su - nagios
/usr/local/nagiosxi/scripts/reconfigure_nagios.sh

3. In Nagios XI select Configure> Quick Tools > Apply configuration, and then run the following command from the shell and post the output to the screen or upload a file.

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log
tail -f /usr/local/nagiosxi/var/cmdsubsys.log > /tmp/cmd.txt

Thank you.

[progressive.nagiosXI]

this a sample output of trial server with the same problem

PROCESS COMMAND: CMD=1150, DATA=remove
CMDLINE=php /usr/local/nagiosxi/html/includes/components/nagiosbpi/api_tool.php --cmd=syncall
PHP Notice: Undefined variable: err in /usr/local/nagiosxi/html/includes/components/nagiosbpi/api_tool.php on line 146
CMD: syncall
MSG: Could not get data for objects. NDO or Core may not be running.
OUTPUT=MSG: Could not get data for objects. NDO or Core may not be running.
RETURNCODE=0
...............
PROCESSED 2 COMMANDS
...........................................................
PROCESSED 0 COMMANDS
............................................................
PROCESSED 0 COMMANDS
..........................................................
PROCESSED 0 COMMANDS
............................................


API problem is very critical

[benjaminsmith]

Hi @progressive.nagiosXI,

It looks like Nagios is not starting up. If you could PM the following information along with the output of the commands below, it would be very helpful.

1. Can you PM your system profile along with your startup script: /etc/init.d/nagios?

2. Please run the commands for the correct lock file path and show the output. The the lock file is either in /usr/local/nagios/var/nagios.lock or /var/run/nagios.lock. Please open up your /usr/local/nagios/etc/nagios.cfg to verify.

Code: Select all

ls -la /usr/local/nagios/var/nagios.lock
cat /usr/local/nagios/var/nagios.lock

or

Code: Select all

ls -la /var/run/nagios.lock
cat /var/run/nagios.lock

Thank you.

[progressive.nagiosXI]

[root@monitoring-nagiosxi ~]# ls -la /usr/local/nagios/var/nagios.lock
-rw-r--r-- 1 nagios nagios 6 Dec 4 15:49 /usr/local/nagios/var/nagios.lock


[root@monitoring-nagiosxi ~]# cat /usr/local/nagios/var/nagios.lock
31428

[benjaminsmith]

Hello @progressive.nagiosXI,

Thank you for sending over your profile. After reviewing your profile, it looks like there are multiple issues here, so please open a support ticket, reference this post, as this will be the fastest way to resolution.

In the meantime, a few things from your profile that can be addressed:

1. There are multiple disk space errors in the database log. Please verify that you are not running out of disk space on any partitions.

Code: Select all

181201 20:12:24 [Warning] mysqld: Disk is full writing '/var/lib/mysql/nagiosxi/xi_auditlog.MYI' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)

2. Crashed tables in the db (181129 11:11:36 [ERROR] mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failed). Take a backup or VM snapshot and run the repair script:

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

[progressive.nagiosXI]

I have created a ticket

Re: Nagios XI crashing frequently #555488

[benjaminsmith]

I have created a ticket.

Thanks. I will close this post, and follow up with the other support tech.