Nagios is crashing again and again

[arista-nagios]

Hello Team,

I have been observed several times that the database of our nagios server "https://cloudmon.ccb.cloudwifi.com/nagiosxi/" is crashing very frequently even not possible to ssh to the nagios server and have to repair database using below command after relaunching it.

/usr/local/nagiosxi/scripts/repair_databases.sh

Attaching image which shows nagios crashing behavior.

Please let us know what information you require to investigate it further.

Thanks,
Bharat Lalwani

[benjaminsmith]

Hello @arista-nagios,

A few questions to help us troubleshoot this issue. Please post the output to the commands below.

1. How is the disk space on the server?

Code: Select all

df -h

2. Post the output to the following query to check the sizes of the database tables.

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

3. Run through the following commands to re-start nagios, repair the databases, and then send us your system profile. Thanks.

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mysqld || systemctl restart mariadb
cd /usr/local/nagiosxi/scripts
./repair_databases.sh
systemctl start npcd
systemctl start crond
systemctl start nagios
systemctl start ndo2db

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message and then reply to this post to bring it up in the queue.