nagios_downtimehistory is marked as crashed

[brucej543]

Change to max connections because we have the Maria DB. I also add the nax open files at 4096.
we will monitor and let you know if this helped

[brucej543]

Still getting the failed messages for the downtimehistory.

[tgriep]

Run another database repair to make sure the datatbase is clean and the errors are not caused from before increasing the max connections.

Code: Select all

mysqlcheck -f -r -u root -pnagiosxi --all-databases --use-frm

[brucej543]

The database repair was run prior to changing the Max_connections. The sqlcheck output is attached

[brucej543]

Please note that I am backing out the changes to the my.cfg file and restarting the nagios. For the past two nights at 11:00 PM. the system CPU usage jumps from about 10% to 100% and then about an hour later it runs out of memory which requires a power off/on to clear.
After the restart, ran the DB repair script which completed around 12:17 AM, the first re-occurrence of the /nagios/nagios_downtimehistory' is marked as crashed came at exactly 2:00 AM

This is becoming a critical issue and my management wants this corrected.

[brucej543]

FYI. I have decided to take another approach to address this issue. I have removed all scheduled and re-occurring downtime's from Nagios. In monitoring the /var/log/mariadb/mariadb.log, I noticed that every time a downtime would start or stop, the error would be written to the log file.
I did not back out the change max_connections and it set at 818

[tgriep]

If you know the names of the processes that are loading the CPU and taking the most memory, can you post the names here?

I will need to get some log files and a System Profile so I can see what was going on.

You can PM them to me if you do not want to post them.

Get the following log files during the time the issue happened and either Post them or PM them to me.

Code: Select all

/var/log/messages
/var/log/cron
/var/log/secure
/var/log/httpd/error_log
/var/log/httpd/ssl_error_log
/usr/local/nagiosxi/var/recurringdowntime.log

The log files will be large so zip them up.

To get your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and upload it to the forum post or PM it.

Run the following as root and post the /tmp/info.txt file so I can check the database table.

Code: Select all

echo 'desc nagios_downtimehistory;' |mysql -t -u root -pnagiosxi nagios >/tmp/info.txt
echo 'select * from nagios_downtimehistory;' |mysql -t -u root -pnagiosxi nagios >>/tmp/info.txt

Thanks.

[brucej543]

Requested files

[tgriep]

I see that the server is running symantec_antivirus. Is it set to run at 11PM to scan the system?

Can you upload the other files here?

Code: Select all

/var/log/messages
/var/log/cron
/var/log/secure
/var/log/httpd/error_log
/var/log/httpd/ssl_error_log
/usr/local/nagiosxi/var/recurringdowntime.log

Run the following as root and post the /tmp/info.txt file.

Code: Select all

ls -lR //usr/local/nagios >/tmp/info.txt

[brucej543]

Requested files