Page 2 of 4
Re: table nagios_notifications crashed (in every hour)
Posted: Thu Nov 21, 2019 3:15 am
by eycklin
Hi,
After fix, it's still crash.
Please see logs attached.
Many Thanks,
Eyck Lin
Re: table nagios_notifications crashed (in every hour)
Posted: Thu Nov 21, 2019 10:07 am
by tgriep
Can you run the following commands as root and post the /tmp/notifications.txt file do I can see the details of the nagios database and the notifications table?
Code: Select all
echo 'select * from nagios_notifications;' |mysql -t -u root -pnagiosxi nagios >/tmp/notifications.txt
mysqldump --no-data nagios -u root -pnagiosxi >>/tmp/notifications.txt
echo 'SELECT @@SQL_MODE, @@GLOBAL.SQL_MODE\G;' |mysql -t -u root -pnagiosxi nagios >>/tmp/notifications.txt
Thanks
Re: table nagios_notifications crashed (in every hour)
Posted: Thu Nov 21, 2019 10:47 am
by eycklin
Hi,
Please see the attached file - notifications.txt.gz.
Thanks,
Eyck Lin
Re: table nagios_notifications crashed (in every hour)
Posted: Thu Nov 21, 2019 2:25 pm
by tgriep
Thanks for the data. The settings on your server look stock, the table and the SQL settings are good.
The data in the table itself I could not find anything wrong, but there are over 16 thousand lines and I may of missed it.
But if the bad data did not get written it would not be there anyways.
I did see a lot of these types of message. It looks like a language issue and that may be causing the issue.
(Last runtime: 2019/11/19 ¤U¤È
See if there is a way to fix this in your check.
The service is called Check Win Tasks, and it is ran on Windows system with this command check_ms_win_tasks.
Lets enable debugging for the ndo2db daemon
edit this file /usr/local/nagios/etc/ndo2db.cfg and change the following from
Code: Select all
debug_level=0
max_debug_file_size=1000000
to
Code: Select all
debug_level=2
max_debug_file_size=10000000
Save the settings.
Run the following as root.
Code: Select all
systemctl stop nagios
systemctl stop ndo2db
mysqlcheck -f -r -u root -pnagiosxi --all-databases --use-frm
systemctl start ndo2db
systemctl start nagios
As soon as you see the notifications table error, get the following file and add it to the post.
Code: Select all
/usr/local/nagios/var/ndo2db.debug
One question, do you have and third party tools running on any system that is accessing the notifications table?
Re: table nagios_notifications crashed (in every hour)
Posted: Tue Nov 26, 2019 3:51 am
by eycklin
Hi,
After fix, it still crashed.
But the table crash period seems to be longer.
Please see attached log file.
BTW, you asked:
"One question, do you have and third party tools running on any system that is accessing the notifications table?"
No, we don't have any third party tools that is accessing the notifications table, as I know.
Thanks,
Eyck Lin
Re: table nagios_notifications crashed (in every hour)
Posted: Tue Nov 26, 2019 2:07 pm
by tgriep
First, upgrade to the latest version of Nagios XI.
There are some changes to Nagios Core that may fix the issue.
After the upgrade
Make sure you stop the nagios process and the ndo2db process so it stops writing to the database.
Repair the database and start up ndo2db and then nagios.
If the issue happens again, run these commands and post it here.
Code: Select all
echo 'show full processlist;' |mysql -t -u root -pnagiosxi nagios
echo 'select * from nagios_instances;' |mysql -t -u root -pnagiosxi nagios
Re: table nagios_notifications crashed (in every hour)
Posted: Wed Nov 27, 2019 2:56 am
by eycklin
Hi,
After upgrade to latest NagiosXI, repaired DB.
It's still the same(DB will crash).
Please see attached work log(data20191127.zip) & syslog(messages.zip).
Thanks,
Eyck Lin
Re: table nagios_notifications crashed (in every hour)
Posted: Wed Nov 27, 2019 11:12 am
by tgriep
Edit your /usr/local/nagios/etc/ndo2db.cfg file and add this option:
While you are in there, disable debugging.
Save the change.
Then restart the services:
Code: Select all
systemctl stop nagios
systemctl restart ndo2db
systemctl start nagios
Re: table nagios_notifications crashed (in every hour)
Posted: Thu Nov 28, 2019 4:49 am
by eycklin
Hi,
After fix, it's still crash.
Please see attached log file.
Do you have any idea about the root cause?
Thanks,
Eyck Lin
Re: table nagios_notifications crashed (in every hour)
Posted: Mon Dec 02, 2019 1:32 am
by eycklin
Hi,
About the DB crash, is there anything updated?
Thanks,
Eyck Lin