table nagios_notifications crashed (in every hour)

[eycklin]

Hi,

After fix, it's still crash.

Please see logs attached.

Many Thanks,
Eyck Lin

[tgriep]

Can you run the following commands as root and post the /tmp/notifications.txt file do I can see the details of the nagios database and the notifications table?

Code: Select all

echo 'select * from nagios_notifications;' |mysql -t -u root -pnagiosxi nagios >/tmp/notifications.txt
mysqldump --no-data nagios -u root -pnagiosxi >>/tmp/notifications.txt
echo 'SELECT @@SQL_MODE, @@GLOBAL.SQL_MODE\G;' |mysql -t -u root -pnagiosxi nagios >>/tmp/notifications.txt

Thanks

[eycklin]

Hi,

Please see the attached file - notifications.txt.gz.

Thanks,
Eyck Lin

[tgriep]

Thanks for the data. The settings on your server look stock, the table and the SQL settings are good.

The data in the table itself I could not find anything wrong, but there are over 16 thousand lines and I may of missed it.
But if the bad data did not get written it would not be there anyways.

I did see a lot of these types of message. It looks like a language issue and that may be causing the issue.

(Last runtime: 2019/11/19 ¤U¤È

See if there is a way to fix this in your check.
The service is called Check Win Tasks, and it is ran on Windows system with this command check_ms_win_tasks.


Lets enable debugging for the ndo2db daemon
edit this file /usr/local/nagios/etc/ndo2db.cfg and change the following from

Code: Select all

debug_level=0
max_debug_file_size=1000000

to

Code: Select all

debug_level=2
max_debug_file_size=10000000

Save the settings.

Run the following as root.

Code: Select all

systemctl stop nagios
systemctl stop ndo2db
mysqlcheck -f -r -u root -pnagiosxi --all-databases --use-frm
systemctl start ndo2db
systemctl start nagios

As soon as you see the notifications table error, get the following file and add it to the post.

Code: Select all

/usr/local/nagios/var/ndo2db.debug

One question, do you have and third party tools running on any system that is accessing the notifications table?

[eycklin]

Hi,

After fix, it still crashed.
But the table crash period seems to be longer.
Please see attached log file.

BTW, you asked:
"One question, do you have and third party tools running on any system that is accessing the notifications table?"
No, we don't have any third party tools that is accessing the notifications table, as I know.

Thanks,
Eyck Lin

[tgriep]

First, upgrade to the latest version of Nagios XI.
There are some changes to Nagios Core that may fix the issue.

After the upgrade
Make sure you stop the nagios process and the ndo2db process so it stops writing to the database.
Repair the database and start up ndo2db and then nagios.

If the issue happens again, run these commands and post it here.

Code: Select all

echo 'show full processlist;' |mysql -t -u root -pnagiosxi nagios
echo 'select * from nagios_instances;' |mysql -t -u root -pnagiosxi nagios

[eycklin]

Hi,

After upgrade to latest NagiosXI, repaired DB.

It's still the same(DB will crash).

Please see attached work log(data20191127.zip) & syslog(messages.zip).

Thanks,
Eyck Lin

[tgriep]

Edit your /usr/local/nagios/etc/ndo2db.cfg file and add this option:

Code: Select all

max_notifications_age=0

While you are in there, disable debugging.

Save the change.
Then restart the services:

Code: Select all

systemctl stop nagios
systemctl restart ndo2db
systemctl start nagios

[eycklin]

Hi,

After fix, it's still crash.
Please see attached log file.

Do you have any idea about the root cause?

Thanks,
Eyck Lin

[eycklin]

Hi,

About the DB crash, is there anything updated?

Thanks,
Eyck Lin