NDO-3

[tbarnett]

I have a lot of this in my nagios.log. this all started a little over a week ago and I have just discovered it.

I will attach a system profile too.

I've checked my setup and everything appears to be correct however I still have this.

Please let me know what next steps to take.

[1628599609] NDO-3: The following query failed while MySQL appears to be connected:
[1628599609] NDO-3: INSERT INTO nagios_servicechecks (instance_id, start_time, start_time_usec, end_time, end_time_usec, service_object_id, check_type, current_check_attempt, max_check_attempts, state, state_type, timeout, early_timeout, execution_time, latency, return_code, output, long_output, perfdata, command_object_id, command_args, command_line) VALUES (1,FROM_UNIXTIME(1628599600),726356,FROM_UNIXTIME(1628599600),745985,11444,0,1,5,0,1,90,0,0.019629,0.041710,0,'OK - 192.168.205.7 rta 1.586ms lost 0%','','rta=1.586ms;3000.000;5000.000;0; pl=0%;80;100;0;100 rtmax=2.813ms;;;; rtmin=1.047ms;;;;',0,'','') ON DUPLICATE KEY UPDATE instance_id = VALUES(instance_id), start_time = VALUES(start_time), start_time_usec = VALUES(start_time_usec), end_time = VALUES(end_time), end_time_usec = VALUES(end_time_usec), service_object_id = VALUES(service_object_id), check_type = VALUES(check_type), current_check_attempt = VALUES(current_check_attempt), max_check_attempts = VALUES(max_check_attempts), state = VALUES(state), state_type = VALUES(state_type), timeout = VALUES(timeout), early_timeout = VALUES(early_timeout), execution_time = VALUES(execution_time), latency = VALUES(latency), return_code = VALUES(return_code), output = VALUES(output), long_output = VALUES(long_output), perfdata = VALUES(perfdata), command_object_id = VALUES(command_object_id), command_args = VALUES(command_args), command_line = VALUES(command_line)
[1628599609] NDO-3: The following query failed while MySQL appears to be connected:
[1628599609] Caught SIGTERM, shutting down...

[tbarnett]

So I tried to do one of these inserts that are failing directly into the nagios_servicechecks table with MySQL workshop and I got the following error.

Error Code: 167. Out of range value for column 'servicecheck_id' at row 1


Right now there are no records in my nagios_servicechecks table.

And there is no mention of servicecheck_id in the INSERT statement.

This seems wrong to me.

Still trying to figure this out before the drive gets full. thx

Tom

[gsmith]

Hi

In /etc/my.cnf.d, create a file called nagios.cnf with the following lines:

Code: Select all

[mysqld]
innodb_log_buffer_size = 32M
innodb_buffer_pool_size = 1G
innodb_log_file_size = 256M
max_allowed_packet = 67108864

Then run the following commands as root:

Code: Select all

/usr/local/nagiosxi/scripts/manage_services.sh stop nagios
/usr/local/nagiosxi/scripts/manage_services.sh stop mysqld
rm /var/lib/mysql/ib_logfile0
rm /var/lib/mysql/ib_logfile1
/usr/local/nagiosxi/scripts/manage_services.sh restart nagios
/usr/local/nagiosxi/scripts/manage_services.sh restart mysqld

Let me know if you run into any issues.

Thanks

[tbarnett]

I am running MariaDb on a remote server so I adjusted some of your commands and ran them.

I still have the same error unfortunately.

Is there anything I should do differently because of the remote server?

Tom

[gsmith]

Hi Tom

Regarding:

So I tried to do one of these inserts that are failing directly into the nagios_servicechecks table with MySQL workshop and I got the following error.
Error Code: 167. Out of range value for column 'servicecheck_id' at row 1
Right now there are no records in my nagios_servicechecks table.
And there is no mention of servicecheck_id in the INSERT statement.
This seems wrong to me.
Still trying to figure this out before the drive gets full. thx

Tom

The error you are seeing is because servicecheck_id is the primary key on that table and is using auto_increment.

Run this in mysql to see what the value currently is:

Code: Select all

select max(servicecheck_id) from nagios_servicechecks;

I am still looking for an answer to the original problem.

Thanks

[tbarnett]

Picture attached

[tbarnett]

[[email protected]@nagiosxidb mysql]$ mysql -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 23063
Server version: 5.5.68-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use Nagios;
ERROR 1049 (42000): Unknown database 'Nagios'
MariaDB [(none)]> use nagios;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [nagios]> select max(servicecheck_id) from nagios_servicechecks;
+----------------------+
| max(servicecheck_id) |
+----------------------+
| NULL |
+----------------------+
1 row in set (0.00 sec)

[gsmith]

Hi,

Now that you added the changes of nagios.cnf please provide:

1. a System Profile of the Nagios server.

To send us your system profile.
1. Login to the Nagios XI GUI using a web browser.
2. Click the "Admin" > "System Profile" Menu
3. Click the "Download Profile" button
4. Save the profile.zip file and share this in a private message and then reply to this post to bring it up in the queue.

2. from the remote db server:
a.

Code: Select all

tar cvzf /tmp/mycnfd.tar.gz  /etc/my.cnf.d

and send us the /tmp/mycnfd.tar.gz file
b.

Code: Select all

tar cvzf /tmp/mysqllog.tar.gz  /var/log/mysql

and send us the /tmp/mysqllog.tar.gz file

Thanks

[tbarnett]

See attached files.

Tom

[gsmith]

Hi

Please run the following and provide the output:

Code: Select all

mysql -u root -p
<password>
SELECTCONCAT(CEILING(ibps/POWER(1024,1)),'K') IBPS_KB,
    CONCAT(CEILING(ibps/POWER(1024,2)),'M') IBPS_MB,
    CONCAT(CEILING(ibps/POWER(1024,3)),'G') IBPS_GB
FROM(SELECTSUM(data_length+index_length) ibps
FROMinformation_schema.tables WHEREengine='InnoDB') A;

SHOW ENGINE INNODB STATUS\G;

I see you have Percona XtraDB installed. Is that db clustering software? Is this db part of a cluster?

Thanks