NDO-3: INSERT INTO nagios_servicechecks

[viatel]

Hi

Since a couple of days I have started getting a problem were my /usr/local/nagios/var/nagios.log file just grows by about 4-5GB / 24 hours and in a couple of days my filesystem is full on my Nagios server. The problem is related to database inserts failing as per:

Code: Select all

[1647354247] NDO-3: INSERT INTO nagios_servicechecks (instance_id, start_time, start_time_usec, end_time, end_time_usec, service_object_id, check_type, current_check_attempt, max_check_attempts, state, state_type, timeout, early_timeout, execution_time, latency, return_code, output, long_output, perfdata, command_object_id, command_args, command_line) VALUES (1,FROM_UNIXTIME(1647354247),621734,FROM_UNIXTIME(1647354247),734849,9379,0,1,1,0,1,60,0,0.113115,0.000277,0,'DNS OK: 0.010 seconds response time. hostname.domain.tld returns 192.168.100.101','','time=0.010104s;2.000000;4.000000;0.000000',0,'','') ON DUPLICATE KEY UPDATE instance_id = VALUES(instance_id), start_time = VALUES(start_time), start_time_usec = VALUES(start_time_usec), end_time = VALUES(end_time), end_time_usec = VALUES(end_time_usec), service_object_id = VALUES(service_object_id), check_type = VALUES(check_type), current_check_attempt = VALUES(current_check_attempt), max_check_attempts = VALUES(max_check_attempts), state = VALUES(state), state_type = VALUES(state_type), timeout = VALUES(timeout), early_timeout = VALUES(early_timeout), execution_time = VALUES(execution_time), latency = VALUES(latency), return_code = VALUES(return_code), output = VALUES(output), long_output = VALUES(long_output), perfdata = VALUES(perfdata), command_object_id = VALUES(command_object_id), command_args = VALUES(command_args), command_line = VALUES(command_line)

I was running 5.8.7 when the problem started this weekend (had been running it since 7th of February this year without any problems) and this morning I tried upgrading to 5.8.8 just to run that latest version before asking for help. Upgrade did not resolve the problem unfortunately.

Server is a hyper-V virtual machine, fully patched CentOS 7:
[root@u3 ~]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
[root@u3 ~]# uname -a
Linux u3.jara.local 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Been running for 5-6 years now, all the way back to version 5.2.5 at least and never had a problem until now. I hope someone can help me with some insights into solving my problem?

Best regards

Anders

[viatel]

Update to my own post;
found this other thread: https://support.nagios.com/forum/viewto ... 3&start=10
and ran the database backup step and repair step;

Code: Select all

[root@u3 ~]# mysqldump -u root -p nagios  nagios_servicechecks  > /tmp/nagios_servicechecks.sql
Enter password:
[root@u3 ~]# ls -la /tmp/nagios_servicechecks.sql
-rw-r--r-- 1 root root 3157 Mar 15 16:06 /tmp/nagios_servicechecks.sql
[root@u3 ~]# less /tmp/nagios_servicechecks.sql
[root@u3 ~]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 20723
Server version: 5.5.68-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use nagios;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [nagios]> check table nagios_servicechecks;
+-----------------------------+-------+----------+----------+
| Table                       | Op    | Msg_type | Msg_text |
+-----------------------------+-------+----------+----------+
| nagios.nagios_servicechecks | check | status   | OK       |
+-----------------------------+-------+----------+----------+
1 row in set (0.00 sec)

MariaDB [nagios]> repair table nagios_servicechecks use_frm;
+-----------------------------+--------+----------+----------+
| Table                       | Op     | Msg_type | Msg_text |
+-----------------------------+--------+----------+----------+
| nagios.nagios_servicechecks | repair | status   | OK       |
+-----------------------------+--------+----------+----------+
1 row in set (0.00 sec)

MariaDB [nagios]> quit

and even without any sign of repairing any rows at least for now the constant flow of error messages to /usr/local/nagios/var/nagios.log have stopped and any regular host alerts etc appear as expected again.

If anyone can confirm that my operation is valid / relevant it would be much appreciated?

Best regards

Anders

[pbroste]

Hello @viatel

Thanks for reaching out and taking the time to follow up and explain what you ran through. After review, it appears that the command basically printed out the status.

At this point, please keep an eye on this issue, and if find that the problem pops-up again please open a new ticket in our new Answer Hub.

We're moving to a new support system!

The Nagios Answer Hub is a place where you can get help with technical questions from our experts. There, you can quickly open tickets and join discussion boards.

Request Nagios Answer Hub access here: https://info.nagios.com/answer-hub-access-new-users

After completing the access form, you will be given access to a portal where new tickets can be created. We will keep the old customer forum sections and ticket system available for current cases to be resolved.

Thanks,
Perry