nagios_downtimehistory is marked as crashed

[brucej543]

Errors in the DB log file:
200803 12:00:09 [ERROR] nagios.nagios_downtimehistory: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_downtimehistory.TMM'
200803 12:00:09 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'

Listing of the files:
[root@bcnagios01 ~]# ls -ltr /var/lib/mysql/nagios/nagios_downtimehistory.TMM
-rw-rw---- 1 mysql mysql 3108864 Jun 25 08:28 /var/lib/mysql/nagios/nagios_downtimehistory.TMM
[root@bcnagios01 ~]# ls -ltr /var/lib/mysql/nagios/nagios_logentries.TMM
-rw-rw---- 1 mysql mysql 0 Jul 31 21:39 /var/lib/mysql/nagios/nagios_logentries.TMM

Do these files need to be deleted so mysql can recreate them?

[brucej543]

good day. we are still getting the below errors in the db log file. I need to get these issues resolved so I can re-add my reoccurring scheduled downtimes back into nagios. This is taking way to long to resolve.

Errors for today from the /var/log/mariadb/mariadb.log log file
200804 0:50:09 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 0:50:09 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 0:50:09 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'
200804 1:55:09 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 1:55:09 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 1:55:09 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'
200804 3:00:08 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 3:00:08 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 3:00:09 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'
200804 4:05:08 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 4:05:08 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 4:05:09 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'
200804 5:05:09 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 5:05:09 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 5:05:09 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'
200804 6:10:08 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 6:10:08 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 6:10:08 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'
200804 7:00:03 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 7:00:03 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 7:10:09 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 7:10:09 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 7:10:09 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'
200804 8:15:04 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 8:15:04 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 8:15:04 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'
200804 9:20:09 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 9:20:09 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 9:20:09 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'
200804 10:35:03 [ERROR] nagios.nagios_downtimehistory: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_downtimehistory.TMM'
200804 10:35:03 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'
200804 11:40:09 [ERROR] nagios.nagios_downtimehistory: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_downtimehistory.TMM'
200804 11:40:09 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'
200804 12:45:08 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 12:45:08 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200804 12:45:08 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'

[ssax]

Code: Select all

/dev/mapper/rootvg-tmp 1.9G 8.8M 1.7G 1% /tmp

Increase your /tmp size:

See here:

https://support.nagios.com/kb/article.php?id=814

You can also change your mariadb temp dir location by adding tmpdir=/path/to/your/dir in your /etc/my.cnf and restart mariadb:

Code: Select all

systemctl mariadb restart

Then repair them again:

Code: Select all

cd /usr/local/nagiosxi/scripts
./repair_databases.sh

[brucej543]

Expanded tmp (added 3 GB) to size is now 4.8 GB with 4.6 GB free.
Did not change the location of the maridb temp directory - Do I need to halt Nagios to make the change and restart the DB and then restart NAgios?
Ran the DB repair script.
Ran the DB list of files of sizes and the table has the size of NULL
nagios | nagios_downtimehistory | NULL |
Still getting errors in DB log file
200805 10:43:37 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200805 10:43:37 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200805 11:10:09 [ERROR] mysqld: Table './nagios/nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200805 11:10:09 [ERROR] mysqld: Table 'nagios_downtimehistory' is marked as crashed and last (automatic?) repair failed
200805 11:10:09 [ERROR] nagios.nagios_logentries: Can't create new tempfile: '/var/lib/mysql/nagios/nagios_logentries.TMM'

Just an FYI, this crash table issue is also causing the nightly nagios backups to fail.

[ssax]

Do I need to halt Nagios to make the change and restart the DB and then restart Nagios?

Just expanding it should work but restarting mariadb should be done to kill the old connections. Do you still get the same errors while running the repair_databases.sh command? (please send us the full output if you have it).

Run it again after restarting mariadb to see if that helps.

Does this show the expanded size?

Code: Select all

df -h
df -i

You can try to repair it like this:

Code: Select all

mysql -h 127.0.0.1 -uroot -pnagiosxi nagios -e "repair table nagios_downtimehistory;"

If you're not concerned about the downtime history you can run this command to drop/recreate it:

Code: Select all

mysql -h 127.0.0.1 -uroot -pnagiosxi nagios -e "truncate table nagios_downtimehistory;"

And for the nagios_logentries table (which only effects Reports > Event Log):

Code: Select all

mysql -h 127.0.0.1 -uroot -pnagiosxi nagios -e "truncate table nagios_logentries;"

[brucej543]

below is the requested verification of the /tmp space being expanded.
Regarding the issue with Mariadb not being able to create the the tempfiles for /var/lib/mysql/nagios/nagios_downtimehistory.TMM and /var/lib/mysql/nagios/nagios_logentries.TMM, I took the following actions:
Halted Nagios and the MariaDB, removed both TMM files and restart MariaDB and Nagios.
I will monitor the DB log file for any errors and update the ticket.

[root@bcnagios01 ~]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 4094336 494 4093842 1% /dev
tmpfs 4097220 1 4097219 1% /dev/shm
tmpfs 4097220 1450 4095770 1% /run
tmpfs 4097220 16 4097204 1% /sys/fs/cgroup
/dev/mapper/rootvg-root 244320 7061 237259 3% /
/dev/mapper/rootvg-usr 305216 129975 175241 43% /usr
/dev/sda1 121920 350 121570 1% /boot
/dev/mapper/rootvg-sysadmin 548864 456 548408 1% /sysadmin
/dev/mapper/rootvg-opt 219888 239 219649 1% /opt
/dev/mapper/rootvg-rootvg--backup 5701632 219 5701413 1% /store
/dev/mapper/rootvg-tmp 317616 586 317030 1% /tmp
/dev/mapper/rootvg-usr_local 2570240 47811 2522429 2% /usr/local
/dev/mapper/rootvg-var 4910160 145775 4764385 3% /var
tmpfs 4097220 1 4097219 1% /run/user/0
tmpfs 4097220 1 4097219 1% /run/user/1003
tmpfs 4097220 1 4097219 1% /run/user/1744104356
[root@bcnagios01 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 1.2G 15G 8% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/rootvg-root 3.7G 201M 3.3G 6% /
/dev/mapper/rootvg-usr 4.5G 3.1G 1.2G 73% /usr
/dev/sda1 453M 193M 233M 46% /boot
/dev/mapper/rootvg-sysadmin 2.1G 18M 2.0G 1% /sysadmin
/dev/mapper/rootvg-opt 3.3G 778M 2.4G 25% /opt
/dev/mapper/rootvg-rootvg--backup 86G 27G 56G 33% /store
/dev/mapper/rootvg-tmp 4.8G 11M 4.6G 1% /tmp
/dev/mapper/rootvg-usr_local 40G 17G 22G 43% /usr/local
/dev/mapper/rootvg-var 76G 9.6G 63G 14% /var
tmpfs 3.2G 0 3.2G 0% /run/user/0
tmpfs 3.2G 0 3.2G 0% /run/user/1003
tmpfs 3.2G 0 3.2G 0% /run/user/1744104356

[ssax]

Ok, that looks good, 4GB may be a little low though, if you have crashed tables it can fill up your /var/log/mariadb/mariadb.log pretty quick.

We'll keep an eye out for your updates.

[brucej543]

This post can be closed. we fix this issue

[scottwilkerson]

This post can be closed. we fix this issue

Great!

Locking thread