SQL Error [ndoutils] : Got error 28 from storage engine

[ace]

/tmp directory is filling up. How cycle the services and correct the issue? Also, Database Maintenance is red in UI


# sudo lsof | egrep '^COMMAND|deleted'
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mtxagent 2201 root 20u REG 253,5 512 262146 /var/tmp/etilqs_**** (deleted)
mtxagent 2201 root 21u REG 253,5 1024 262147 /var/tmp/etilqs_****** (deleted)
mtxagent 2201 root 25u REG 253,5 512 262148 /var/tmp/etilqs_*****(deleted)
mtxagent 2201 root 26u REG 253,5 1024 262149 /var/tmp/etilqs_***** (deleted)
mtxagent 2201 root 29u REG 253,5 2048 262150 /var/tmp/sqlite_******(deleted)
mtxagent 2201 root 31u REG 253,5 0 262151 /var/tmp/sqlite_****** (deleted)
mtxagent 2201 root 33u REG 253,5 2048 262152 /var/tmp/sqlite_******(deleted)
mtxagent 2201 root 35u REG 253,5 0 262153 /var/tmp/sqlite_*******(deleted)
mtxagent 2201 root 37u REG 253,5 2048 262154 /var/tmp/sqlite_*******(deleted)
mtxagent 2201 root 39u REG 253,5 0 262155 /var/tmp/sqlite_******* (deleted)
mtxagent 2201 root 49u REG 253,5 512 262156 /var/tmp/etilqs_******* (deleted)
mtxagent 2201 root 50u REG 253,5 1024 262157 /var/tmp/etilqs_******* (deleted)
mtxagent 2201 root 61u REG 253,5 512 262158 /var/tmp/etilqs_******* (deleted)
mtxagent 2201 root 62u REG 253,5 1024 262159 /var/tmp/etilqs_********(deleted)
ndo2db 2537 nagios 3u REG 253,4 1000881 1310769 /usr/local/nagios/var/ndo2db.debug.old (deleted)
ndo2db 7937 nagios 3u REG 253,4 1000507 1310759 /usr/local/nagios/var/ndo2db.debug.old (deleted)
nagios 7948 nagios 20w REG 253,4 18267 1313585 /usr/local/nagios/var/spool/perfdata/******.perfdata.host-PID-8327 (deleted)
nagios 7948 nagios 21w REG 253,4 82550 1311760 /usr/local/nagios/var/spool/perfdata/*******.perfdata.service-PID-8328 (deleted)
postmaste 14064 postgres 18u REG 253,5 16777216 3147817 /var/lib/pgsql/data/pg_xlog/*******(deleted)
mysqld 28331 mysql 4u REG 253,3 0 53 /tmp/ibLAfbz6 (deleted)
mysqld 28331 mysql 5u REG 253,3 0 54 /tmp/ibPPvEJD (deleted)
mysqld 28331 mysql 6u REG 253,3 0 58 /tmp/ibU9i8Ta (deleted)
mysqld 28331 mysql 7u REG 253,3 0 59 /tmp/ibiagc5H (deleted)
mysqld 28331 mysql 11u REG 253,3 0 60 /tmp/ibnPKJif (deleted)
mysqld 28331 mysql 19u REG 253,3 434212923 5539 /tmp/*****(deleted)
mysqld 28331 mysql 69u REG 253,3 433045504 5540 /tmp/ST8TUHXY (deleted)

[ace]

can I just restart mysqld or do I need to restart in a certain order? service mysqld restart

[ace]

Will this work or should I just reboot?

service nagios stop
service npcd stop
service ndo2db stop
service mysqld stop
service postgresql stop
service httpd stop
service crond stop
service crond start
service httpd start
service postgresql start
service mysqld start
service ndo2db start
service npcd start
service nagios start

[npolovenko]

Hello, @ace. Please run the repair DB script as root first and make sure that your server is not running out of disk space "df -h":

/usr/local/nagiosxi/scripts/repair_databases.sh

Next, run these commands if you are on CentOS/RHEL 6.X:

service crond stop
service npcd stop
service nagios stop
service ndo2db stop
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
service mysqld restart
service ndo2db start
service nagios start
service npcd start
service crond start

Run these commands if you are on CentOS/RHEL 7.X:

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mariadb
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

[ace]

the database repair fixed the issue and freed up space on /tmp
I didn't restart any of the services/

Thanks!

[npolovenko]

@ace, No problem. I will be closing this thread as resolved but feel free to open a new one if something else comes up.