Page 1 of 2
xi 5.4.12 event log disappeared
Posted: Fri May 03, 2019 11:51 am
by benhank
What a mystery. all of the xi event logs on my server have disappeared.
If I go to Admin/monitoring processes/ eventlog
and do a search from any time period nothing shows up.
Re: xi 5.4.12 event log disappeared
Posted: Fri May 03, 2019 12:58 pm
by npolovenko
Hello,
@benhank. Did anyone truncate the nagios_logentries table in mysql? What's the output of:
Code: Select all
echo "SELECT * FROM nagios_logentries;" | mysql -u root -pnagiosxi nagios
Also, please run this command and let me know if event log entries come back.
Code: Select all
echo 'repair table nagios_logentries use_frm;' | mysql -t -u root -pnagiosxi nagios
Re: xi 5.4.12 event log disappeared
Posted: Fri May 03, 2019 1:40 pm
by benhank
I ran it and got the following but the data has not returned
Code: Select all
echo 'repair table nagios_logentries use_frm;' | m ysql -t -u root -pnagiosxi nagios
+--------------------------+--------+----------+-------------------------------- -------+
| Table | Op | Msg_type | Msg_text |
+--------------------------+--------+----------+-------------------------------- -------+
| nagios.nagios_logentries | repair | warning | Number of rows changed from 0 t o 9146 |
| nagios.nagios_logentries | repair | status | OK |
+--------------------------+--------+----------+-------------------------------- -------+
Re: xi 5.4.12 event log disappeared
Posted: Fri May 03, 2019 1:46 pm
by npolovenko
@benhank, Please take a screenshot of the empty event log page in XI and upload it in this thread.
Could you send in your Nagios XI System Profile so I can review it?
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and send it to me in a private message.
Re: xi 5.4.12 event log disappeared
Posted: Mon May 06, 2019 10:42 am
by benhank
sent =D
And no we didn't truncate anything.
Re: xi 5.4.12 event log disappeared
Posted: Mon May 06, 2019 11:56 am
by npolovenko
@benhank, I'm still seeing entries related to a corrupted MySQL table.
Did you run the command I gave you on the server that hosts the offloaded database?
Please run this command a couple more times on the offloaded server:
echo 'repair table nagios_logentries use_frm;' | mysql -t -u root -pnagiosxi nagios
After that please run the following commands in order:
Code: Select all
service crond stop
service npcd stop
service nagios stop
service ndo2db stop
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
service mysqld restart
service ndo2db start
service nagios start
service npcd start
service crond start
Close the browser and reload the web page.
If this solution doesn't work, you may need to restore from a previous DB backup or restore just the logentries table. Can you show me the output of:
Re: xi 5.4.12 event log disappeared
Posted: Mon May 06, 2019 12:07 pm
by benhank
sigh, no I didnt run it on the offloaded server, because I forgot it was offloaded, just ran it with the following results:
Code: Select all
root@lkenfusionp01 ~]# echo 'repair table nagios_logentries use_frm;' | mysql -t -u root -pnagiosxi nagios
+--------------------------+--------+----------+-------------------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+--------------------------+--------+----------+-------------------------------------------------------------+
| nagios.nagios_logentries | repair | error | Can't create new tempfile: './nagios/nagios_logentries.TMD' |
| nagios.nagios_logentries | repair | status | Operation failed
Re: xi 5.4.12 event log disappeared
Posted: Mon May 06, 2019 12:17 pm
by npolovenko
@benhank, Your repair command didn't work. Is the offloaded MySQL server running out of space?
Also, try moving the nagios_logentries.TMD to a backup directory and rerunning the repair command one more time:
Code: Select all
mv /nagios/nagios_logentries.TMD /tmp/
Re: xi 5.4.12 event log disappeared
Posted: Mon May 06, 2019 12:32 pm
by benhank
yes im running it on the offloaded server (this time). The server isnt running out of space
Code: Select all
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_lkenfusiondev-lv_root
96G 14G 78G 16% /
tmpfs 7.8G 80K 7.8G 1% /dev/shm
/dev/sda1 477M 130M 322M 29% /boot
/dev/mapper/vg_lkenfusiondev-lv_home
446G 4.0G 420G 1% /home
/dev/mapper/vg_fusiondev900-lv_nagdb
413G 71M 392G 1% /nagdb
/dev/mapper/vg_fusiondev900-lv_perf
413G 71M 392G 1% /perf
[root@lkenfusionp01 nagios]#
and
Code: Select all
mv /nagios/nagios_logentries.TMD /tmp/
mv: cannot stat `/nagios/nagios_logentries.TMD': No such file or directory
[root@lkenfusionp01 ~]# cd /usr/local/nagios/
[root@lkenfusionp01 nagios]# mv /nagios/nagios_logentries.TMD /tmp/
mv: cannot stat `/nagios/nagios_logentries.TMD': No such file or directory
Re: xi 5.4.12 event log disappeared
Posted: Mon May 06, 2019 1:28 pm
by npolovenko
@benhank, Did you run the following commands I provided earlier?
service crond stop
service npcd stop
service nagios stop
service ndo2db stop
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
service mysqld restart
service ndo2db start
service nagios start
service npcd start
service crond start
You have duplicate nagios processes running and these commands should get rid of it.
And please try the following command to move the .TMD table:
mv /var/lib/mysql/nagios/nagios_logentries.TMD /tmp/
When done, please run the repair table command one more time. And upload the /var/log/mariadb/mariadb.log or /var/log/mysqld.log from the offloaded mysql server.