xi 5.4.12 event log disappeared

[benhank]

What a mystery. all of the xi event logs on my server have disappeared.
If I go to Admin/monitoring processes/ eventlog
and do a search from any time period nothing shows up.

[npolovenko]

Hello, @benhank. Did anyone truncate the nagios_logentries table in mysql? What's the output of:

Code: Select all

echo "SELECT * FROM nagios_logentries;" | mysql -u root -pnagiosxi nagios

Also, please run this command and let me know if event log entries come back.

Code: Select all

echo 'repair table nagios_logentries use_frm;' | mysql -t -u root -pnagiosxi nagios

[benhank]

I ran it and got the following but the data has not returned

Code: Select all

  echo 'repair table nagios_logentries use_frm;' | m                                 ysql -t -u root -pnagiosxi nagios
+--------------------------+--------+----------+--------------------------------                                 -------+
| Table                    | Op     | Msg_type | Msg_text                                                               |
+--------------------------+--------+----------+--------------------------------                                 -------+
| nagios.nagios_logentries | repair | warning  | Number of rows changed from 0 t                                 o 9146 |
| nagios.nagios_logentries | repair | status   | OK                                                                     |
+--------------------------+--------+----------+--------------------------------                                 -------+

[npolovenko]

@benhank, Please take a screenshot of the empty event log page in XI and upload it in this thread.

Could you send in your Nagios XI System Profile so I can review it?
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and send it to me in a private message.

[benhank]

sent =D

And no we didn't truncate anything.

[npolovenko]

@benhank, I'm still seeing entries related to a corrupted MySQL table.
Did you run the command I gave you on the server that hosts the offloaded database?
Please run this command a couple more times on the offloaded server:

echo 'repair table nagios_logentries use_frm;' | mysql -t -u root -pnagiosxi nagios

After that please run the following commands in order:

Code: Select all

service crond stop
service npcd stop
service nagios stop
service ndo2db stop
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
service mysqld restart
service ndo2db start
service nagios start
service npcd start
service crond start

Close the browser and reload the web page.

If this solution doesn't work, you may need to restore from a previous DB backup or restore just the logentries table. Can you show me the output of:

Code: Select all

 ls -lR /store/backups/

[benhank]

sigh, no I didnt run it on the offloaded server, because I forgot it was offloaded, just ran it with the following results:

Code: Select all

root@lkenfusionp01 ~]# echo 'repair table nagios_logentries use_frm;' | mysql -t -u root -pnagiosxi nagios
+--------------------------+--------+----------+-------------------------------------------------------------+
| Table                    | Op     | Msg_type | Msg_text                                                    |
+--------------------------+--------+----------+-------------------------------------------------------------+
| nagios.nagios_logentries | repair | error    | Can't create new tempfile: './nagios/nagios_logentries.TMD' |
| nagios.nagios_logentries | repair | status   | Operation failed     

[npolovenko]

@benhank, Your repair command didn't work. Is the offloaded MySQL server running out of space?

Code: Select all

df -h

Also, try moving the nagios_logentries.TMD to a backup directory and rerunning the repair command one more time:

Code: Select all

mv /nagios/nagios_logentries.TMD /tmp/

[benhank]

yes im running it on the offloaded server (this time). The server isnt running out of space

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_lkenfusiondev-lv_root
                       96G   14G   78G  16% /
tmpfs                 7.8G   80K  7.8G   1% /dev/shm
/dev/sda1             477M  130M  322M  29% /boot
/dev/mapper/vg_lkenfusiondev-lv_home
                      446G  4.0G  420G   1% /home
/dev/mapper/vg_fusiondev900-lv_nagdb
                      413G   71M  392G   1% /nagdb
/dev/mapper/vg_fusiondev900-lv_perf
                      413G   71M  392G   1% /perf
[root@lkenfusionp01 nagios]#

and

Code: Select all

 mv /nagios/nagios_logentries.TMD /tmp/
mv: cannot stat `/nagios/nagios_logentries.TMD': No such file or directory
[root@lkenfusionp01 ~]# cd /usr/local/nagios/
[root@lkenfusionp01 nagios]#  mv /nagios/nagios_logentries.TMD /tmp/
mv: cannot stat `/nagios/nagios_logentries.TMD': No such file or directory

[npolovenko]

@benhank, Did you run the following commands I provided earlier?

service crond stop
service npcd stop
service nagios stop
service ndo2db stop
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
service mysqld restart
service ndo2db start
service nagios start
service npcd start
service crond start

You have duplicate nagios processes running and these commands should get rid of it.

And please try the following command to move the .TMD table:

mv /var/lib/mysql/nagios/nagios_logentries.TMD /tmp/

When done, please run the repair table command one more time. And upload the /var/log/mariadb/mariadb.log or /var/log/mysqld.log from the offloaded mysql server.