Host deletion and acknowledgements

[djobidon]

ls -ld /usr/local/nagios/var

drwxrwxr-x 6 nagios nagios 4096 Mar 17 11:47 /usr/local/nagios/var

ls -l /usr/local/nagios/var

drwxrwxr-x 2 nagios nagios 36864 Mar 16 23:59 archives
-rw-r--r-- 1 apache apache 14917 Feb 15 16:32 graphapi.log
-rw-r--r-- 1 nagios nagios 2807 Mar 17 11:48 host-perfdata
-rw-r--r-- 1 nagios nagios 19493 Mar 17 03:11 nagios.configtest
-rw-r--r-- 1 nagios nagios 5 Mar 15 13:37 nagios.lock
-rw-r--r-- 1 nagios nagios 348223 Mar 17 11:43 nagios.log
-rw-r--r-- 1 nagios nagios 5 Mar 15 13:37 ndo2db.lock
-rw-r--r-- 1 nagios nagios 0 Mar 15 13:35 ndomod.tmp
srwxr-xr-x 1 nagios nagios 0 Mar 15 13:37 ndo.sock
-rw-r--r-- 1 nagios nagios 9335464 Mar 17 05:08 npcd.log
-rw-r--r-- 1 nagios nagios 1442812 Mar 15 13:37 objects.cache
-rw-r--r-- 1 nagios nagios 1442812 Mar 17 03:11 objects.precache
-rw-rw-rw- 1 nagios nagios 4469021 Mar 17 05:08 perfdata.log
-rw------- 1 nagios nagios 2184078 Mar 17 11:37 retention.dat
drwxrwsr-x 2 nagios nagcmd 4096 Mar 15 13:37 rw
-rw-r--r-- 1 nagios nagios 5483 Mar 17 11:48 service-perfdata
drwxr-xr-x 5 nagios nagios 4096 Feb 12 2014 spool
drwxr-xr-x 2 nagios nagios 4096 Dec 29 2014 stats
-rw-rw-r-- 1 nagios nagios 2189920 Mar 17 11:48 status.dat

[ssax]

Are you seeing any crashed tables in your /var/log/mysqld.log?

[djobidon]

Lines since last start:

160315 13:35:52 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160315 13:37:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160315 13:37:00 InnoDB: Initializing buffer pool, size = 8.0M
160315 13:37:00 InnoDB: Completed initialization of buffer pool
160315 13:37:00 InnoDB: Started; log sequence number 0 142283
160315 13:37:00 [Note] Event Scheduler: Loaded 0 events
160315 13:37:00 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

However earlier I have the following bits:

150605 09:32:15 mysqld_safe Number of processes running now: 0
150605 09:32:27 mysqld_safe mysqld restarted
150605 9:33:13 InnoDB: Initializing buffer pool, size = 8.0M
150605 9:33:13 InnoDB: Completed initialization of buffer pool
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150605 9:33:13 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
150605 9:33:15 InnoDB: Started; log sequence number 0 142063
150605 9:33:16 [Note] Event Scheduler: Loaded 0 events
150605 9:33:16 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150605 10:27:03 mysqld_safe Number of processes running now: 0
150605 10:27:16 mysqld_safe mysqld restarted
150605 10:27:30 InnoDB: Initializing buffer pool, size = 8.0M
150605 10:27:30 InnoDB: Completed initialization of buffer pool
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150605 10:27:34 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
150605 10:27:37 InnoDB: Started; log sequence number 0 142063
150605 10:27:38 [Note] Event Scheduler: Loaded 0 events
150605 10:27:38 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160315 13:33:02 [Note] /usr/libexec/mysqld: Normal shutdown

[lmiltchev]

Even though you don't have any errors in the mysqld.log, you had some issues before.

150605 9:33:13 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.

Try running the database repair script to see if this is going to resolve your issue.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

[djobidon]

Ran database repair. Did a few acknowledgements then did apply configuration (I did not have any delete or host/service change to make). All acknoledgements I did were undone immediately.

[tgriep]

Can you login to the Xi server, run the following commands and post the output?

Code: Select all

cd /usr/local/nagiosxi/scripts
su nagios
./reconfigure_nagios.sh

Thanks.

[djobidon]

Here is the output

[tgriep]

Can you check the following files and see if your acknowledgements are in both files?

Code: Select all

/usr/local/nagios/var/status.dat
/usr/local/nagios/var/retention.dat

Can you post your full nagios.cfg file?

[djobidon]

Here is my result. I picked the first host I found that I repeatedly acknowledge

Code: Select all

action                in status.dat    in retention.dat
----------------------------------------------------
acknowledge an host   acknowledged=1   acknowledge=0
apply config          acknowledged=0   acknowledge=1
acknowledge same host acknowledged=1   acknowledge=1
apply config          acknowledged=0   acknowledge=1

[tmcdonald]

This is probably going to need a remote session. Please email [email protected] with a descriptive subject, and at the very least a link back to this thread (though a quick recap of the problem is always appreciated). We'll pick it up there, get a few more items from you, then if needed set up a WebEx.