passive service history retention

[WillemDH]

Hello,

i'm 100% sure last week we had several passive events in one of my passive services which catches realtime Windows events. Yesterday we rebooted this server in the hope the issue causing the passive events to be solved. So today I wanted to check the service history of this passive service. But it seems the history is completely empty. What could have caused this?

See screenshots for the config of the passive service.

Grtz

Willem

[ssax]

Are they missing from all your passive services or just this one?

[WillemDH]

Just this one it seems.

[jolson]

Please run the following on your Nagios box:

Code: Select all

mysql -pnagiosxi nagios
select host_object_id,alias,display_name from nagios_hosts;
select object_id,output,statehistory_id,last_state,last_hard_state from nagios_statehistory where object_id = <hostobjectID>;

Please replace <hostobjectID> with the hosts object ID that is in question. Once you pull up that table, look for any states related to the passive check. Do any exist?

[WillemDH]

Getting access denied. Also when I try with root pw....

Code: Select all

mysql -pnagiosxi nagios
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

[jolson]

Can you please perform the following procedure to reset your root database password:

Before doing anything, stop mysql:

Code: Select all

service mysqld stop

Start MySQL in safe mode:

Code: Select all

 mysqld_safe --skip-grant-tables

In a different terminal, log into mysql as root, using the mysql DB:

Code: Select all

mysql mysql 

In the MySQL Shell, reset the root DB password:

Code: Select all

update user set Password=PASSWORD('new-password') where user='root';
flush privileges;
exit;

The FLUSH statement tells the server to reload the grant tables into memory so that it notices the password change.
Stop the server, then restart it normally:

Code: Select all

service mysqld stop
service mysqld start

http://dev.mysql.com/doc/refman/5.0/en/ ... sions.html

This does not have any impact on the functionality of your system, I made sure to test the results in a lab quickly.

EDIT:
There is one password you will need to change - it's located in the following file:

Code: Select all

vi /root/scripts/automysqlbackup

You will need to change the 'default' password to your newly created root password.

[WillemDH]

Jolson,

Just one question before I start following your procedure. Why would I suddenly need to reset this password? I don't get it. My backup is working correctly etc. I don't understand.

Grtz

Willem

[lmiltchev]

Just one question before I start following your procedure. Why would I suddenly need to reset this password? I don't get it. My backup is working correctly etc. I don't understand.

You don't have to do this. I believe what jolson was saying is that if you changed the root mysql password, you would also need to change it in the "/root/scripts/automysqlbackup" script. If you know the "new" root password, use it to obtain the info.

Code: Select all

mysql -p<you new password> nagios
select host_object_id,alias,display_name from nagios_hosts;
select object_id,output,statehistory_id,last_state,last_hard_state from nagios_statehistory where object_id = <hostobjectID>;

If you don't know your new password, you can try:

Code: Select all

mysql -u ndoutils -pn@gweb nagios
select host_object_id,alias,display_name from nagios_hosts;
select object_id,output,statehistory_id,last_state,last_hard_state from nagios_statehistory where object_id = <hostobjectID>;
\q

[WillemDH]

Ah,

Ty Ludmill, my problem was that I copy pasted Jolson's command not knowing that the password had to be right after the p. he should have said

Code: Select all

mysql -p<mysqlpassword> nagios 

or sth similar.

Ok, the output of the third command:

Code: Select all

mysql> select object_id,output,statehistory_id,last_state,last_hard_state from nagios_statehistory where object_id = 6989;
+-----------+-----------------------------------------------------------------------------+-----------------+------------+-----------------+
| object_id | output                                                                      | statehistory_id | last_state | last_hard_state |
+-----------+-----------------------------------------------------------------------------+-----------------+------------+-----------------+
|      6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% |          476522 |          0 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% |          476528 |          0 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% |          476538 |          0 |              -1 |
|      6989 | OK - 10.10.10.101: rta 2.167ms, lost 0%                                     |          476564 |          1 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% |          483689 |          0 |              -1 |
|      6989 | OK - 10.10.10.101: rta 0.711ms, lost 0%                                     |          483700 |          0 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: rta nan, lost 100%                                 |          506205 |          0 |              -1 |
|      6989 | OK - 10.10.10.101: rta 0.185ms, lost 0%                                     |          506206 |          0 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% |          567717 |          0 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% |          567736 |          0 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% |          567749 |          0 |              -1 |
|      6989 | OK - 10.10.10.101: rta 0.442ms, lost 0%                                     |          570042 |          1 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: rta nan, lost 100%                                 |          589962 |          0 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% |          589964 |          0 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% |          589969 |          0 |              -1 |
|      6989 | OK - 10.10.10.101: rta 2.116ms, lost 0%                                     |          589972 |          1 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: rta nan, lost 100%                                 |          735457 |          0 |              -1 |
|      6989 | OK - 10.10.10.101: rta 0.272ms, lost 0%                                     |          735458 |          0 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: rta nan, lost 100%                                 |          758845 |          0 |              -1 |
|      6989 | OK - 10.10.10.101: rta 0.385ms, lost 0%                                     |          758846 |          0 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: rta nan, lost 100%                                 |          913363 |          0 |              -1 |
|      6989 | OK - 10.10.10.101: rta 0.348ms, lost 0%                                     |          913365 |          0 |              -1 |
|      6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% |         1051750 |          0 |              -1 |
|      6989 | OK - 10.10.10.101: rta 0.344ms, lost 0%                                     |         1051752 |          0 |              -1 |
+-----------+-----------------------------------------------------------------------------+-----------------+------------+-----------------+
24 rows in set (0.00 sec)

It seems this command only shows the state history of the host? The history I was missing was that of one service of this host, named EVT_Application.

Grtz

Willem

[WillemDH]

Ok.. My I discovered this server is a 2003 and gpo events are stored in System eventlog there it seems. So I was looking in the wrong service history. Sorry!

But while we are talking about passive services, I've got another question. The json query to get recentalerts of the last 24 hours I'm doing on another server is giving me more results then the service history in Nagios. I vaguely remember some passive service setting to not overwrite the service if the state was already non healthy. You can see the settings of my passive service in one of my earlier posts.
if the json query to get recent alerts is showing more information then the service history in Nagios, this is kind of confusing.

This is the json query:

Code: Select all

query=alertlist&objecttypes=host+service&statetypes=hard&servicestates=warning+critical+unknown&hostname=<servername>&starttime=$Start&endtime=$End") 

The results you can see in the attachments.
So what setting was it again to overwrite passive services? And is there a way to edit the json query so it shows the same results as Nagios XI service history?

Grtz