Page 1 of 2
passive service history retention
Posted: Thu Mar 12, 2015 7:31 am
by WillemDH
Hello,
i'm 100% sure last week we had several passive events in one of my passive services which catches realtime Windows events. Yesterday we rebooted this server in the hope the issue causing the passive events to be solved. So today I wanted to check the service history of this passive service. But it seems the history is completely empty. What could have caused this?
See screenshots for the config of the passive service.
Grtz
Willem
Re: passive service history retention
Posted: Thu Mar 12, 2015 9:03 am
by ssax
Are they missing from all your passive services or just this one?
Re: passive service history retention
Posted: Thu Mar 12, 2015 9:49 am
by WillemDH
Just this one it seems.
Re: passive service history retention
Posted: Thu Mar 12, 2015 10:18 am
by jolson
Please run the following on your Nagios box:
Code: Select all
mysql -pnagiosxi nagios
select host_object_id,alias,display_name from nagios_hosts;
select object_id,output,statehistory_id,last_state,last_hard_state from nagios_statehistory where object_id = <hostobjectID>;
Please replace <hostobjectID> with the hosts object ID that is in question. Once you pull up that table, look for any states related to the passive check. Do any exist?
Re: passive service history retention
Posted: Thu Mar 12, 2015 11:50 am
by WillemDH
Getting access denied. Also when I try with root pw....
Code: Select all
mysql -pnagiosxi nagios
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
Re: passive service history retention
Posted: Thu Mar 12, 2015 1:24 pm
by jolson
Can you please perform the following procedure to reset your root database password:
Before doing anything, stop mysql:
Start MySQL in safe mode:
In a different terminal, log into mysql as root, using the mysql DB:
In the MySQL Shell, reset the root DB password:
Code: Select all
update user set Password=PASSWORD('new-password') where user='root';
flush privileges;
exit;
The FLUSH statement tells the server to reload the grant tables into memory so that it notices the password change.
Stop the server, then restart it normally:
Code: Select all
service mysqld stop
service mysqld start
This does not have any impact on the functionality of your system, I made sure to test the results in a lab quickly.
EDIT:
There is one password you will need to change - it's located in the following file:
You will need to change the 'default' password to your newly created root password.
Re: passive service history retention
Posted: Thu Mar 12, 2015 3:31 pm
by WillemDH
Jolson,
Just one question before I start following your procedure. Why would I suddenly need to reset this password? I don't get it. My backup is working correctly etc. I don't understand.
Grtz
Willem
Re: passive service history retention
Posted: Thu Mar 12, 2015 3:49 pm
by lmiltchev
Just one question before I start following your procedure. Why would I suddenly need to reset this password? I don't get it. My backup is working correctly etc. I don't understand.
You don't have to do this. I believe what jolson was saying is that if you changed the root mysql password, you would also need to change it in the "/root/scripts/automysqlbackup" script. If you know the "new" root password, use it to obtain the info.
Code: Select all
mysql -p<you new password> nagios
select host_object_id,alias,display_name from nagios_hosts;
select object_id,output,statehistory_id,last_state,last_hard_state from nagios_statehistory where object_id = <hostobjectID>;
If you don't know your new password, you can try:
Code: Select all
mysql -u ndoutils -pn@gweb nagios
select host_object_id,alias,display_name from nagios_hosts;
select object_id,output,statehistory_id,last_state,last_hard_state from nagios_statehistory where object_id = <hostobjectID>;
\q
Re: passive service history retention
Posted: Fri Mar 13, 2015 2:52 am
by WillemDH
Ah,
Ty Ludmill, my problem was that I copy pasted Jolson's command not knowing that the password had to be right after the p. he should have said
or sth similar.
Ok, the output of the third command:
Code: Select all
mysql> select object_id,output,statehistory_id,last_state,last_hard_state from nagios_statehistory where object_id = 6989;
+-----------+-----------------------------------------------------------------------------+-----------------+------------+-----------------+
| object_id | output | statehistory_id | last_state | last_hard_state |
+-----------+-----------------------------------------------------------------------------+-----------------+------------+-----------------+
| 6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% | 476522 | 0 | -1 |
| 6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% | 476528 | 0 | -1 |
| 6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% | 476538 | 0 | -1 |
| 6989 | OK - 10.10.10.101: rta 2.167ms, lost 0% | 476564 | 1 | -1 |
| 6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% | 483689 | 0 | -1 |
| 6989 | OK - 10.10.10.101: rta 0.711ms, lost 0% | 483700 | 0 | -1 |
| 6989 | CRITICAL - 10.10.10.101: rta nan, lost 100% | 506205 | 0 | -1 |
| 6989 | OK - 10.10.10.101: rta 0.185ms, lost 0% | 506206 | 0 | -1 |
| 6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% | 567717 | 0 | -1 |
| 6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% | 567736 | 0 | -1 |
| 6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% | 567749 | 0 | -1 |
| 6989 | OK - 10.10.10.101: rta 0.442ms, lost 0% | 570042 | 1 | -1 |
| 6989 | CRITICAL - 10.10.10.101: rta nan, lost 100% | 589962 | 0 | -1 |
| 6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% | 589964 | 0 | -1 |
| 6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% | 589969 | 0 | -1 |
| 6989 | OK - 10.10.10.101: rta 2.116ms, lost 0% | 589972 | 1 | -1 |
| 6989 | CRITICAL - 10.10.10.101: rta nan, lost 100% | 735457 | 0 | -1 |
| 6989 | OK - 10.10.10.101: rta 0.272ms, lost 0% | 735458 | 0 | -1 |
| 6989 | CRITICAL - 10.10.10.101: rta nan, lost 100% | 758845 | 0 | -1 |
| 6989 | OK - 10.10.10.101: rta 0.385ms, lost 0% | 758846 | 0 | -1 |
| 6989 | CRITICAL - 10.10.10.101: rta nan, lost 100% | 913363 | 0 | -1 |
| 6989 | OK - 10.10.10.101: rta 0.348ms, lost 0% | 913365 | 0 | -1 |
| 6989 | CRITICAL - 10.10.10.101: Host unreachable @ 10.54.24.99. rta nan, lost 100% | 1051750 | 0 | -1 |
| 6989 | OK - 10.10.10.101: rta 0.344ms, lost 0% | 1051752 | 0 | -1 |
+-----------+-----------------------------------------------------------------------------+-----------------+------------+-----------------+
24 rows in set (0.00 sec)
It seems this command only shows the state history of the host? The history I was missing was that of one service of this host, named EVT_Application.
Grtz
Willem
Re: passive service history retention
Posted: Fri Mar 13, 2015 3:39 am
by WillemDH
Ok.. My I discovered this server is a 2003 and gpo events are stored in System eventlog there it seems. So I was looking in the wrong service history. Sorry!
But while we are talking about passive services, I've got another question. The json query to get recentalerts of the last 24 hours I'm doing on another server is giving me more results then the service history in Nagios. I vaguely remember some passive service setting to not overwrite the service if the state was already non healthy. You can see the settings of my passive service in one of my earlier posts.
if the json query to get recent alerts is showing more information then the service history in Nagios, this is kind of confusing.
This is the json query:
Code: Select all
query=alertlist&objecttypes=host+service&statetypes=hard&servicestates=warning+critical+unknown&hostname=<servername>&starttime=$Start&endtime=$End")
The results you can see in the attachments.
So what setting was it again to overwrite passive services? And is there a way to edit the json query so it shows the same results as Nagios XI service history?
Grtz