livestatus VERY slow
Posted: Mon Sep 30, 2019 10:34 am
Hi,
we are using from years nagioscore + livestatus with success on our big linux clusters, and a problem arose just in the last cluster.
This is our last installation:
Nagios Core 4.4.3
Livestatus 1.5.0p11
We have to to this simple query:
[root@CLUSTER1 ~]# cat query_services
GET services
Columns: host_name description last_hard_state host_groups scheduled_downtime_depth host_scheduled_downtime_depth acknowledged host_acknowledged active_checks_enabled plugin_output host_downtimes downtimes state state_type current_attempt max_check_attempts notes_expanded contact_groups check_command
It works, but it take too long, 100x the usual time:
[root@CLUSTER1 ~]# time cat query_services | /usr/local/bin/unixcat /install/nagios/var/rw/live | wc -l
13902
real 0m11.751s
user 0m0.001s
sys 0m0.008s
If I remove the field "notes_expanded" in the query we obtain the usual timing, around 0.1 seconds:
[root@CLUSTER1 ~]# cat query_services-without-notes_expanded
GET services
Columns: host_name description last_hard_state host_groups scheduled_downtime_depth host_scheduled_downtime_depth acknowledged host_acknowledged active_checks_enabled plugin_output host_downtimes downtimes state state_type current_attempt max_check_attempts contact_groups check_command
[root@CLUSTER1 ~]# time cat query_services-without-notes_expanded | /usr/local/bin/unixcat /install/nagios/var/rw/live | wc -l
13902
real 0m0.130s
user 0m0.001s
sys 0m0.009s
Any tip? Is a nagios or livestatus problem?
Thanks,
Federico
PS: in other installations, for example
Nagios Core 4.3.2
livestatus-1.2.6p16
all works in reasonable time: the presence of notes_expanded triple the query time, not make it 100x!!
[root@CLUSTER2 ~]# time cat query_services | /usr/local/bin/unixcat /install/nagios/var/rw/live | wc -l
51553
real 0m0.469s
user 0m0.003s
sys 0m0.026s
[root@master02(A3) ~]# time cat query_services-without-notes_expanded | /usr/local/bin/unixcat /install/nagios/var/rw/live | wc -l
51553
real 0m0.176s
user 0m0.004s
sys 0m0.013s
we are using from years nagioscore + livestatus with success on our big linux clusters, and a problem arose just in the last cluster.
This is our last installation:
Nagios Core 4.4.3
Livestatus 1.5.0p11
We have to to this simple query:
[root@CLUSTER1 ~]# cat query_services
GET services
Columns: host_name description last_hard_state host_groups scheduled_downtime_depth host_scheduled_downtime_depth acknowledged host_acknowledged active_checks_enabled plugin_output host_downtimes downtimes state state_type current_attempt max_check_attempts notes_expanded contact_groups check_command
It works, but it take too long, 100x the usual time:
[root@CLUSTER1 ~]# time cat query_services | /usr/local/bin/unixcat /install/nagios/var/rw/live | wc -l
13902
real 0m11.751s
user 0m0.001s
sys 0m0.008s
If I remove the field "notes_expanded" in the query we obtain the usual timing, around 0.1 seconds:
[root@CLUSTER1 ~]# cat query_services-without-notes_expanded
GET services
Columns: host_name description last_hard_state host_groups scheduled_downtime_depth host_scheduled_downtime_depth acknowledged host_acknowledged active_checks_enabled plugin_output host_downtimes downtimes state state_type current_attempt max_check_attempts contact_groups check_command
[root@CLUSTER1 ~]# time cat query_services-without-notes_expanded | /usr/local/bin/unixcat /install/nagios/var/rw/live | wc -l
13902
real 0m0.130s
user 0m0.001s
sys 0m0.009s
Any tip? Is a nagios or livestatus problem?
Thanks,
Federico
PS: in other installations, for example
Nagios Core 4.3.2
livestatus-1.2.6p16
all works in reasonable time: the presence of notes_expanded triple the query time, not make it 100x!!
[root@CLUSTER2 ~]# time cat query_services | /usr/local/bin/unixcat /install/nagios/var/rw/live | wc -l
51553
real 0m0.469s
user 0m0.003s
sys 0m0.026s
[root@master02(A3) ~]# time cat query_services-without-notes_expanded | /usr/local/bin/unixcat /install/nagios/var/rw/live | wc -l
51553
real 0m0.176s
user 0m0.004s
sys 0m0.013s