Re: [Nagios-devel] nagios 3.2.3 localtime deadlock
Posted: Tue Oct 12, 2010 4:44 pm
On Fri, Oct 8, 2010 at 9:08 AM, Matthew Kent wrote:
> On Fri, Oct 8, 2010 at 3:52 AM, Andreas Ericsson wrote:
>> On 10/07/2010 08:43 PM, Matthew Kent wrote:
>>> Hello all,
>>>
>>
>> Hey you. First of all, thanks for including a backtrace. That's really
>> neat.
>>
>
> Thanks for looking
>
>>> Setting up a new nagios 3.2.3 install and occasionally (once in 24
>>> hours) I'm seeing a child deadlock when calling localtime() like so:
>>>
>>> (gdb) bt
>>> #0 =A00x00000033d5edfade in __lll_lock_wait_private () from /lib64/libc=
.so.6
>>> #1 =A00x00000033d5e8d1cd in _L_lock_1685 () from /lib64/libc.so.6
>>> #2 =A00x00000033d5e8cf17 in __tz_convert () from /lib64/libc.so.6
>>> #3 =A00x000000000043e23e in get_datetime_string (raw_time=3D>> optimized out>, buffer=3D0x2aaab014feb0,
>>> buffer_length=3D48, type=3D0) at utils.c:1696
>>> #4 =A00x0000000000430990 in grab_datetime_macro (macro_type=3D7, arg1=
=3D0x0,
>>> arg2=3D0x0, output=3D0x6998f8) at ../common/macros.c:1533
>>> #5 =A00x0000000000432cbf in grab_macrox_value (macro_type=3D-4, arg1=3D=
0x0,
>>> arg2=3D0x0, output=3D0x6998f8, free_macro=3D0x2) at ../common/macros.c:=
1089
>>> #6 =A00x0000000000433586 in set_macrox_environment_vars (set=3D1) at
>>> ../common/macros.c:3166
>>> #7 =A00x00000000004335bb in set_all_macro_environment_vars (set=3D1) at
>>> ../common/macros.c:3134
>>> #8 =A00x000000000041b4c3 in run_async_service_check (svc=3D0x8d62560,
>>> check_options=3D, latency=3D,
>>> scheduled_check=3D1, reschedule_check=3D1,
>>> =A0 =A0 =A0time_is_valid=3D, preferred_time=3D>> optimized out>) at checks.c:658
>>> #9 =A00x000000000041d56d in run_scheduled_service_check (svc=3D0x8d6256=
0,
>>> check_options=3D0, latency=3D0.68999999999999995) at checks.c:260
>>> #10 0x000000000042a45a in handle_timed_event (event=3D0x2aaab011af30) a=
t
>>> events.c:1257
>>> #11 0x000000000042abe6 in event_execution_loop () at events.c:1143
>>> #12 0x0000000000413055 in main (argc=3D,
>>> argv=3D, env=3D0x7fffa0670758) at nagios.c:850
>>>
>>> this leads to Nagios being completely frozen until I manually kill the =
child.
>>>
>>
>>
>> Looking at the glibc code, I see no possible way that a single thread
>> can hold on to the lock in __tz_convert() for any extended period of
>> time. What version of glibc are you using?
>>
>
> glibc-2.5-49.el5_5.4.x86_64
>
>>> Some light Googling tells me this can happen with localtime in certain
>>> cases, but I see no indication of other people with this issue in
>>> Nagios.
>>>
>>
>> Since this seems to happen in the codepath that exports macros as
>> environment variables, I'd like to know if it happens if you turn
>> that stuff off. Unless you really, really need it it's a good idea
>> to do that anyways, since computing a bazillion macros each time
>> Nagios runs a check is quite expensive. Set
>>
>> =A0use_large_installation_tweaks=3D1
>> or
>> =A0enable_environment_macros=3D0
>>
>> in your nagios.cfg file.
>>
>> use_large_installation_tweaks=3D1 is a really good idea anyways unless
>> you're running Nagios on Windows 95, where a process' used memory
>> was never reclaimed by the system unless manually free()'d.
>>
>
> Yeah we don't even use the environment variables. Thanks for all the info=
.
>
>>> It's a pretty standard Nagios install on CentOS 5.5 - except for the
>>> fact I'm using the mk-livestatus event broker. We have a couple
>>> thousand checks configured on a pretty aggressive interval.
>>>
>>
>> First try disabling environment macros. Then try without the
>> mk-livestatus module. Seeing it happen in a pristine Nagios would mean
>> we don't need to speculate about where the problem happens.
>
> Good call, I'll disable the env macros and run it over the weekend,
> then reenable them and with livestatus off for good measure and report
> back here. We'll see what happens!
>
Oops, this was originally supposed to go to the list.
For the record setting
enable_e
...[email truncated]...
This post was automatically imported from historical nagios-devel mailing list archives
Original poster: [email protected]
> On Fri, Oct 8, 2010 at 3:52 AM, Andreas Ericsson wrote:
>> On 10/07/2010 08:43 PM, Matthew Kent wrote:
>>> Hello all,
>>>
>>
>> Hey you. First of all, thanks for including a backtrace. That's really
>> neat.
>>
>
> Thanks for looking
>
>>> Setting up a new nagios 3.2.3 install and occasionally (once in 24
>>> hours) I'm seeing a child deadlock when calling localtime() like so:
>>>
>>> (gdb) bt
>>> #0 =A00x00000033d5edfade in __lll_lock_wait_private () from /lib64/libc=
.so.6
>>> #1 =A00x00000033d5e8d1cd in _L_lock_1685 () from /lib64/libc.so.6
>>> #2 =A00x00000033d5e8cf17 in __tz_convert () from /lib64/libc.so.6
>>> #3 =A00x000000000043e23e in get_datetime_string (raw_time=3D>> optimized out>, buffer=3D0x2aaab014feb0,
>>> buffer_length=3D48, type=3D0) at utils.c:1696
>>> #4 =A00x0000000000430990 in grab_datetime_macro (macro_type=3D7, arg1=
=3D0x0,
>>> arg2=3D0x0, output=3D0x6998f8) at ../common/macros.c:1533
>>> #5 =A00x0000000000432cbf in grab_macrox_value (macro_type=3D-4, arg1=3D=
0x0,
>>> arg2=3D0x0, output=3D0x6998f8, free_macro=3D0x2) at ../common/macros.c:=
1089
>>> #6 =A00x0000000000433586 in set_macrox_environment_vars (set=3D1) at
>>> ../common/macros.c:3166
>>> #7 =A00x00000000004335bb in set_all_macro_environment_vars (set=3D1) at
>>> ../common/macros.c:3134
>>> #8 =A00x000000000041b4c3 in run_async_service_check (svc=3D0x8d62560,
>>> check_options=3D, latency=3D,
>>> scheduled_check=3D1, reschedule_check=3D1,
>>> =A0 =A0 =A0time_is_valid=3D, preferred_time=3D>> optimized out>) at checks.c:658
>>> #9 =A00x000000000041d56d in run_scheduled_service_check (svc=3D0x8d6256=
0,
>>> check_options=3D0, latency=3D0.68999999999999995) at checks.c:260
>>> #10 0x000000000042a45a in handle_timed_event (event=3D0x2aaab011af30) a=
t
>>> events.c:1257
>>> #11 0x000000000042abe6 in event_execution_loop () at events.c:1143
>>> #12 0x0000000000413055 in main (argc=3D,
>>> argv=3D, env=3D0x7fffa0670758) at nagios.c:850
>>>
>>> this leads to Nagios being completely frozen until I manually kill the =
child.
>>>
>>
>>
>> Looking at the glibc code, I see no possible way that a single thread
>> can hold on to the lock in __tz_convert() for any extended period of
>> time. What version of glibc are you using?
>>
>
> glibc-2.5-49.el5_5.4.x86_64
>
>>> Some light Googling tells me this can happen with localtime in certain
>>> cases, but I see no indication of other people with this issue in
>>> Nagios.
>>>
>>
>> Since this seems to happen in the codepath that exports macros as
>> environment variables, I'd like to know if it happens if you turn
>> that stuff off. Unless you really, really need it it's a good idea
>> to do that anyways, since computing a bazillion macros each time
>> Nagios runs a check is quite expensive. Set
>>
>> =A0use_large_installation_tweaks=3D1
>> or
>> =A0enable_environment_macros=3D0
>>
>> in your nagios.cfg file.
>>
>> use_large_installation_tweaks=3D1 is a really good idea anyways unless
>> you're running Nagios on Windows 95, where a process' used memory
>> was never reclaimed by the system unless manually free()'d.
>>
>
> Yeah we don't even use the environment variables. Thanks for all the info=
.
>
>>> It's a pretty standard Nagios install on CentOS 5.5 - except for the
>>> fact I'm using the mk-livestatus event broker. We have a couple
>>> thousand checks configured on a pretty aggressive interval.
>>>
>>
>> First try disabling environment macros. Then try without the
>> mk-livestatus module. Seeing it happen in a pristine Nagios would mean
>> we don't need to speculate about where the problem happens.
>
> Good call, I'll disable the env macros and run it over the weekend,
> then reenable them and with livestatus off for good measure and report
> back here. We'll see what happens!
>
Oops, this was originally supposed to go to the list.
For the record setting
enable_e
...[email truncated]...
This post was automatically imported from historical nagios-devel mailing list archives
Original poster: [email protected]