Freshness threshold not working during apply configuration

[WillemDH]

Hello,

I've seen it multiple times on different passive services. When a passive check is received with a configured freshness during an apply configuration, the freshness is not working. This seems like a bug to me and makes the freshness directive unreliable.

Grtz

Willem

[rkennedy]

Hey Willem - When you mention that the freshness is not working, are you referring to it skipping the active check that should be executing? We'll have to do some testing on our end, but just trying to understand what you're experiencing exactly.

[WillemDH]

Clint,

It is indeed like you say, the active check (check_dummy) is not executing, resulting in the critical state not resetting. We have two apply configurations configured with Rundeck, one at 07:00 and one at 19:00. I've got some passive services set up with a freshness threshold of 60 seconds which multiple passive checks a day. When I notice these services popup in the open service problem for longer then a minute it's always at 07:00 and 19:00 or duirng a manual apply.

Grtz

Willem

[ssax]

What do you have set for the services in your /usr/local/nagios/var/objects.cache OR /var/nagiosramdisk/objects.cache? I'm wondering about the Retain status information (retain_status_information) and Retain non-status information (retain_nonstatus_information) settings.

Thank you

[WillemDH]

Sean,

Here you go:

cat /var/nagiosramdisk/objects.cache | sed -rn "/define service \{/{:a;N;/}/{/.host_name.cashpc0003/p;d};ba}"

This is one of the passive services giving me issues:

Code: Select all

define service {
        host_name       cashpc0003
        service_description     EVT_Cash_Quota
        display_name    EVT_System
        check_period    xi_timeperiod_24x7
        check_command   check_dummy!0!"Dummy check passed"!!!!!!
        contacts        nagiosadmin,nagiosadmin
        contact_groups  cg_dummy
        notification_period     xi_timeperiod_24x7
        initial_state   o
        importance      0
        check_interval  1440.000000
        retry_interval  1.000000
        max_check_attempts      1
        is_volatile     0
        parallelize_check       1
        active_checks_enabled   1
        passive_checks_enabled  1
        obsess  1
        event_handler_enabled   1
        low_flap_threshold      0.000000
        high_flap_threshold     0.000000
        flap_detection_enabled  0
        flap_detection_options  a
        freshness_threshold     60
        check_freshness 1
        notification_options    a
        notifications_enabled   1
        notification_interval   1440.000000
        first_notification_delay        0.000000
        stalking_options        o,w,u,c
        process_perf_data       0
        icon_image      windowseventlog.png
        retain_status_information       1
        retain_nonstatus_information    1
        _XIWIZARD       windowseventlog
        }

Grtz

Willem

[tgriep]

How many host and service checks are setup on this server and what version of XI is it running.
For a test, can you increase the freshness threshold to 120 seconds and see if the issue goes away or is reduced?
I am thinking that the checks that are causing the issue may be close to the 60 seconds threshold and then the Apply Config is run and during the time it takes to run that, the 60 seconds are up and the active check doesn't happen.
Also, do you see any messages in the /usr/local/nagios/var/nagios.log file at those times?

[WillemDH]

+- 21k hosts and svces on 5.4. I also think exactly what you are saying is happening. But I still consider this a bug... We expect the freshness to just do the work its configured to do... Enfin I'll set it higher, but that is not exactly what we need. These checks aren't supposed to show up on our dashboards. They just need to send an email alert when coming in, and then reset immediately.

[tgriep]

Thanks for trying this out for us. Changing those settings and the issue is resolves or lessens, would give us a hint on where to start to troubleshoot this.