Problem: Event handler runs in soft state

[parisa]

Hello
I have around 100 hosts and I defined around 25 services for each host in Nagios XI (around 2500 services.), I use SNMP Trap (passive check) to check our services.
Also I have defined "event handler" , "retry interval" and "max check attempts" for each service.
When I receive a trap for a service, the soft state of service changes but because of I have defined setting as below:
Retry interval: 10 mins
Max check attempts: 4
it's expected that soft state changes to hard state after 30 minutes if the alarm remained, then run event handler.

But I faced with a problem, the event handler run in soft state immediately.
is it true that event handler run in hard state even in passive check, I don't have any idea why this case happens.
would you please guide me about this?

[npolovenko]

Hello, @parisa. Event handlers run in soft and hard states.

When Are Event Handlers Executed?
Event handlers are executed when a service or host:

Is in a SOFT problem state
Initially goes into a HARD problem state
Initially recovers from a SOFT or HARD problem state

https://assets.nagios.com/downloads/nag ... dlers.html

If you want the event handler to ONLY execute on a hard critical state you can either:
Create a condition in the event handler script that checks whether the state is hard or soft before executing the main part of the script.
Or you can try assigning your event handler script as a notification command. Notification commands only activate of Hard states.

[parisa]

Thanks for your answer @npolovenko .
So, I should use "state_type" field from "nagios_servicestatus" table or "SERVICESTATETYPE$" macro in event-handler scripts to be able to recognize hard/soft state.

I have another question ,when an alarm happens and recovers before it goes to hard state (recovers in soft state), are state_type=soft in OK state of that service ?

[lmiltchev]

I have another question ,when an alarm happens and recovers before it goes to hard state (recovers in soft state), are state_type=soft in OK state of that service ?

Yes, this would be a soft recovery.

Quote from our official documentation:

When a service or host recovers from a soft error. This is considered a soft recovery.

https://assets.nagios.com/downloads/nag ... types.html

[parisa]

I faced with another reaction, I have a service that it has been configured as below:
Retry interval =1
Max check attempts=2
Active checks enabled=off
Active checks enabled=on

But based attached file, When the service recovered from a soft state error, OK state is hard. Would you please guide me about the case?

[lmiltchev]

You may see a "HARD" state in the GUI, but the service could've been is a "SOFT" state prior to this. A soft recovery is marked as SOFT;x;OK then it goes to HARD set to 1 in the interface. To be sure, you will need to check the nagios log.

Also, you posted this:

Retry interval =1
Max check attempts=2
Active checks enabled=off
Active checks enabled=on

What is the actual value of active check enabled - on or off? What is the version of Nagios XI that you are currently using?

[parisa]

Excuse me, I use passive check (SNMP Trap )the config is as below:
Retry interval =1
Max check attempts=2
Active checks enabled=off
Passive checks enabled=on

I have installed last version (5.5.11)

[lmiltchev]

There was a bug with soft recoveries, but since you are using the latest XI, the bug would be fixed on your system.

As I mentioned - "A soft recovery is marked as SOFT;x;OK then it goes to HARD set to 1 in the interface." Did you have a chance to review the nagios.log?

[parisa]

excuse me for my delay
I checked nagios.log and observe the same log.
related log is attached.

[lmiltchev]

Oh, I believe you are experiencing a different bug. You service never went to a hard non-ok state, but it recovered to a hard ok state on the very last check attempt (Max check attempts=2).

The issue is described here:

https://github.com/NagiosEnterprises/na ... issues/624