SERVICE ALERT shows in nagios log sporadically

[jagden]

I am using a command line to push events to a passive Nagios service, for example:

echo "['date +%s'] PROCESS_SERVICE_CHECK_RESULT;myhost;mytstsvc;2;This is an error" > /usr/local/nagios/var/rw/nagios.cmd

When I'm doing that, the status is always updated in Nagios UI and I can see the reason text and state updated fine.
However, in the Nagios log, 99% of the times, I'm not seeing anything when sending this event.

Sometimes I see:

[1446285712] SERVICE ALERT: myhost;mytstsvc;CRITICAL;HARD;3;This is an error

The problem is, that from GUI I cannot this way track older events or notifications for this service if only once in a while it outputs information when getting my passive alarm.

in nagios.cfg:

log_notifications = 1

I can also see notifications and alerts from other "active" services.

Any idea ?

[Box293]

I believe the source of your error is because your command does not end with a new line \n

I think it needs to be:

Code: Select all

echo "['date +%s'] PROCESS_SERVICE_CHECK_RESULT;myhost;mytstsvc;2;This is an error\n" > /usr/local/nagios/var/rw/nagios.cmd

[jagden]

Nope, did not help. Same problem.

[Box293]

Ahhh my mistake on that sorry.

In nagios.cfg you need:

Code: Select all

log_passive_checks=1

https://assets.nagios.com/downloads/nag ... gmain.html

[jagden]

It is on already.

[Box293]

Lets say the service is currently in an OK state (0).
If you submit a passive with a CRITICAL state (2) and then another one shortly thereafter as CRITICAL again, does it log it both times or only the first time?

The service might need to be defined as volatile is_volatile=1 in the service definition so that it gets logged each time.

The service may also need flapping disabled flap_detection_enabled=0 however I don't think that would affect the logging.

[jagden]

It has already.

Here is how it is defined:

templates.cfg:

define service{
name passive-service
active_checks_enabled 0
passive_checks_enabled 1
parallelize_check 1
notifications_enabled 1
event_handler_enabled 1
register 0
is_volatile 1
parallelize_check 1
obsess_over_service 1
check_freshness 0
flap_detection_enabled 0
failure_prediction_enabled 1
process_perf_data 1
retain_status_information 1
retain_nonstatus_information 1
check_period 24x7
max_check_attempts 3
normal_check_interval 10
retry_check_interval 2
contact_groups admins
notification_options w,u,c,r
notification_interval 10
notification_period 24x7
}


services.cfg:

define service {
use passive-service
host_name myhost
service_description tst2
check_command check_dummy
#check_command check_nrpe!tst
passive_checks_enabled 1
contact_groups dba-alert-group
contacts monitoring-mobile
normal_check_interval 1
retry_check_interval 5
register 1
}

[tgriep]

What version of Core are you running and can you post your full nagios.cfg file?

[jagden]

Nagios Core version: 3.5.0

Here is my full nagios.cfg file:

accept_passive_host_checks=1
accept_passive_service_checks=1
additional_freshness_latency=15
admin_email=nagios@localhost
admin_pager=pagenagios@localhost
auto_reschedule_checks=0
auto_rescheduling_interval=30
auto_rescheduling_window=180
bare_update_check=0
cached_host_check_horizon=15
cached_service_check_horizon=15
cfg_file=/usr/local/nagios/etc/objects/commands.cfg
cfg_file=/usr/local/nagios/etc/objects/contacts.cfg
cfg_file=/usr/local/nagios/etc/objects/hosts.cfg
cfg_file=/usr/local/nagios/etc/objects/services.cfg
cfg_file=/usr/local/nagios/etc/objects/templates.cfg
cfg_file=/usr/local/nagios/etc/objects/timeperiods.cfg
check_external_commands=1
check_for_orphaned_hosts=1
check_for_orphaned_services=1
check_for_updates=1
check_host_freshness=1
check_result_path=/usr/local/nagios/var/spool/checkresults
check_result_reaper_frequency=10
check_service_freshness=1
child_processes_fork_twice=0
command_check_interval=-1
command_file=/usr/local/nagios/var/rw/nagios.cmd
daemon_dumps_core=0
date_format=us
debug_file=/usr/local/nagios/var/nagios.debug
debug_level=0
debug_verbosity=1
enable_embedded_perl=0
enable_environment_macros=0
enable_event_handlers=1
enable_flap_detection=1
enable_notifications=1
enable_predictive_host_dependency_checks=1
enable_predictive_service_dependency_checks=1
event_broker_options=-1
event_handler_timeout=300
execute_host_checks=1
execute_service_checks=1
external_command_buffer_slots=4096
high_host_flap_threshold=20.0
high_service_flap_threshold=20.0
host_check_timeout=30
host_freshness_check_interval=60
host_inter_check_delay_method=s
host_perfdata_command=nagios-process-host-perfdata
host_perfdata_file_mode=a
host_perfdata_file_processing_command=nagios-process-host-perfdata-file
host_perfdata_file_processing_interval=86400
illegal_macro_output_chars=`~$&|'"<>
illegal_object_name_chars=`~!$%^&*|'"<>?,()=
interval_length=60
lock_file=/usr/local/nagios/var/nagios.lock
log_archive_path=/usr/local/nagios/var/archives
log_event_handlers=1
log_external_commands=1
log_file=/usr/local/nagios/var/nagios.log
log_host_retries=0
log_initial_states=0
log_notifications=1
log_passive_checks=1
log_rotation_method=d
log_service_retries=0
low_host_flap_threshold=5.0
low_service_flap_threshold=5.0
max_check_result_file_age=3600
max_check_result_reaper_time=30
max_concurrent_checks=0
max_debug_file_size=1000000
max_host_check_spread=30
max_service_check_spread=30
nagios_group=nagios
nagios_user=nagios
notification_timeout=30
object_cache_file=/usr/local/nagios/var/objects.cache
obsess_over_hosts=0
obsess_over_services=0
ocsp_timeout=5
p1_file=/usr/local/nagios/bin/p1.pl
passive_host_checks_are_soft=0
perfdata_timeout=5
perfdata_timeout=5
precached_object_file=/usr/local/nagios/var/objects.precache
process_performance_data=0
process_performance_data=1
resource_file=/usr/local/nagios/etc/resource.cfg
retain_state_information=1
retained_contact_host_attribute_mask=0
retained_contact_service_attribute_mask=0
retained_host_attribute_mask=0
retained_process_host_attribute_mask=0
retained_process_service_attribute_mask=0
retained_service_attribute_mask=0
retention_update_interval=60
service_check_timeout=300
service_freshness_check_interval=60
service_inter_check_delay_method=s
service_interleave_factor=s
service_perfdata_command=nagios-process-service-perfdata
service_perfdata_file_mode=a
service_perfdata_file_processing_command=nagios-process-service-perfdata-file
service_perfdata_file_processing_interval=86400
sleep_time=0.25
soft_state_dependencies=0
state_retention_file=/usr/local/nagios/var/retention.dat
status_file=/usr/local/nagios/var/status.dat
status_update_interval=10
temp_file=/usr/local/nagios/var/nagios.tmp
temp_path=/tmp
translate_passive_host_checks=0
use_aggressive_host_checking=0
use_embedded_perl_implicitly=0
use_large_installation_tweaks=1
use_regexp_matching=0
use_retained_program_state=1
use_retained_scheduling_info=1
use_syslog=0
use_true_regexp_matching=0

[jdalrymple]

I loaded your config into a test system (albeit a 4.X system) and it worked perfectly.

I suggest at this point you turn on debugging and see if you can spot anything in the debug.log:

Code: Select all

debug_file=/usr/local/nagios/var/nagios.debug
debug_level=-1
debug_verbosity=2
max_debug_file_size=10000000

Just as a point of clarification - do you get ANYTHING in the nagios.log when you submit your command?