Changed host IP now get SSL handshake failed from NRPE

[ScottG]

Hi, I migrated a host from one network to another and subsequently had to change the IP address. Now, my check_nrpe commands return SSL error on the Nagios XI server:

Code: Select all

CHECK_NRPE: Error - Could not complete SSL handshake.

And on the client syslog:

Code: Select all

Nov 13 15:50:54 redrad01 nrpe[28397]: Error: Network server getpeername() failure (107: Transport endpoint is not connected)
Nov 13 15:50:54 redrad01 nrpe[28397]: Error: Could not complete SSL handshake with : 5

Ping checks work fine, and I can confirm with tcpdump that the packets are getting to and from source and destination. The packet capture reveals that the 3-way handshake is happening, and then the Nagios XI server is sending a RST packet. I have tried deleting the old config and re-adding it back, but I still get the error.

[dwasswa]

Hi @ScottG,

The following kb article provides you with instructions to resolve the CHECK_NRPE: Error - Could not complete SSL handshake error.

nrpe-check_nrpe-error-could-not-complete-ssl-handshake

Please follow the instructions in the kb article and let me know if that solves your issue.

[kyang]

Hey ScottG, just checking in to see if your issue is resolved?

Did you have any more questions? Or did you figure this out?

[ScottG]

Hey ScottG, just checking in to see if your issue is resolved?

Did you have any more questions? Or did you figure this out?

I apologize. I have not circled back around to this yet. I can say that the troubleshooting tips in that link did not solve my issue. I need to do more looking at it though. I appreciate you checking in.

[kyang]

No problem,

Could you post your nagios.cfg for us also?

What version of NRPE do you have?

Code: Select all

./check_nrpe -V

With that, could you send us a profile?

On the XI Home Page click "Admin" > "System Profile" --> "Download Profile" button
Save the profile.zip file and upload it here or PM me. Respond back here so I know you sent it.


Profile Received! Share with the Support Team.

[ScottG]

No problem, Could you post your nagios.cfg for us also?

Code: Select all

# MODIFIED
admin_email=root@localhost
admin_pager=root@localhost
translate_passive_host_checks=1
log_event_handlers=0
use_large_installation_tweaks=1
enable_environment_macros=0


# NDOUtils module
broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg


# PNP settings - bulk mode with NCPD
process_performance_data=1
# service performance data
service_perfdata_file=/usr/local/nagios/var/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$\tLONGSERVICEOUTPUT::$LONGSERVICEOUTPUT$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
# host performance data
host_perfdata_file=/usr/local/nagios/var/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tHOSTOUTPUT::$HOSTOUTPUT$\tLONGHOSTOUTPUT::$LONGHOSTOUTPUT$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file-bulk


# OBJECTS - UNMODIFIED
#cfg_file=/usr/local/nagios/etc/objects/commands.cfg
#cfg_file=/usr/local/nagios/etc/objects/contacts.cfg
#cfg_file=/usr/local/nagios/etc/objects/localhost.cfg
#cfg_file=/usr/local/nagios/etc/objects/templates.cfg
#cfg_file=/usr/local/nagios/etc/objects/timeperiods.cfg


# STATIC OBJECT DEFINITIONS (THESE DON'T GET EXPORTED/IMPORTED BY NAGIOSQL)
cfg_dir=/usr/local/nagios/etc/static

# OBJECTS EXPORTED FROM NAGIOSQL
cfg_file=/usr/local/nagios/etc/contacttemplates.cfg
cfg_file=/usr/local/nagios/etc/contactgroups.cfg
cfg_file=/usr/local/nagios/etc/contacts.cfg
cfg_file=/usr/local/nagios/etc/timeperiods.cfg
cfg_file=/usr/local/nagios/etc/commands.cfg
cfg_file=/usr/local/nagios/etc/hostgroups.cfg
cfg_file=/usr/local/nagios/etc/servicegroups.cfg
cfg_file=/usr/local/nagios/etc/hosttemplates.cfg
cfg_file=/usr/local/nagios/etc/servicetemplates.cfg
cfg_file=/usr/local/nagios/etc/servicedependencies.cfg
cfg_file=/usr/local/nagios/etc/serviceescalations.cfg
cfg_file=/usr/local/nagios/etc/hostdependencies.cfg
cfg_file=/usr/local/nagios/etc/hostescalations.cfg
cfg_file=/usr/local/nagios/etc/hostextinfo.cfg
cfg_file=/usr/local/nagios/etc/serviceextinfo.cfg
cfg_dir=/usr/local/nagios/etc/hosts
cfg_dir=/usr/local/nagios/etc/services

# GLOBAL EVENT HANDLERS
global_host_event_handler=xi_host_event_handler
global_service_event_handler=xi_service_event_handler



# UNMODIFIED
accept_passive_host_checks=1
accept_passive_service_checks=1
additional_freshness_latency=15
auto_reschedule_checks=1
auto_rescheduling_interval=30
auto_rescheduling_window=45
bare_update_check=0
cached_host_check_horizon=15
cached_service_check_horizon=15
check_external_commands=1
check_for_orphaned_hosts=1
check_for_orphaned_services=1
check_for_updates=1
check_host_freshness=0
check_result_path=/usr/local/nagios/var/spool/checkresults
check_result_reaper_frequency=10
check_service_freshness=1
command_file=/usr/local/nagios/var/rw/nagios.cmd
daemon_dumps_core=0
date_format=us
debug_file=/usr/local/nagios/var/nagios.debug
debug_level=0
debug_verbosity=1
enable_event_handlers=1
enable_flap_detection=1
enable_notifications=1
enable_predictive_host_dependency_checks=1
enable_predictive_service_dependency_checks=1
event_broker_options=-1
event_handler_timeout=30
execute_host_checks=1
execute_service_checks=1
high_host_flap_threshold=20.0
high_service_flap_threshold=20.0
host_check_timeout=30
host_freshness_check_interval=60
host_inter_check_delay_method=s
illegal_macro_output_chars=`~$&|'"<>
illegal_object_name_chars=`~!$%^&*|'"<>?,()=
interval_length=60
lock_file=/usr/local/nagios/var/nagios.lock
log_archive_path=/usr/local/nagios/var/archives
log_external_commands=0
log_file=/usr/local/nagios/var/nagios.log
log_host_retries=1
log_initial_states=0
log_notifications=1
log_passive_checks=0
log_rotation_method=d
log_service_retries=1
low_host_flap_threshold=5.0
low_service_flap_threshold=5.0
max_check_result_file_age=3600
max_check_result_reaper_time=30
max_concurrent_checks=0
max_debug_file_size=1000000
max_host_check_spread=30
max_service_check_spread=30
nagios_group=nagios
nagios_user=nagios
notification_timeout=30
object_cache_file=/usr/local/nagios/var/objects.cache
obsess_over_hosts=0
obsess_over_services=0
ocsp_timeout=5
passive_host_checks_are_soft=0
perfdata_timeout=5
precached_object_file=/usr/local/nagios/var/objects.precache
resource_file=/usr/local/nagios/etc/resource.cfg
retained_contact_host_attribute_mask=0
retained_contact_service_attribute_mask=0
retained_host_attribute_mask=0
retained_process_host_attribute_mask=0
retained_process_service_attribute_mask=0
retained_service_attribute_mask=0
retain_state_information=1
retention_update_interval=60
service_check_timeout=60
service_freshness_check_interval=60
service_inter_check_delay_method=s
service_interleave_factor=s
soft_state_dependencies=0
state_retention_file=/usr/local/nagios/var/retention.dat
status_file=/usr/local/nagios/var/status.dat
status_update_interval=10
temp_file=/usr/local/nagios/var/nagios.tmp
temp_path=/tmp
use_aggressive_host_checking=0
use_regexp_matching=0
use_retained_program_state=1
use_retained_scheduling_info=1
use_syslog=1
use_true_regexp_matching=0

What version of NRPE do you have?

Code: Select all

./check_nrpe -V

2.15

With that, could you send us a profile?

On the XI Home Page click "Admin" > "System Profile" --> "Download Profile" button
Save the profile.zip file and upload it here or PM me. Respond back here so I know you sent it.

[/quote]

I sent the profile.zip in a PM. Thanks again.

[npolovenko]

@ScottG, Can you verify that the machine you're trying to monitor has the new Nagios IP address in the allowed settings?

On the remote server that you're trying to minitor with NRPE, you'd need to either go to:

Code: Select all

/etc/xinetd.d/
only_from = 127.0.0.1 Nagios_Server_IP

or in

Code: Select all

/usr/local/nagios/etc/nrpe.cfg
allowed_hosts=127.0.0.1,Nagios_Server_IP

(Depending on your configuration).
*Notice how in the first case the IP address are separated by space but in the second case, they're separated by a comma.

After that you need to restart nrpe with either service nrpe restart or service xinetd restart.

[ScottG]

Code: Select all

/usr/local/nagios/etc/nrpe.cfg
allowed_hosts=127.0.0.1,Nagios_Server_IP

Mine is actually in a different directory. This is on RHEL7 and installed from repo.

Code: Select all

[root@redrad01 sgardne]# cat /etc/nagios/nrpe.cfg | grep allowed_hosts=
allowed_hosts=127.0.0.1,10.7.2.37,130.184.253.9

The Nagios server is the 10.7.2.37 entry.

[npolovenko]

@ ScottG, Are you able to nmap the nrpe client server from the Nagios server?

Code: Select all

nmap <target IP> -p 5666

What happens when you run service nrpe status on the remote server? Do you get any errors? I've seen some old bug where nrpe.pid file would not get properly created/replaced. So if you're able to find nrpe.pid presumably located in /var/run/nagios/nrpe.pid, please delete it and restart NRPE. That should force nrpe to recreate this file.

[ScottG]

@ ScottG, Are you able to nmap the nrpe client server from the Nagios server?

Code: Select all

nmap <target IP> -p 5666

What happens when you run service nrpe status on the remote server? Do you get any errors? I've seen some old bug where nrpe.pid file would not get properly created/replaced. So if you're able to find nrpe.pid presumably located in /var/run/nagios/nrpe.pid, please delete it and restart NRPE. That should force nrpe to recreate this file.

systemd says it's running and shows the log errors I mentioned in the OP.

Code: Select all

[sgardne@redrad01 ~]$ sudo systemctl status nrpe
● nrpe.service - Nagios Remote Program Executor
   Loaded: loaded (/usr/lib/systemd/system/nrpe.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2017-11-29 09:49:02 CST; 1h 19min ago
     Docs: http://www.nagios.org/documentation
  Process: 11883 ExecStart=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d $NRPE_SSL_OPT (code=exited, status=0/SUCCESS)
 Main PID: 11884 (nrpe)
   CGroup: /system.slice/nrpe.service
           ├─11884 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
           ├─12919 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
           └─35127 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

Nov 29 09:49:02 redrad01.uark.edu systemd[1]: Starting Nagios Remote Program Executor...
Nov 29 09:49:02 redrad01.uark.edu nrpe[11884]: Starting up daemon
Nov 29 09:49:02 redrad01.uark.edu systemd[1]: Started Nagios Remote Program Executor.
Nov 29 09:49:02 redrad01.uark.edu nrpe[11884]: Server listening on 0.0.0.0 port 5666.
Nov 29 09:49:02 redrad01.uark.edu nrpe[11884]: Server listening on :: port 5666.
Nov 29 09:49:02 redrad01.uark.edu nrpe[11884]: Warning: Daemon is configured to accept command arguments from clients!
Nov 29 09:49:02 redrad01.uark.edu nrpe[11884]: Listening for connections on port 5666
Nov 29 09:49:02 redrad01.uark.edu nrpe[11884]: Allowing connections from: 127.0.0.1,10.7.2.37,130.184.253.9
Nov 29 09:50:10 redrad01.uark.edu nrpe[12117]: Error: Network server getpeername() failure (107: Transport endpoint is not connected)
Nov 29 09:50:43 redrad01.uark.edu nrpe[12219]: Error: Network server getpeername() failure (107: Transport endpoint is not connected)