Change in Nagios Server IP, 5666 connection refused.

[jim]

Dear All,

As I type its already late but I think it might be my fault.

My Nagios Server is CentOS 6.8 and was working well before.

I think I did mess big time as I now when I go to run check_nrpe on nagios server itself it say connection refused.

Its 3am and my brain is not working well, may be tomorrow morning it might be better. But I am in no man's land. I had one nagios monitoring two severs including nagios server itself.

Here is the output from my nagios server

Code: Select all

[root@puppet libexec]# cat /etc/services |grep nrpe
[root@puppet libexec]# service nrpe restart
nrpe: unrecognized service
[root@puppet libexec]# service xinetd restart
xinetd: unrecognized service
[root@puppet libexec]# netstat -nap|grep nrpe
[root@puppet libexec]# ./check_nrpe -H localhost
connect to address ::1 port 5666: Connection refused
connect to address 127.0.0.1 port 5666: Connection refused
connect to host localhost port 5666: Connection refused[root@puppet libexec]# 

Please help as I don't think this is a good picture...

Any help would be very much appreciated

I can login to nagios with http://192.168.1.2/nagios previous ip was 192.168.1.7

Thanks a million Guys

[jim]

Hi Nagios Lovers,

check_nrpe -H <Nagios serverIP> was certainly working before and now it is not working now.

Code: Select all

./check_nrpe -H localhost
connect to address ::1 port 5666: Connection refused
connect to address 127.0.0.1 port 5666: Connection refused
connect to host localhost port 5666: Connection refused[root@puppet libexec]# 

Code: Select all

[root@puppet libexec]# ./check_nrpe -H 192.168.1.2
connect to address 192.168.1.2 port 5666: Connection refused
connect to host 192.168.1.2 port 5666: Connection refused[root@puppet libexec]# 
[root@puppet libexec]# 

I cannot see entry for NRPE port in /etc/services.((Please refer my previous post)

Also

Code: Select all

[root@puppet libexec]# nmap 192.168.1.2

Starting Nmap 5.51 ( http://nmap.org ) at 2016-06-19 23:24 AEST
Nmap scan report for puppet (192.168.1.2)
Host is up (0.0000020s latency).
Not shown: 997 closed ports
PORT    STATE SERVICE
22/tcp  open  ssh
80/tcp  open  http
111/tcp open  rpcbind

Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds
[root@puppet libexec]# getenforce
Enforcing
[root@puppet libexec]# 

My best guess is nrpe on nagios server is not running, but I am not sure what to troubleshoot that.

What does surprise me is nrpe file is missing under /etc/xinetd.d/nrpe !

[Box293]

What does surprise me is nrpe file is missing under /etc/xinetd.d/nrpe !

This guide shows you how it runs without xinetd:

https://support.nagios.com/kb/article.php?id=92


It sounds like NRPE is just not running:

Can you please run this command and show us the output:

Code: Select all

ps -C nrpe

Also this guide may help with your issue:

https://support.nagios.com/kb/article.php?id=140

If this doesn't fix your problem, please let us know what you tried (including commands typed) and what the output/result is.

[jim]

Thank you very much. I just didn't knew how to start nrpe :( I am writing the command to start nrpe in bold in my notes.

Code: Select all

[root@puppet ~]# ps -C nrpe
  PID TTY          TIME CMD
[root@puppet ~]# /usr/local/nagios/bin/nrpe -c /usr/local/nagios/etc/nrpe.cfg -d
[root@puppet ~]# ps -C nrpe
  PID TTY          TIME CMD
 2448 ?        00:00:00 nrpe

However, I do get error now like below:

Code: Select all

[root@puppet libexec]# ./check_nrpe -H localhost
[b]CHECK_NRPE: Error - Could not complete SSL handshake.[/b]
[root@puppet libexec]# ./check_nrpe -H 192.168.1.2
[b]CHECK_NRPE: Error - Could not complete SSL handshake.[/b]
[root@puppet libexec]# cat /usr/local/nagios/etc/nrpe.cfg |grep allowed
# that are allowed to talk to the NRPE daemon. Network addresses with a bit mask
[b]allowed_hosts=127.0.0.1 192.168.1.2[/b]
[root@puppet libexec]# 

Note that IP of Nagios server was not present in nagios server nrpe.cfg file but I added now and restarted the nagios daemon. I have no idea why its went missing....

More than likely it seems I may need to need to recompile and install nrpe but why I have when it was working in first place. What cause it to break?

[Box293]

Code: Select all

[root@puppet libexec]# cat /usr/local/nagios/etc/nrpe.cfg |grep allowed
# that are allowed to talk to the NRPE daemon. Network addresses with a bit mask
allowed_hosts=127.0.0.1 192.168.1.2

# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames

It should be:

Code: Select all

allowed_hosts=127.0.0.1,192.168.1.2

[jim]

Thanks mate. Got it working but still SSL handshake error is keeping me on my toes. Like your site and documentation. What might cause SSL handshake errors? It was working before for sure.

[rkennedy]

When you run the check_nrpe command from the Nagios machine, take a look at /var/log/messages on the client machine, do you see any output there regarding NRPE? This should help us find a solution.

[jim]

Thank you very much. I shall look at it tonight.
However I want to understand what happens when a user inputs the following. I want to understand which logs I should look at in each instance.


On remote Host
./check_nrpe.sh -H localhost
./check_nrpe.sh -H <Remote Server Hostname>
./check_nrpe.sh -H <Remote Server IP>
./check_nrpe.sh -H <Nagios Server IP>
./check_nrpe.sh -H <Nagio Server Hostname>


On nagio server
./check_nrpe.sh -H localhost
./check_nrpe.sh -H <Nagios Server Hostname>
./check_nrpe.sh -H <Nagios Server IP>
./check_nrpe.sh -H <Remote Server Host IP>
./check_nrpe.sh -H <Remote Server Hostname>

Also in each instance where I should look(log file)

This would greatly help me troubleshoot issue.

My main interest is what causes for SSL Handshake to break.

Many Thanks in advance !!

Jim

[jim]

Thanks you to Forum Moderators for tolerating me. While my post may ask some basic and silly questions it gives good in depth knowledge of nagios.

I did this:

Code: Select all

[root@puppet libexec]# 
[root@puppet libexec]# ./check_nrpe -H 192.168.1.4
connect to address 192.168.1.4 port 5666: No route to host
connect to host 192.168.1.4 port 5666: No route to host[root@puppet libexec]# 

here 192.168.1.4 is the IP of remote linux host. No information seen in /var/log/messages of nagios server as well as Remote Linux server.

Then I did the following on 192.168.1.4(agent2) and guess what !

Code: Select all

[root@agent2 ~]# cd /usr/local/nagios/libexec/
[root@agent2 libexec]# ./check_nrpe -H agent2
CHECK_NRPE: Error - Could not complete SSL handshake.
[root@agent2 libexec]# ./check_nrpe -H 192.168.1.4
CHECK_NRPE: Error - Could not complete SSL handshake.
[root@agent2 libexec]# ps 5938
  PID TTY      STAT   TIME COMMAND
[root@agent2 libexec]# ps -C nrpe
  PID TTY          TIME CMD
[root@agent2 libexec]# ps -aef|grep nrpe|grep -v grep
[root@agent2 libexec]# 

here is the vital information from that server(agent2) /var/log/messages

Code: Select all

Jun 21 23:47:14 agent2 xinetd[1419]: unexpected signal: 18 (Continued) in signal pipe
Jun 21 23:47:14 agent2 xinetd[1419]: Exiting...
Jun 21 23:47:14 agent2 systemd: Stopping Xinetd A Powerful Replacement For Inetd...
Jun 21 23:47:14 agent2 systemd: Starting Xinetd A Powerful Replacement For Inetd...
Jun 21 23:47:14 agent2 systemd: PID file /var/run/xinetd.pid not readable (yet?) after start.
Jun 21 23:47:14 agent2 systemd: Started Xinetd A Powerful Replacement For Inetd.
Jun 21 23:47:14 agent2 xinetd[4997]: xinetd Version 2.3.15 started with libwrap loadavg labeled-networking options compiled in.
Jun 21 23:47:14 agent2 xinetd[4997]: Started working: 1 available service






Jun 21 23:48:26 agent2 systemd: Started Session 7 of user jim.
Jun 21 23:48:26 agent2 systemd-logind: New session 7 of user jim.
Jun 21 23:48:26 agent2 systemd: Starting Session 7 of user jim.
Jun 21 23:48:28 agent2 systemd-logind: Removed session 7.
Jun 21 23:49:10 agent2 xinetd[4997]: START: nrpe pid=5342 from=::ffff:192.168.1.4
Jun 21 23:49:10 agent2 xinetd[5342]: FAIL: nrpe address from=::ffff:192.168.1.4
Jun 21 23:49:10 agent2 xinetd[4997]: EXIT: nrpe status=0 pid=5342 duration=0(sec)

Jun 22 00:02:36 agent2 xinetd[4997]: START: nrpe pid=5938 from=::ffff:192.168.1.4
Jun 22 00:02:36 agent2 xinetd[5938]: FAIL: nrpe address from=::ffff:192.168.1.4
Jun 22 00:02:36 agent2 xinetd[4997]: EXIT: nrpe status=0 pid=5938 duration=0(sec)

Now, I believe nrpe is not running on remote server but xinetd is running and nrpe is compile to run under xinetd so it should work ??

Code: Select all

[root@agent2 libexec]# ps -aef|grep xinetd|grep -v grep
root      6392     1  0 00:12 ?        00:00:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid
[root@agent2 libexec]# 

Would greatly appreciate any help. I think we are close.

[rkennedy]

On remote Host
./check_nrpe.sh -H localhost
./check_nrpe.sh -H <Remote Server Hostname>
./check_nrpe.sh -H <Remote Server IP>
./check_nrpe.sh -H <Nagios Server IP>
./check_nrpe.sh -H <Nagio Server Hostname>


On nagio server
./check_nrpe.sh -H localhost
./check_nrpe.sh -H <Nagios Server Hostname>
./check_nrpe.sh -H <Nagios Server IP>
./check_nrpe.sh -H <Remote Server Host IP>
./check_nrpe.sh -H <Remote Server Hostname>

On both server's it's going to do the same thing.
1. Checks itself
2. Checks the Nagios Server (could run into issues if you try itself, from the Nagios server)
3. ^
4. Checks the remote server
5. ^

Code: Select all

[root@puppet libexec]#
[root@puppet libexec]# ./check_nrpe -H 192.168.1.4
connect to address 192.168.1.4 port 5666: No route to host
connect to host 192.168.1.4 port 5666: No route to host[root@puppet libexec]# 

From this machine, can you ping 192.168.1.4? What is the output of nmap 192.168.1.4? Is this machine your Nagios server? I suspect this could be a firewall / routing issue.

Then I did the following on 192.168.1.4(agent2) and guess what !

This is failing most likely because of the allowed_from line in your nrpe file. Take a look at /etc/xinetd.d/nrpe, and make sure 127.0.0.1 is in there, and also 192.168.1.4. This will allow you to run check_nrpe against yourself. After making any changes in that file, make sure to do service restart xinetd