Monitoring nagios with another nagios.

[Box293]

So the server is listening, it's most likely that the port need to be opened on the Core server firewall.

This command executed from the other server will confirm it:

Code: Select all

nmap 10.0.60.10 -p 5666

If the port is opened, the result will be like:

Code: Select all

Starting Nmap 6.47 ( http://nmap.org ) at 2016-05-18 10:55 AEST
Nmap scan report for centos01 (10.25.13.10)
Host is up (0.00031s latency).
rDNS record for 10.25.13.10: centos01.box293.local
PORT     STATE SERVICE
5666/tcp open  nrpe
MAC Address: 00:50:56:AB:89:1C (VMware)

Nmap done: 1 IP address (1 host up) scanned in 0.11 seconds

This being open is what you need to see:
5666/tcp open nrpe

[SaltyBear]

So the server is listening, it's most likely that the port need to be opened on the Core server firewall.

This command executed from the other server will confirm it:

Code: Select all

nmap 10.0.60.10 -p 5666

If the port is opened, the result will be like:

Code: Select all

Starting Nmap 6.47 ( http://nmap.org ) at 2016-05-18 10:55 AEST
Nmap scan report for centos01 (10.25.13.10)
Host is up (0.00031s latency).
rDNS record for 10.25.13.10: centos01.box293.local
PORT     STATE SERVICE
5666/tcp open  nrpe
MAC Address: 00:50:56:AB:89:1C (VMware)

Nmap done: 1 IP address (1 host up) scanned in 0.11 seconds

This being open is what you need to see:
5666/tcp open nrpe

It does indeed say open, I ran this command from the actual 10.0.60.10 server, if I run it from the other nagios server which is monitoring that one I get a filtered output.

[Box293]

if I run it from the other nagios server which is monitoring that one I get a filtered output.

Then there is a firewall causing the issue.

On the server being monitored (10.0.60.10 I think) what is the output of:

Code: Select all

sudo iptables --list

If the result is the same as this:

Code: Select all

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Then it's not the firewall on the ubuntu box. There is something between the two Nagios servers blocking port 5666. Make sure you check the rules in both directions.

[SaltyBear]

This is what I get, so then the firewall is still blocking port 5666 even though we opened it?

Code: Select all

root@Nightswatch:/home/snow# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

[tmcdonald]

More likely something in-between the two devices as @Box293 mentioned. Do you have any sort of FW on the switch/router between them? Anything they proxy through, or an IDS/IPS of any kind?

[SaltyBear]

More likely something in-between the two devices as @Box293 mentioned. Do you have any sort of FW on the switch/router between them? Anything they proxy through, or an IDS/IPS of any kind?

There are 2 LAN's
They are virtualized on a blade provided for us, each LAN has a firewall and lan B also has a DMZ.
Between these there is also a VPN connection, port 5666 is open on both firewalls since monitoring other windows/ubuntu servers/hosts works fine.

[rkennedy]

Can you attempt to run a telnet and see what happens? Please post the result for us to see. As we've all mentioned, I don't believe this is a network issue.

Are the devices that are working on the same subnet as the machine you're unable to monitor?

[nozlaf]

You may need to open the ports on the Nagios server as well:

Code: Select all

sudo iptables -I INPUT -p tcp --destination-port 5666 -j ACCEPT

I think you need to run this as well:

Code: Select all

sudo apt-get install -y iptables-persistent

This command should confirm if your Ubuntu server is listening on port 5666:

Code: Select all

sudo lsof -i :5666

What output does it produce?

I didn't run the first 2 command

you should have run the first two commands they would fix the issue you have

[Box293]

More likely something in-between the two devices as @Box293 mentioned. Do you have any sort of FW on the switch/router between them? Anything they proxy through, or an IDS/IPS of any kind?

There are 2 LAN's
They are virtualized on a blade provided for us, each LAN has a firewall and lan B also has a DMZ.
Between these there is also a VPN connection, port 5666 is open on both firewalls since monitoring other windows/ubuntu servers/hosts works fine.

It might be helpful to draw a diagram that explains this topology, explaining what ports have been opened in what direction.

[nozlaf]

we opened the ports on the firewall, not sure what you mean pfsense/OBSD?

I believe the issue here is that @saltybear thinks everyone is talking about his pfsense firewall and not the iptables firewall on the ubuntu nagios box

and @saltybear never opened the firewall on the ubuntu box

I didn't run the first 2 commands

so run

Code: Select all

sudo iptables -I INPUT -p tcp --destination-port 5666 -j ACCEPT

and you will find that it magically works

also somewhere he says thatg he can connect to 5666 from the affected server but not elsewhere, this is also consistent with not having allowed iptables firewall rule

an he has confirmed that other hosts on that side of the network can be monitored via port 5666