Nagios XI https Deny TCP every now and then

[michal.nastaly]

The logical view of this setup is as follows:

Nagios Fusion -------------------IN-|Firewall1|-OUT-------------------(Private Cloud)-------------------OUT-|Firewall2|-IN-------------------Nagios XI

sorry for the primitive diagram, but that's easiest to represent the logical setup.

When i do real time monitoring on Firewall2 i can see that something is closing the conection on the INSIDE interface and then traying to send more packet on that connection. We don't see this issue when we use http, it only happens when https is enabled.

Nagios XI and Fusion are on different LANs

[tmcdonald]

When this happens, do you notice anything on the error_log with the XI machine?

Can you please let us know about the error logs? They will likely be located in /var/log/httpd/ssl_error_log if this only happens with SSL.

[michal.nastaly]

The log file in /var/log/httpd/ssl_error_log is blank.

and the error_log has only got the following error occurring on regular basis:

[Wed Mar 16 10:41:14 2016] [error] [client 10.27.40.4] PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/xicore/status-object-detail.inc.php on line 2474, referer: https://<NagiosXiAddress>/nagiosxi/includes/components/xicore/status.php?show=services

[hsmith]

Can we see the output of this command on both servers?

iptables -L -n

[michal.nastaly]

Nagios Fusion:

Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTAB LISHED
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2 2
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:8 0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:4 43
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-ho st-prohibited

Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-ho st-prohibited

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Nagios XI:

Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:5667
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

[hsmith]

Can you attempt to disable the firewall on these machines and see if the issue persists?

[michal.nastaly]

Unfortunately we cannot disable the firewall on these servers for security reasons. Any other ideas?

[tmcdonald]

Everything I am seeing seems to point to the firewalls:

https://supportforums.cisco.com/discuss ... on-syn-ack
https://www.reddit.com/r/networking/com ... t_sees_no/
http://whatwasthatcommand.blogspot.com/ ... ction.html

You'll need to talk to your network/security team to figure out why this is happening, as this clearly looks like a firewall issue.

[michal.nastaly]

But surely if that was asymmetric routing issue then http traffic would have the same problem as https.

[tmcdonald]

Potentially, yes, but SSL traffic requires more handshaking and overhead in general that might be tripping up your firewalls.

From a more broad perspective, is this impacting the Fusion or XI servers at all? I learned a long time ago that if you pursue every error in your logs you will eventually fix your environment so well that it no longer works :) Short of having a running traffic capture and analyzing the packets to and from your systems, I don't know what else could be done to narrow this down, and that's a little overkill for what we can do without paid support.