Unusual traffic from Nagios XI Server

[sagesoft]

Hi;

I would like to ask your expertise on why is our Nagios XI(172.16.66.22) server is sending an unusual traffic to one of the server(172.16.66.46 in port 22) as per checking in the firewall logs. Is this a usual behavior of Nagios XI? Refer attached firewall logs.

Thank you in advance.

[sagesoft]

Just additional input -- Can I use the Nagios Event log report to prove that this is not Nagios XI application doing? or there are other report I can use?

[mbellerue]

That could be related to port reuse. Has this been going on for a while, or did it just start up recently?

Could you send in your system profile. Just go to Admin -> System Profile -> Download Profile.

What is the 172.16.66.46 address, and can you ping it from the Nagios server?

And finally, if you could run a tcpdump near the next 15 minute interval so that we can see the traffic as the 15 minute interval rolls over.

Code: Select all

tcpdump host 172.16.66.46 > /tmp/tcpdump172.16.66.46.pcap

[sagesoft]

@mbellerue ,


Refer attached.

To provide more context to this issue—the one that Nagios keeps connecting to every millisecond was decommissioned/ turned off before the issue occurred. Also, what we see in the logs of Nagios is not consistent with the logs from our firewall—between 2:00 AM to 2:05 AM of September 13, Nagios only logged around 50 connections to the server that was decommissioned. During the same duration, our firewall logged more than 300 connection attempts to the same server. To validate these data, we extracted traffic logs from our traffic monitoring tool and the results from this tool was consistent with what was logged by the firewall.

It was then observed that Nagios was indeed connecting continuously and consistently to this server (every millisecond).

Our question is, is this a normal behavior of Nagios ? And is Nagios’ traffic (every millisecond) enough to flood a network and cause a slow down?

Support edit: profile (4).zip downloaded and shared with team.

[mbellerue]

I'm going to take a look at the profile, but I want to just say that no, this is not normal behavior for Nagios. It's possible that some kind of configuration bug happened, and this was the result. I will take a look at the profile to see if I can find any evidence to that effect.

[sagesoft]

Did you find anything?

[mbellerue]

I'm not finding anything that would explain why Nagios would be sending so many packets to that server. The configuration files show that you don't have that many checks going to the server. There are not indications that I can find that the Nagios application was responsible for the packet storm.

If this is something you can reproduce, we might be able to look at it a little further.

[sagesoft]

@mbellerue;

Thank you for your reply. Reproducing might be an option, since they already decommissioned/turned off the XI server. And the most unusual thing that happened is that even if the server is already turned off they can still see in the firewall logs that the server still sending packets. Is there a cache file that the Nagios Server keeps? if there is what is the directory?

[mbellerue]

Just to clarify, you are saying if the Nagios server is powered off, it still sends packets? That is definitely not normal behavior. I don't know how we would go about doing something like that. Basically if the Nagios daemon is stopped, there should be no traffic from Nagios itself.