NagiosXI Network Maxed

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
abishop
Posts: 25
Joined: Fri Dec 13, 2019 9:59 am

NagiosXI Network Maxed

Post by abishop »

When NagiosXI trial is running our network maxes out, and slows everything down. It is contacting 180.188.20.33:80. Any idea how to stop this besides blocking it; we've tried and the local network is still flooded. I assume it's communicating to this IP since it's a trial.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: NagiosXI Network Maxed

Post by scottwilkerson »

This is IP is not us at all, it is an IP out of China

Is this Nagios XI server public?

did you install custom plugins?
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: NagiosXI Network Maxed

Post by ssax »

We don't call out to china for anything, was there anything done to this system as far as installing custom software/etc?

Any custom repositories using a mirrorlist?

Code: Select all

yum repolist
What does this show?

Code: Select all

netstat -tuapn | grep 180.188.20.33
If this is a VM, power it off and take a VM snapshot.

The first step would be to power it off, the second step is to boot with no network connected and investigate what is running on the system (as root) and scan it with antivirus/antirootkit/antimalware, change passwords, etc:

Code: Select all

ps aux
Then do something like this to capture a pcap file to investigate what it is trying to send:

Code: Select all

https://www.wireshark.org/docs/wsug_html_chunked/AppToolstcpdump.html
abishop
Posts: 25
Joined: Fri Dec 13, 2019 9:59 am

Re: NagiosXI Network Maxed

Post by abishop »

scottwilkerson wrote:This is IP is not us at all, it is an IP out of China

Is this Nagios XI server public?
It is default Hyper-V vm from the website. Nothing extra configured, so I do not believe it is pointed to the public. It has an internal IP.
did you install custom plugins?
Other than setting up 83 hosts for ping services nothing additional. This is a Hyper-V VM downloaded form Nagios website
ssax wrote:We don't call out to china for anything, was there anything done to this system as far as installing custom software/etc?

Any custom repositories using a mirrorlist?
Not that I set up

Code: Select all

yum repolist
[root@coanagiosxi ~]# yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.grid.uchicago.edu
* epel: mirror.uic.edu
* extras: mirror.team-cymru.com
* updates: mirror.steadfastnet.com
repo id repo name status
!base/7/x86_64 CentOS-7 - Base 10,097
!epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 13,491
!extras/7/x86_64 CentOS-7 - Extras 305
!nagios-base Nagios 96
!nagiosxi-deps Nagios XI Dependencies 31
!updates/7/x86_64 CentOS-7 - Updates 972
repolist: 24,992

net

Code: Select all

netstat -tuapn | grep 180.188.20.33
[root@coanagiosxi ~]# netstat -tuapn | grep 180.188.20.33
-bash: netstat: command not found


If this is a VM, power it off and take a VM snapshot.

The first step would be to power it off, the second step is to boot with no network connected and investigate what is running on the system (as root) and scan it with antivirus/antirootkit/antimalware, change passwords, etc:
I am not sure how to run an antivirus scan on a non windows machine. Running one on the VMHost revealed nothing.

Code: Select all

ps aux
[root@coanagiosxi ~]# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.4 0.0 125652 4052 ? Ss 13:56 0:06 /usr/lib/syste
root 2 0.0 0.0 0 0 ? S 13:56 0:00 [kthreadd]
root 4 0.0 0.0 0 0 ? S< 13:56 0:00 [kworker/0:0H]
root 6 0.0 0.0 0 0 ? R 13:56 0:00 [ksoftirqd/0]
root 7 0.0 0.0 0 0 ? S 13:56 0:00 [migration/0]
root 8 0.0 0.0 0 0 ? S 13:56 0:00 [rcu_bh]
root 9 0.1 0.0 0 0 ? R 13:56 0:02 [rcu_sched]
root 10 0.0 0.0 0 0 ? S< 13:56 0:00 [lru-add-drain
root 11 0.0 0.0 0 0 ? S 13:56 0:00 [watchdog/0]
root 12 0.0 0.0 0 0 ? S 13:56 0:00 [watchdog/1]
root 13 0.0 0.0 0 0 ? S 13:56 0:00 [migration/1]
root 14 0.1 0.0 0 0 ? S 13:56 0:01 [ksoftirqd/1]
root 16 0.0 0.0 0 0 ? S< 13:56 0:00 [kworker/1:0H]
root 18 0.0 0.0 0 0 ? S 13:56 0:00 [kdevtmpfs]
root 19 0.0 0.0 0 0 ? S< 13:56 0:00 [netns]
root 20 0.0 0.0 0 0 ? S 13:56 0:00 [khungtaskd]
root 21 0.0 0.0 0 0 ? S< 13:56 0:00 [writeback]
root 22 0.0 0.0 0 0 ? S< 13:56 0:00 [kintegrityd]
root 23 0.0 0.0 0 0 ? S< 13:56 0:00 [bioset]
root 24 0.0 0.0 0 0 ? S< 13:56 0:00 [bioset]
root 25 0.0 0.0 0 0 ? S< 13:56 0:00 [bioset]
root 26 0.0 0.0 0 0 ? S< 13:56 0:00 [kblockd]
root 27 0.0 0.0 0 0 ? S< 13:56 0:00 [md]
root 28 0.0 0.0 0 0 ? S< 13:56 0:00 [edac-poller]
root 29 0.0 0.0 0 0 ? S< 13:56 0:00 [watchdogd]
root 30 0.0 0.0 0 0 ? S 13:56 0:01 [kworker/1:1]
root 35 0.0 0.0 0 0 ? S 13:56 0:00 [kswapd0]
root 36 0.0 0.0 0 0 ? SN 13:56 0:00 [ksmd]
root 37 0.0 0.0 0 0 ? SN 13:56 0:00 [khugepaged]
root 38 0.0 0.0 0 0 ? S< 13:56 0:00 [crypto]
root 46 0.0 0.0 0 0 ? S< 13:56 0:00 [kthrotld]
root 47 0.0 0.0 0 0 ? S 13:56 0:00 [kworker/u256:
root 48 0.0 0.0 0 0 ? S< 13:56 0:00 [kmpath_rdacd]
root 49 0.0 0.0 0 0 ? S< 13:56 0:00 [kaluad]
root 51 0.0 0.0 0 0 ? S< 13:56 0:00 [kpsmoused]
root 52 0.0 0.0 0 0 ? S 13:56 0:01 [kworker/0:2]
root 53 0.0 0.0 0 0 ? S< 13:56 0:00 [ipv6_addrconf
root 66 0.0 0.0 0 0 ? S< 13:56 0:00 [deferwq]
root 101 0.0 0.0 0 0 ? S 13:56 0:00 [kauditd]
root 274 0.0 0.0 0 0 ? S 13:56 0:00 [kworker/u256:
root 275 0.0 0.0 0 0 ? S< 13:56 0:00 [hv_vmbus_con]
root 377 0.0 0.0 0 0 ? S< 13:56 0:00 [ata_sff]
root 411 0.0 0.0 0 0 ? S 13:56 0:00 [scsi_eh_0]
root 416 0.0 0.0 0 0 ? S< 13:56 0:00 [scsi_tmf_0]
root 418 0.0 0.0 0 0 ? S 13:56 0:00 [scsi_eh_1]
root 419 0.0 0.0 0 0 ? S< 13:56 0:00 [scsi_tmf_1]
root 553 0.0 0.0 0 0 ? S 13:56 0:00 [hv_balloon]
root 558 0.0 0.0 0 0 ? S 13:56 0:00 [scsi_eh_2]
root 559 0.0 0.0 0 0 ? S< 13:56 0:00 [scsi_tmf_2]
root 560 0.0 0.0 0 0 ? S< 13:56 0:00 [storvsc_error
root 561 0.0 0.0 0 0 ? S 13:56 0:00 [scsi_eh_3]
root 562 0.0 0.0 0 0 ? S< 13:56 0:00 [scsi_tmf_3]
root 564 0.0 0.0 0 0 ? S< 13:56 0:00 [storvsc_error
root 633 0.0 0.0 0 0 ? S< 13:56 0:00 [kdmflush]
root 634 0.0 0.0 0 0 ? S< 13:56 0:00 [bioset]
root 644 0.0 0.0 0 0 ? S< 13:56 0:00 [kdmflush]
root 645 0.0 0.0 0 0 ? S< 13:56 0:00 [bioset]
root 660 0.0 0.0 0 0 ? S 13:56 0:00 [jbd2/dm-0-8]
root 661 0.0 0.0 0 0 ? S< 13:56 0:00 [ext4-rsv-conv
root 740 0.1 0.0 39080 7472 ? Ss 13:56 0:01 /usr/lib/syste
root 760 0.0 0.0 198568 3404 ? Ss 13:56 0:00 /usr/sbin/lvme
root 769 0.0 0.0 45244 2356 ? Ss 13:56 0:00 /usr/lib/syste
root 981 0.0 0.0 0 0 ? S 13:56 0:00 [jbd2/sda1-8]
root 990 0.0 0.0 0 0 ? S< 13:56 0:00 [ext4-rsv-conv
root 1129 0.0 0.0 55528 892 ? S<sl 13:56 0:00 /sbin/auditd
polkitd 1154 0.0 0.0 612816 12780 ? Ssl 13:56 0:00 /usr/lib/polki
root 1156 0.0 0.0 21536 1212 ? Ss 13:56 0:00 /usr/sbin/irqb
dbus 1161 0.2 0.0 58432 2656 ? Ss 13:56 0:03 /usr/bin/dbus-
chrony 1167 0.0 0.0 117804 1832 ? S 13:56 0:00 /usr/sbin/chro
root 1172 0.0 0.0 26508 1848 ? Ss 13:56 0:01 /usr/lib/syste
root 1174 0.0 0.0 126320 1652 ? Ss 13:56 0:00 /usr/sbin/cron
root 1207 0.0 0.1 358748 29492 ? Ssl 13:56 0:01 /usr/bin/pytho
root 1208 0.0 0.0 0 0 ? S< 13:56 0:00 [kworker/0:1H]
root 1209 0.0 0.0 552128 8996 ? Ssl 13:56 0:00 /usr/sbin/Netw
root 1362 0.0 0.0 0 0 ? S< 13:56 0:00 [kworker/1:1H]
root 1531 0.0 0.0 229064 6888 ? Ss 13:56 0:00 /usr/sbin/snmp
root 1538 0.0 0.1 519944 20016 ? Ss 13:56 0:00 /usr/sbin/http
root 1539 0.0 0.0 27168 1080 ? Ss 13:56 0:00 /usr/sbin/xine
root 1540 0.0 0.0 216420 5756 ? Ssl 13:56 0:00 /usr/sbin/rsys
root 1541 0.0 0.0 112920 4364 ? Ss 13:56 0:00 /usr/sbin/sshd
root 1544 0.0 0.1 574200 17428 ? Ssl 13:56 0:00 /usr/bin/pytho
shellin+ 1548 0.0 0.0 42908 2764 ? Ss 13:56 0:00 /usr/sbin/shel
nagios 1556 0.0 0.0 371248 1008 ? Sl 13:56 0:00 /usr/local/nag
root 1557 0.5 0.0 18808 1004 ? Ssl 13:56 0:08 /usr/sbin/rsys
root 1558 16.1 0.0 59216 1312 ? Ssl 13:56 4:13 netstat -antop
shellin+ 1580 0.0 0.0 42908 808 ? S 13:56 0:00 /usr/sbin/shel
mysql 1767 0.0 0.0 113316 1596 ? Ss 13:56 0:00 /bin/sh /usr/b
root 1993 0.0 0.0 193436 11624 ? Ss 13:56 0:00 /usr/bin/perl
snmptt 1994 0.0 0.0 195560 12232 ? Ss 13:56 0:00 /usr/bin/perl
apache 1996 0.2 0.1 640644 31112 ? S 13:56 0:03 /usr/sbin/http
apache 1998 0.2 0.1 635692 26452 ? S 13:56 0:03 /usr/sbin/http
apache 1999 0.2 0.1 638292 29128 ? S 13:56 0:04 /usr/sbin/http
apache 2001 0.2 0.1 635700 26388 ? S 13:56 0:03 /usr/sbin/http
apache 2002 0.2 0.1 635692 26456 ? S 13:56 0:04 /usr/sbin/http
mysql 2037 1.8 0.5 1569432 92912 ? Sl 13:56 0:29 /usr/libexec/m
root 2103 0.0 0.0 89700 2184 ? Ss 13:56 0:00 /usr/libexec/p
postfix 2105 0.0 0.0 89980 4308 ? S 13:56 0:00 qmgr -l -t uni
apache 2199 0.1 0.1 641156 31568 ? S 13:56 0:02 /usr/sbin/http
nagios 2303 0.0 0.0 56172 2812 ? Ss 13:56 0:00 /usr/local/nag
root 2323 0.0 0.0 99208 2688 ? Ss 13:56 0:00 login -- root
nagios 2340 0.1 0.0 21908 2300 ? Ss 13:56 0:02 /usr/local/nag
nagios 2341 0.0 0.0 10848 1124 ? S 13:56 0:00 /usr/local/nag
nagios 2342 0.0 0.0 10848 1120 ? S 13:56 0:00 /usr/local/nag
nagios 2343 0.0 0.0 10848 1112 ? S 13:56 0:00 /usr/local/nag
nagios 2344 0.0 0.0 10848 1124 ? S 13:56 0:00 /usr/local/nag
nagios 2347 0.0 0.0 129904 2000 ? S 13:56 0:00 /usr/local/nag
nagios 2348 0.2 0.0 130180 2328 ? S 13:56 0:03 /usr/local/nag
nagios 2353 0.0 0.0 21392 1052 ? S 13:56 0:00 /usr/local/nag
root 2835 0.0 0.0 115448 2064 tty1 Ss+ 13:57 0:00 -bash
root 12300 0.0 0.0 0 0 ? S 14:07 0:00 [kworker/1:3]
root 14358 0.1 0.0 0 0 ? S 14:10 0:01 [kworker/0:1]
root 18868 0.1 0.0 0 0 ? S 14:16 0:00 [kworker/0:0]
root 18920 0.0 0.0 0 0 ? S 14:16 0:00 [kworker/1:0]
root 20947 0.0 0.0 102896 5488 ? S 14:19 0:00 /sbin/dhclient
apache 20966 1.1 0.1 635700 26476 ? S 14:19 0:02 /usr/sbin/http
apache 21074 0.8 0.1 635692 26392 ? S 14:19 0:01 /usr/sbin/http
postfix 21672 0.0 0.0 89804 4064 ? S 14:19 0:00 pickup -l -t u
root 22183 0.0 0.0 0 0 ? S 14:19 0:00 [kworker/u256:
apache 22188 0.9 0.1 635700 26376 ? S 14:19 0:01 /usr/sbin/http
root 22637 0.1 0.0 159380 6452 ? Ss 14:20 0:00 sshd: root@pts
root 22716 0.0 0.0 115448 2060 pts/0 Ss 14:20 0:00 -bash
root 23794 0.0 0.0 112920 4292 ? Ss 14:21 0:00 sshd: [accepte
apache 23795 3.3 0.1 635692 26452 ? S 14:21 0:02 /usr/sbin/http
sshd 23885 0.0 0.0 112920 2224 ? S 14:21 0:00 sshd: [net]
root 24248 0.0 0.0 182404 2532 ? S 14:22 0:00 /usr/sbin/CRON
root 24249 0.0 0.0 182404 2532 ? S 14:22 0:00 /usr/sbin/CRON
root 24250 0.0 0.0 182404 2532 ? S 14:22 0:00 /usr/sbin/CRON
root 24251 0.0 0.0 182404 2532 ? S 14:22 0:00 /usr/sbin/CRON
root 24252 0.0 0.0 182404 2532 ? S 14:22 0:00 /usr/sbin/CRON
root 24253 0.0 0.0 182404 2532 ? S 14:22 0:00 /usr/sbin/CRON
nagios 24255 0.0 0.0 113184 1204 ? Ss 14:22 0:00 /bin/sh -c /us
nagios 24256 0.0 0.0 113184 1204 ? Ss 14:22 0:00 /bin/sh -c /us
nagios 24257 0.0 0.0 113184 1204 ? Ss 14:22 0:00 /bin/sh -c /us
nagios 24260 0.7 0.1 444752 25908 ? S 14:22 0:00 /usr/bin/php -
nagios 24261 0.6 0.1 444620 25728 ? S 14:22 0:00 /usr/bin/php -
nagios 24262 1.2 0.2 451664 32736 ? S 14:22 0:00 /usr/bin/php -
nagios 24264 0.0 0.0 113184 1204 ? Ss 14:22 0:00 /bin/sh -c /us
nagios 24266 0.0 0.0 113184 1204 ? Ss 14:22 0:00 /bin/sh -c /us
nagios 24269 0.0 0.0 113184 1204 ? Ss 14:22 0:00 /bin/sh -c /us
nagios 24272 1.0 0.2 451924 32904 ? S 14:22 0:00 /usr/bin/php -
nagios 24273 0.6 0.1 444752 25828 ? S 14:22 0:00 /usr/bin/php -
nagios 24275 0.7 0.1 444748 25956 ? S 14:22 0:00 /usr/bin/php -
root 24347 0.1 0.0 0 0 ? R 14:22 0:00 [kworker/0:3]
root 24924 0.0 0.0 1452 840 ? Ss 14:22 0:00 cd /etc
root 24931 0.0 0.0 1452 840 ? Ss 14:22 0:00 pwd
root 24938 0.0 0.0 1452 840 ? Ss 14:22 0:00 route -n
root 24943 0.0 0.0 1452 840 ? Ss 14:22 0:00 route -n
root 24944 0.0 0.0 1452 836 ? Ss 14:22 0:00 gnome-terminal
nagios 24967 0.0 0.0 113184 1320 ? S 14:22 0:00 sh -c /usr/bin
nagios 24968 0.0 0.0 108000 812 ? S 14:22 0:00 /usr/bin/iosta
nagios 24969 0.0 0.0 107992 684 ? S 14:22 0:00 tail --lines=2
nagios 24970 0.0 0.0 107964 668 ? S 14:22 0:00 head --lines=1
nagios 24971 0.0 0.0 113544 976 ? S 14:22 0:00 awk { print $1
root 24983 0.0 0.0 1396 868 ? Ss 14:22 0:00 /usr/sbin/rsys
root 24986 0.0 0.0 1396 868 ? Ss 14:22 0:00 /lib/systemd/s
root 24989 0.0 0.0 1396 868 ? Ss 14:22 0:00 nautilus -n
root 24992 0.0 0.0 1396 872 ? Ss 14:22 0:00 automount
root 24994 0.0 0.0 1396 864 ? Ss 14:22 0:00 /usr/sbin/acpi
nagios 24997 11.0 0.0 135720 4296 ? R 14:22 0:00 /usr/bin/perl
nagios 24998 10.0 0.0 136116 4560 ? R 14:22 0:00 /usr/bin/perl
root 25000 0.0 0.0 155372 1880 pts/0 R+ 14:22 0:00 ps aux


Then do something like this to capture a pcap file to investigate what it is trying to send:

Code: Select all

https://www.wireshark.org/docs/wsug_html_chunked/AppToolstcpdump.html
[root@coanagiosxi ~]# https://www.wireshark.org/docs/wsug_htm ... pdump.html
-bash: https://www.wireshark.org/docs/wsug_htm ... pdump.html: No such file or directory


I am not a centos user. I dabble only a little
-Adam
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: NagiosXI Network Maxed

Post by tgriep »

Please run this as root to install netstat and tcpdump.

Code: Select all

yum install tcpdump  net-tools -y
Run as root and post the outout to the ticket.

Code: Select all

netstat -anp | grep 180.188.20.33

Then, run the following to capture the traffic in to a tcpdump capture file so you can upload it to the forum for analysis.

Code: Select all

tcpdump -i any -s 65535 -w host.cap host 180.188.20.33
Let it run for 5 to 10 minutes, stop the tcpdump and upload the host.cap file so we can view it.
Be sure to check out our Knowledgebase for helpful articles and solutions!
abishop
Posts: 25
Joined: Fri Dec 13, 2019 9:59 am

Re: NagiosXI Network Maxed

Post by abishop »

tgriep wrote:Please run this as root to install netstat and tcpdump.

Code: Select all

yum install tcpdump  net-tools -y
DONE
Run as root and post the outout to the ticket.

Code: Select all

netstat -anp | grep 180.188.20.33
I ran this, but nothing outputted. Is there a location for me to get this at with winscp?
Then, run the following to capture the traffic in to a tcpdump capture file so you can upload it to the forum for analysis.

Code: Select all

tcpdump -i any -s 65535 -w host.cap host 180.188.20.33
Let it run for 5 to 10 minutes, stop the tcpdump and upload the host.cap file so we can view it.
host.cap file not allowed for upload. I renamed it to host.txt
You do not have the required permissions to view the files attached to this post.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: NagiosXI Network Maxed

Post by scottwilkerson »

That's basically empty. What were the results of ?

Code: Select all

netstat -anp | grep 180.188.20.33
Where did you see that the Nagios XI server was reaching out to 180.188.20.33:80 ?
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
abishop
Posts: 25
Joined: Fri Dec 13, 2019 9:59 am

Re: NagiosXI Network Maxed

Post by abishop »

scottwilkerson wrote:That's basicallt empty. What were the results of ?

Code: Select all

netstat -anp | grep 180.188.20.33
Nothing Happens
Where did you see that the Nagios XI server was reaching out to 180.188.20.33:80 ?
pfsense router firewall logs, source being the ip of the XI box and destination 180.188.20.33:80
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: NagiosXI Network Maxed

Post by scottwilkerson »

Again
What were the results of ?

Code: Select all

netstat -anp | grep 180.188.20.33
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
abishop
Posts: 25
Joined: Fri Dec 13, 2019 9:59 am

Re: NagiosXI Network Maxed

Post by abishop »

Nothing... When I put that in nothing happens.


[root@coanagiosxi ~]# netstat -anp | grep 180.188.20.33
[root@coanagiosxi ~]#
You do not have the required permissions to view the files attached to this post.
Locked