Page 1 of 1
Carbon Black "next generation" antivirus on Nagios XI
Posted: Mon Mar 12, 2018 1:27 pm
by DFaught
Our security group insists that the Carbon Black endpoint protection software be installed on the Nagios XI servers. When this was installed, the more active Nagios XI servers went immediately to 100% CPU and started showing signs of stress like service checks timing out that did not before. The Security group of course also insists that if there are any problems that it must not be Carbon Black that is the cause. Are there certain things on the Nagios XI server that should be excluded from antivirus protection? Is there something else within reason that I should look at to resolve this situation?
Thank you for your help.
Moderator Edit: Profile received and shared with team
Re: Carbon Black "next generation" antivirus on Nagios XI
Posted: Mon Mar 12, 2018 1:39 pm
by tmcdonald
Unfortunately, very often the answer to the question of "Will my AV break XI?" is "It works until it doesn't".
I'd start by looking at what processes are pegging the CPU, and posting them here. A screenshot of top would be a good start. If they are XI processes we can help narrow it down and build up an exclusion list. Otherwise you might want to contact the Carbon Black vendor and see if they can look at it from their side. We can't really tell you how their software might affect ours any more than they can tell you the reverse, but at least we can help decide where the blame lies.
Update: There are a *lot* of defunct processes in your profile. I would see about excluding check_rrdtraf in CB and see if that improves anything.
Re: Carbon Black "next generation" antivirus on Nagios XI
Posted: Mon Mar 12, 2018 1:45 pm
by DFaught
Here is a top display. This seems to be pretty typical now.
Code: Select all
[dfaught@mlwnag22]:[/home/dfaught]# top
top - 14:42:58 up 2:17, 1 user, load average: 8.75, 9.00, 9.22
Tasks: 1627 total, 8 running, 365 sleeping, 0 stopped, 1254 zombie
%Cpu(s): 2.6 us, 97.4 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16251592 total, 12364592 free, 2163460 used, 1723540 buff/cache
KiB Swap: 8388604 total, 8388604 free, 0 used. 13436080 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4017 nagios 20 0 11268 1700 840 R 83.4 0.0 66:28.53 nagios
4016 nagios 20 0 11276 1692 824 R 59.0 0.0 66:12.98 nagios
4021 nagios 20 0 11288 1704 824 R 53.7 0.0 66:29.31 nagios
4018 nagios 20 0 11272 1712 840 R 51.8 0.0 66:21.81 nagios
4020 nagios 20 0 11280 1696 824 R 50.2 0.0 65:44.44 nagios
4019 nagios 20 0 11272 1692 824 R 49.8 0.0 66:21.26 nagios
4013 nagios 20 0 49736 23160 1492 R 49.5 0.1 66:17.29 nagios
108004 dfaught 20 0 163736 4036 1708 R 1.0 0.0 0:00.20 top
564 root 20 0 0 0 0 S 0.3 0.0 0:06.01 b9-DirtyTrackin
1 root 20 0 199436 4272 2524 S 0.0 0.0 0:07.04 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.21 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:00.16 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 0:02.82 rcu_sched
10 root rt 0 0 0 0 S 0.0 0.0 0:00.04 watchdog/0
11 root rt 0 0 0 0 S 0.0 0.0 0:00.02 watchdog/1
12 root rt 0 0 0 0 S 0.0 0.0 0:00.15 migration/1
13 root 20 0 0 0 0 S 0.0 0.0 0:00.19 ksoftirqd/1
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
16 root rt 0 0 0 0 S 0.0 0.0 0:00.02 watchdog/2
17 root rt 0 0 0 0 S 0.0 0.0 0:00.15 migration/2
18 root 20 0 0 0 0 S 0.0 0.0 0:00.22 ksoftirqd/2
20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H
21 root rt 0 0 0 0 S 0.0 0.0 0:00.02 watchdog/3
22 root rt 0 0 0 0 S 0.0 0.0 0:00.15 migration/3
23 root 20 0 0 0 0 S 0.0 0.0 0:00.19 ksoftirqd/3
25 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H
27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
28 root 0 -20 0 0 0 S 0.0 0.0 0:00.02 netns
29 root 20 0 0 0 0 S 0.0 0.0 0:00.01 khu
Re: Carbon Black "next generation" antivirus on Nagios XI
Posted: Mon Mar 12, 2018 1:53 pm
by tmcdonald
Please see the update to my post, which I will include here:
tmcdonald wrote:There are a *lot* of defunct processes in your profile. I would see about excluding check_rrdtraf in CB and see if that improves anything.
If CB is killing off those processes but leaving them defunct that would certainly explain why the
nagios processes are spinning in place.
Re: Carbon Black "next generation" antivirus on Nagios XI
Posted: Tue Mar 13, 2018 1:30 pm
by gwakem
I noticed the same issue in an install of Carbon Black on XI some time ago. We also found it affected any RHEL servers running BIND, causing kernel segfaults. Due to the way Carbon Black checks every process, the RRD writing did cause CB to produce enormously high loads, as did many of the other checks. I would not recommend running it on XI (not that it makes a difference in my experience when dealing with the security people.)
Re: Carbon Black "next generation" antivirus on Nagios XI
Posted: Tue Mar 13, 2018 3:45 pm
by npolovenko
@gwakem, Thanks for sharing your experience with us. I'm sure other users will find it helpful.
Do you have other questions so far or it ok to lock the thread?