localhost CPU load average Critical ...

[PhilG]

Hello:
I have upgraded Nagios XI to the latest version 5.8.1.
After the upgrade, I'm seeing that the localhost CPU current load is critical (red) for almost an hour as of this time Wed. 1/27/2021, 11:13 am CST - CRITICAL - load average: 4.31, 4.34, 4.38

What should I be looking for to resolve this?

[dchurch]

Have you inspected the process list to see what's taking up CPU? Run the following command to inspect this

Code: Select all

top

If you send over a system profile, I can diagnose further.

[PhilG]

The localhost CPU load has calmed down and we're keeping an eye one it.
I have noted something suspicious that was running on the server: /tmp/avalonsaber
I have killed that for now and made it non-executable and will be looked into, but I'm not saying this was the main issue.
Please keep this open, but I may not respond for some time. Okay? Thank you.

[PhilG]

The "/tmp/avalonsaber" executable magically recreated itself and was running again.

Both Apache and Apache group are owners of the file, and I haven’t heard of that executable before.

This is what I could identify by running “ps -aux | grep "avalonsaber"”
apache 5083 166 14.6 2439040 2402280 ? Sl 16:20 13:38 /tmp/avalonsaber --rig-id=19 -o sslcer.justdied.com:3333

The Domain reference “sslcer.justdied.com” has IP 185.183.84.197. That IP was not found in the “AbuseIPDB” database. That IP is not in our subnet and we don’t own that IP/Domain.

I did a search through the logs I know and didn’t get a hit on “avalonsaber”.

My Google searching has come up empty.

Does Nagios/Nagios XI create this/use this?

[dchurch]

I have some bad news: Avalonsaber is malware that mines BitCoin for the attacker who put it on your system. It's not something that any Nagios product ships with.

You can read about this security disclosure at the nist.gov database, under CVE-2021-3193. You can see this, past, and future Nagios security disclosures here.

Good new is that we have patched the vulnerability that lead to the infection. What you will want to do is:

1. Update your Docker config Wizard to 1.1.2 (or higher) as soon as you can. Instructions for how to do that can be found at this link: https://support.nagios.com/kb/article/n ... s-836.html

2. You will also want to run as the root user to disinfect the machine (script version 2021-02-15):

Code: Select all

for (( x = 0; x < 100; ++x)); do
for (( x = 0; x < 100; ++x)); do
chattr -i /etc/crontab
chattr -i /tmp/avalonsaber
chattr -R -i /var/spool/cron
crontab -r -u apache
crontab -r -u www-data

pkill -9 avalonsaber
pkill -9 lwp-download
pkill -9 curl
pkill -9 wget

rm -rf /tmp/avalonsaber
done

Is this a public-facing machine? Or inside your firewall?

[PhilG]

First, THANK YOU for that information!
It "should" be behind a Firewall, BUT has ports 80 and 443 open for public facing access.
I will need to encourage getting it inside the DMZ and have the support team access only through VPN.

[dchurch]

I will need to encourage getting it inside the DMZ and have the support team access only through VPN.

Yes, other customers have done that, and logically that would be an effective countermeasure. The infection same over an unprotected page; if the whole internet had access to pull up the login page, then they had access to the page with this exploit.

One gotcha I can think of is that passive checks come in to the HTTP server over the same port and web server that the Nagios XI web interface is running on; if you restrict the IP addresses that can access the HTTP site, you may have to also allow from the IP addresses of the hosts sending passive checks.

If you're not using passive checks, this point is moot though.