Nagios server compromised

[bsingh]

Hi,

We noticed CPU usage going very and then noticed process ".resyslogd" was the culprit.
The location of this file is under below path :

/var/tmp/.VMX/.resyslogd

Should I consider as system compromised ? I have no clue about ".resyslogd" . Note - it is not rsyslog.
Because ".resyslogd" is chewing up all resources on system so everything on NagiOS is slow and getting all kind of issues.

Here config detail :

OS - CentOS 7
Nagios version : Nagios XI 5.4.3

This machine is VM off of the proxmox virtual environment. What are recommended settings on machine as for as configuration goes ? Can anyone help ?

Thanks

[tmcdonald]

That is nothing of ours.

Standard security procedure would be to quarantine and investigate offline, but that is out of our scope. I would contact your security team immediately.

[tmcdonald]

For an update on this, I was able to identify one other user who had the same issue and after a remote session we determined that it was a vulnerability in an outdated XI version that allowed for this:

https://www.nagios.com/news/2018/05/sec ... xi-5-4-13/

I would strongly urge you either to update to the latest XI version, or to apply the patch provided in the link above.

[bsingh]

I ran vuln_patch.sh and all went OK. Then ran comp_detec.sh and be is output I get :

./comp_detect.sh

This script detects whether your Nagios XI system has been compromised by an
as-yet-unnamed exploit we detected on a customer system. If it has been,
please make sure to clean the affected system and upgrade to the latest version
of Nagios XI (5.4.13 at the time of this writing).

[!] Indicator of compromise detected in Apache logs

Now, trying to delete below file but fails :

rm -rf /var/tmp/.VMX/
rm: cannot remove ‘/var/tmp/.VMX/.resyslogd’: Operation not permitted


Remember ".resyslogd" is the main culprit which is hogging CPU and we need to remove it.
Can you please help ?

Thanks

[tmcdonald]

From what I was able to gather from the other user, an offline removal of that file was required. There are active defenses in-place that the process employs to prevent direct removal. My advice is to power off the machine, and either load a Live CD version of a Linux OS, or to mount the disk on another machine and remove the suspect file.

[bsingh]

It looks like I'm able to delete that file with changing attribute & booting in single user mode.

chattr -R -i /var/tmp/.VMX
rm -rf /var/tmp/.VMX


Now running ./comp_detect.sh :

./comp_detect.sh

This script detects whether your Nagios XI system has been compromised by an
as-yet-unnamed exploit we detected on a customer system. If it has been,
please make sure to clean the affected system and upgrade to the latest version
of Nagios XI (5.4.13 at the time of this writing).

[!] Indicator of compromise detected in Apache logs

The process ".resyslogd" has not run yet about in an hour. Does it mean we are out of danger ?
Can you pls suggest what other files need to be removed (if any) ?

Thanks

[tmcdonald]

To clarify, the comp_detect.sh only checks the Apache logs for indications that the vulnerability has been exploited - it does not look for the specific malware you are dealing with.

The execution scheme for the malware was an in-memory cron job that ran every 12 minutes, as well as some accompanying bash processes that killed off other processes that might be trying to remove the malware. Without that in-memory job, and with the malware binary off the system, I believe you are safe as far as the infection goes. You will still need to patch or upgrade the system however to prevent further infection.

I do need to point out though that I have only done very minimal investigation into the malware binary itself, and while I believe it is a simple cryptocurrency miner only, I am not a professional malware analyst by trade. It would be advisable to have a professional on your team take a look at the system.

[riton]

Hi,
Happy to find you , i get exactly the same problem as describe on a VPS server (ubuntu 14.04.5 LTS) (nothing to do with Nagios) :

I identify the process by using top command or htop command :
#top
PID NAME
2035 ./.resyslogd 100% PROC USER

To identify files used by this process :
# lsof -p 2035
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
.resyslog 2035 user cwd DIR 9,1 4096 2 /
.resyslog 2035 user rtd DIR 9,1 4096 2 /
.resyslog 2035 user txt REG 9,2 670380 524295 /var/tmp/.VMX/.resyslogd
.resyslog 2035 user mem REG 9,1 101240 573 /lib/x86_64-linux-gnu/libresolv-2.19.so
.resyslog 2035 user mem REG 9,1 22952 591 /lib/x86_64-linux-gnu/libnss_dns-2.19.so
.resyslog 2035 user mem REG 9,1 149120 582 /lib/x86_64-linux-gnu/ld-2.19.so
.resyslog 2035 user mem REG 9,1 1857312 593 /lib/x86_64-linux-gnu/libc-2.19.so
.resyslog 2035 user mem REG 9,1 43616 571 /lib/x86_64-linux-gnu/libnss_files-2.19.so
.resyslog 2035 user 0r CHR 1,3 0t0 1029 /dev/null
.resyslog 2035 user 1w CHR 1,3 0t0 1029 /dev/null
.resyslog 2035 user 2w CHR 1,3 0t0 1029 /dev/null
.resyslog 2035 user 3u 0000 0,10 0 2058 anon_inode
.resyslog 2035 user 4r FIFO 0,9 0t0 375153 pipe
.resyslog 2035 user 5w FIFO 0,9 0t0 375153 pipe
.resyslog 2035 user 6r FIFO 0,9 0t0 372454 pipe
.resyslog 2035 user 7w FIFO 0,9 0t0 372454 pipe
.resyslog 2035 user 8u 0000 0,10 0 2058 anon_inode
.resyslog 2035 user 9r CHR 1,3 0t0 1029 /dev/null
.resyslog 2035 user 10u IPv4 375155 0t0 TCP server.ip-211-11-11.eu:42274->server.ip-166-155-277.eu:http (ESTABLISHED)

Can you specify the way you identify/locate the file to delete in a safe mode to definitly clean this process ?
If system is compromised, what shoud I clean : files/database ?
Can you also specify the vulnerabilities used by this malware and how to correct (what to update) ?

As indicated, it seems to be a cryptocurrency mining malware :
https://www.hackread.com/bytecoin-crypt ... nap-store/

Thanks,

[tmcdonald]

Thanks for the extra input.

I am going to close this up as it has been a while since we have heard from the OP and there is sufficient information to resolve the issue.