Too many Nagios3 processes

[eoin.kim]

Hi all,

I recently installed Nagios3 on a new Linux virtual machine and it doesn't look right for me. We are using Windows Server 2012 R2 with Hyper-V. The VM is 64 bit Debian Jessie. I downloaded Nagios 3 from the Debian repository and patched for SIGSEGV issue - https://bugs.launchpad.net/ubuntu/+sour ... ug/1372284. After installation, I confirmed that SIGSEGV issue is not happening but found another strange issue which I couldn't figure out why.

So, Nagios3 works fine for a while. However, few hours later, no checking is happening and all I can see is below (I typed systemctl status nagios3.status). This list gets really big (I guess processes are not killed automatically) and eventually the VM suffers from a heavy load (I could see from the top command) and starts sending many false alarms (because plugins get timeout). The issue is gone when I restart the Nagios3. Could I please get help regarding this issue? Thanks in advance.

├─23675 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─23694 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─25414 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─27033 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─28102 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─30149 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─30288 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─31489 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─32064 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─33239 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─34238 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─35172 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─35368 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─36141 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─36995 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─37518 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─38045 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─38325 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─41136 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─42054 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─43243 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─43306 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─43347 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─44215 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─44962 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─46146 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─46320 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─47098 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─48178 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─48263 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─48955 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─48971 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─50300 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─50690 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─51560 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─52805 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─52898 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─53789 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─53972 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─54124 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─54234 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─55534 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─57242 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─57346 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─58034 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─59992 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─62278 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─62511 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─63672 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─64062 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─64177 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
├─64906 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg
└─65420 /usr/sbin/nagios3 -d /etc/nagios3/nagios.cfg

[dwhitfield]

Suspect this was fixed in 4.2.0. Any chance you could update to 4.3.1? Please note the last steps of https://assets.nagios.com/downloads/nag ... ading.html regarding the new apache configs

Fixed init script to wait up to 90 seconds then kill the nagios process (John Frickson)

[eoin.kim]

Hi,

Thanks for your message. I installed Nagios3 from Debian repository. So, I believe the directory structure would be different from the source code. I am not in the level where I can confidently find and set the same directory path. I am also running NagVis and pnp4nagios at the same time. Touching production is risky but if I can do it, I would like to know how to do it a bit more detail. Thanks.

[rkennedy]

Your production is already broken - there isn't much left to lose at this point. All those other components running will affect Nagios as well.

Are the processes all spawned from the same parent, or are their actual multiples?

Spin up a new, clean minimal machine and run with the install instructions here - https://support.nagios.com/kb/article.p ... ategory=58

As you're unfamiliar with the setup, we will be as well. Migrating the configs should be as easy as copying over your etc and plugins directories.

[eoin.kim]

Your production is already broken - there isn't much left to lose at this point. All those other components running will affect Nagios as well.
- Well, I wouldn't say it's broken. It is working fine. It just needs some labour at this stage.

Are the processes all spawned from the same parent, or are their actual multiples?
- I checked the PPID of the items and they are all the same. So, I believe they are all spawned from the same parent.

Spin up a new, clean minimal machine and run with the install instructions here - https://support.nagios.com/kb/article.p ... ategory=58
- Will do, thanks.

As you're unfamiliar with the setup, we will be as well. Migrating the configs should be as easy as copying over your etc and plugins directories.
- Yeah, that's what I've done for this version from the old one. I'll give another go. Cheers.

[dwhitfield]

Spin up a new, clean minimal machine and run with the install instructions here - https://support.nagios.com/kb/article.p ... ategory=58
- Will do, thanks.

Please let us know if you continue to see the problem after the migration.

[eoin.kim]

Hi all,

In the end, the company decided to put the old machine back online because too much downtime is not allowed. A bit tough but all good. Thanks for your help.

[dwhitfield]

I don't know how much time you have for such a project, but it might be worth your time to spin up a Nagios Core 4.3.1 test machine and see if you still have the issues on your network. If you do, then there might be some sort of bug we could fix before you actually look to upgrade in prod.