Multiple Nagios processes running

[awilson]

Hi. One of our Nagios XI servers is running six Nagios processes. I haven't noticed performance issues. I just noticed it checking another point.

Code: Select all

[root@nagiosserver ~]# ps -ef | grep "/usr/local/nagios/bin/nagios -d"
root       794 39081  0 14:58 pts/0    00:00:00 grep /usr/local/nagios/bin/nagios -d
nagios    4726     1  0 Jul29 ?        00:00:02 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios    4904  4726  0 Jul29 ?        00:07:50 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   18336     1  1 13:58 ?        00:01:10 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   18588 18336  0 13:58 ?        00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   32326     1  0 Aug08 ?        00:01:23 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   32504 32326  0 Aug08 ?        00:07:14 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

I ran pstree -l after reviewing other posts

Code: Select all

[root@nagiosserver ~]# pstree -l
init─┬─VGAuthService
     ├─abrt-dump-oops
     ├─abrtd
     ├─atd
     ├─auditd───{auditd}
     ├─certmonger
     ├─crond───11*[crond───sh───php]
     ├─dbus-daemon
     ├─hald─┬─hald-runner─┬─hald-addon-acpi
     │      │             └─hald-addon-inpu
     │      └─{hald}
     ├─httpd───11*[httpd]
     ├─irqbalance
     ├─lighttpd
     ├─master─┬─cleanup
     │        ├─local
     │        ├─pickup
     │        ├─qmgr
     │        ├─2*[smtp]
     │        └─trivial-rewrite
     ├─6*[mingetty]
     ├─mysqld_safe───mysqld───55*[{mysqld}]
     ├─2*[nagios───19*[nagios]]
     ├─nagios─┬─2*[nagios─┬─2*[check_icmp]]
     │        │           └─2*[check_nrpe]]
     │        ├─nagios───6*[check_icmp]
     │        ├─nagios─┬─4*[check_icmp]
     │        │        └─check_nrpe
     │        ├─nagios─┬─2*[check_icmp]
     │        │        └─4*[check_nrpe]
     │        ├─2*[nagios─┬─check_icmp]
     │        │           └─2*[check_nrpe]]
     │        ├─nagios─┬─check_http
     │        │        ├─4*[check_icmp]
     │        │        └─2*[check_nrpe]
     │        ├─nagios─┬─3*[check_icmp]
     │        │        └─check_tcp
     │        ├─nagios─┬─3*[check_icmp]
     │        │        └─2*[check_nrpe]
     │        ├─2*[nagios───check_icmp]
     │        ├─nagios─┬─check_icmp
     │        │        └─check_nrpe
     │        ├─nagios─┬─2*[check_icmp]
     │        │        └─check_tcp
     │        ├─nagios─┬─3*[check_icmp]
     │        │        └─check_nrpe
     │        ├─nagios─┬─2*[check_icmp]
     │        │        └─3*[check_nrpe]
     │        ├─nagios─┬─3*[check_icmp]
     │        │        ├─check_nrpe
     │        │        └─check_tcp
     │        ├─nagios───check_nrpe
     │        └─nagios
     ├─ndo2db───3*[ndo2db───ndo2db]
     ├─npcd
     ├─ntpd
     ├─oddjobd
     ├─osad
     ├─portreserve
     ├─postmaster───27*[postmaster]
     ├─python───16*[{python}]
     ├─python───{python}
     ├─rhsmcertd
     ├─rhsmd
     ├─rpcbind
     ├─rsyslogd───3*[{rsyslogd}]
     ├─snmptrapd───sh───snmptthandler
     ├─snmptt───snmptt
     ├─splunkd─┬─splunkd
     │         └─36*[{splunkd}]
     ├─sshd───sshd───sshd───bash───sudo───su───bash───pstree
     ├─sssd─┬─sssd_be
     │      ├─sssd_nss
     │      └─sssd_pam
     ├─tcl
     ├─udevd
     ├─vmtoolsd───vmtoolsd
     ├─wrapper───java───20*[{java}]
     └─xinetd
[root@nagiosserver ~]#

Is this normal?

Thanks.

[scottwilkerson]

That is not normal.

Run the following

Code: Select all

service nagios stop
killall -9 nagios
rm -rf /var/run/nagios.lock
service nagios start

[awilson]

Now there are two:

Code: Select all

nagios   62083     1  6 15:23 ?        00:00:01 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   62249 62083  0 15:23 ?        00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

[scottwilkerson]

Now there are two:

Code: Select all

nagios   62083     1  6 15:23 ?        00:00:01 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   62249 62083  0 15:23 ?        00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

This is exactly how it should look, only 1 parent process

[awilson]

Are you aware of a standard service check that could watch for this condition? Are there log files that we should monitor to identify that multiple processes are active?

Thx

[scottwilkerson]

I don't know of any, but you could make a plugin from the following command

Code: Select all

ps -ef|grep nagios.cfg|grep -v grep|grep -v `cat /var/run/nagios.lock`|wc -l

If that returns anything but 0 you may have a problem

[ssax]

This should work:

Add this to /usr/local/nagios/libexec/check_nagios_procs.sh:

Code: Select all

#!/bin/bash
NPROCCOUNT=`ps aux | grep nagios.cfg | grep -v grep | wc -l`

if [ "$NPROCCOUNT" -gt "2" ]; then
    echo "CRITICAL - Too many nagios service processes, should NEVER have more than 2"
    exit 2
else
    echo "OK - Total Nagios Service Processes less than 2"
    exit 0
fi

Then run these commands:

Code: Select all

chown nagios.nagios /usr/local/nagios/libexec/check_nagios_procs.sh
chmod +x /usr/local/nagios/libexec/check_nagios_procs.sh

Then go to Configure > Core Config Manager > Commands:
- Click Add New
- Use these values:

Command Name: check_nagios_procs
Command Line: $USER1$/check_nagios_procs.sh
Command Type: check command

- Check the Active box
- Save
- Setup a new service and select the command to use.
- Apply Configuration

That should do it, let us know the results.

[benjaminsmith]

Hello @awilson,

It's been while our last post. Did you have any other questions or shall we close this post? When you have a moment, please let us know. Thanks.

[awilson]

You can close. Thank you for the suggestion. I'll try the script.

[benjaminsmith]

Hello @awilson.

Happy to help. We'll lock this up. Feel free to open another post if you have any questions.