Nagios not detecting changes in configurations.

[sami21]

Hi there,

I am facing one trouble with my nagios.
It is not detecting changes done in configurations.
I tried below methods.
1. recreation of objects.cache and objects.precache file using option -p and -u
2. deleted retention.dat and status.dat
Is there anything I am missing?
Thanks in Advance.

In Addition - I am facing weird behavior of nagios, sometimes it displays the changes and sometimes it does not.
At one point I thought its working fine, and again on next browser refresh I saw that changes are not visible.

[tmcdonald]

Let's get some basic information:

• What Nagios Core version are you using?
• What OS and version is it installed on?
• Did you compile from source or use something like yum or apt-get to install?
• After you are changing the configs, are you making sure to restart the nagios process?

[sami21]

Hi,
Thanks for your reply.

I upgraded the nagios 4.0.8 to 4.1.1 on centos version 6.x
Yes I did compile the nagios from source.
and yes i am restarting the nagios process after every change.
please let me know if u need any more details.

[rkennedy]

Can you post the full output of ps -aef? It sounds like you may have multiple processes running.

[sami21]

Hi,
Here is the output of ps -aef | grep nagios
NOTE : I deleted plugin processes which were exposing confidential data.

0 S <user> 5195 1 1 80 0 - 5201 ep_pol 06:41 ? 00:04:03 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
1 S <user> 5200 5195 0 80 0 - 3439 poll_s 06:41 ? 00:00:03 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
0 R <user> 8389 1 0 80 0 - 5200 - Jul27 ? 00:08:15 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
0 Z <user> 8391 8389 0 80 0 - 0 exit Jul27 ? 00:00:00 [nagios] <defunct>
0 Z <user> 8392 8389 0 80 0 - 0 exit Jul27 ? 00:00:00 [nagios] <defunct>
0 Z <user> 8393 8389 0 80 0 - 0 exit Jul27 ? 00:00:00 [nagios] <defunct>
1 S <user> 8394 8389 0 80 0 - 3438 poll_s Jul27 ? 00:00:07 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
0 S <user> 8470 1 0 80 0 - 5232 ep_pol Jul27 ? 00:08:16 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
0 Z <user> 8471 8470 0 80 0 - 0 exit Jul27 ? 00:00:00 [nagios] <defunct>
0 Z <user> 8472 8470 0 80 0 - 0 exit Jul27 ? 00:00:00 [nagios] <defunct>
0 Z <user> 8473 8470 0 80 0 - 0 exit Jul27 ? 00:00:00 [nagios] <defunct>
0 Z <user> 8474 8470 0 80 0 - 0 exit Jul27 ? 00:00:00 [nagios] <defunct>
1 S <user> 8475 8470 0 80 0 - 3438 poll_s Jul27 ? 00:00:08 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
0 R <user> 8727 8729 0 80 0 - 2504 - 11:53 ? 00:00:28 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 8728 8729 0 80 0 - 2504 ep_pol 11:53 ? 00:00:29 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
1 S <user> 8729 1 0 80 0 - 6260 ep_pol 10:24 ? 00:01:06 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
0 R <user> 8731 8729 0 80 0 - 2504 - 11:53 ? 00:00:29 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 8732 8729 0 80 0 - 2504 ep_pol 11:53 ? 00:00:29 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
1 S <user> 8802 8729 0 80 0 - 6112 poll_s 10:24 ? 00:00:01 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
0 S <user> 9109 1 0 80 0 - 5199 ep_pol Jul27 ? 00:08:05 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
0 Z <user> 9110 9109 0 80 0 - 0 exit Jul27 ? 00:00:00 [nagios] <defunct>
0 Z <user> 9111 9109 0 80 0 - 0 exit Jul27 ? 00:00:00 [nagios] <defunct>
0 Z <user> 9112 9109 0 80 0 - 0 exit Jul27 ? 00:00:00 [nagios] <defunct>
0 Z <user> 9113 9109 0 80 0 - 0 exit Jul27 ? 00:00:00 [nagios] <defunct>
1 S <user> 9114 9109 0 80 0 - 3438 poll_s Jul27 ? 00:00:09 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
0 R <user> 9460 1 1 80 0 - 5181 - 12:58 ? 00:00:11 ../bin/nagios -u ../etc/nagios.cfg
0 S <user> 9503 9460 0 80 0 - 2504 ep_pol 12:58 ? 00:00:04 /usr/local/nagios-4.1.1/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 9504 9460 0 80 0 - 2504 ep_pol 12:58 ? 00:00:04 /usr/local/nagios-4.1.1/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 9505 9460 0 80 0 - 2504 ep_pol 12:58 ? 00:00:03 /usr/local/nagios-4.1.1/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 9506 9460 0 80 0 - 2504 ep_pol 12:58 ? 00:00:04 /usr/local/nagios-4.1.1/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
1 S <user> 9778 9460 0 80 0 - 5023 poll_s 12:58 ? 00:00:00 ../bin/nagios -u ../etc/nagios.cfg
1 R <user> 11549 1 0 80 0 - 5186 ? 13:12 ? 00:00:02 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
0 R <user> 11564 11549 0 80 0 - 2504 - 13:12 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 R <user> 11565 11549 0 80 0 - 2504 - 13:12 ? 00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 11566 11549 0 80 0 - 2504 ep_pol 13:12 ? 00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 R <user> 11567 11549 0 80 0 - 2504 - 13:12 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 11767 6967 0 80 0 - 25828 pipe_w 13:16 pts/15 00:00:00 grep nagios
1 S <user> 12441 11549 0 80 0 - 5057 poll_s 13:12 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
0 S <user> 12782 8389 0 80 0 - 2504 ep_pol Jul27 ? 00:07:55 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 12783 8389 0 80 0 - 2504 ep_pol Jul27 ? 00:08:02 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 12784 8389 0 80 0 - 2504 ep_pol Jul27 ? 00:07:54 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 12785 8389 0 80 0 - 2504 ep_pol Jul27 ? 00:07:52 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 R <user> 14695 9109 0 80 0 - 2504 - Jul27 ? 00:07:42 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 14696 9109 0 80 0 - 2504 ep_pol Jul27 ? 00:07:35 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 R <user> 14697 9109 0 80 0 - 2504 - Jul27 ? 00:07:44 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 14698 9109 0 80 0 - 2504 ep_pol Jul27 ? 00:07:46 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 15940 8470 0 80 0 - 2504 ep_pol Jul27 ? 00:07:54 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 R <user> 15941 8470 0 80 0 - 2504 - Jul27 ? 00:08:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 15942 8470 0 80 0 - 2504 ep_pol Jul27 ? 00:07:58 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 15943 8470 0 80 0 - 2504 ep_pol Jul27 ? 00:07:50 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 19780 5195 0 80 0 - 2504 ep_pol 08:59 ? 00:01:36 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 R <user> 19781 5195 0 80 0 - 2504 - 08:59 ? 00:01:33 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 S <user> 19782 5195 0 80 0 - 2504 ep_pol 08:59 ? 00:01:32 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 R <user> 19783 5195 0 80 0 - 2504 - 08:59 ? 00:01:32 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
0 Z <user> 26277 8729 0 80 0 - 0 exit 11:47 ? 00:00:02 [nagios] <defunct>
0 Z <user> 26278 8729 0 80 0 - 0 exit 11:47 ? 00:00:02 [nagios] <defunct>
0 Z <user> 26280 8729 0 80 0 - 0 exit 11:47 ? 00:00:02 [nagios] <defunct>
<user> 5195 1 0 Jul28 ? 00:34:10 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
<user> 5200 5195 0 Jul28 ? 00:00:51 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
<user> 8389 1 0 Jul27 ? 00:38:45 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
<user> 8391 8389 0 Jul27 ? 00:00:00 [nagios] <defunct>
<user> 8392 8389 0 Jul27 ? 00:00:00 [nagios] <defunct>
<user> 8393 8389 0 Jul27 ? 00:00:00 [nagios] <defunct>
<user> 8394 8389 0 Jul27 ? 00:01:08 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
<user> 8470 1 0 Jul27 ? 00:39:00 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
<user> 8471 8470 0 Jul27 ? 00:00:00 [nagios] <defunct>
<user> 8472 8470 0 Jul27 ? 00:00:00 [nagios] <defunct>
<user> 8473 8470 0 Jul27 ? 00:00:00 [nagios] <defunct>
<user> 8474 8470 0 Jul27 ? 00:00:00 [nagios] <defunct>
<user> 8475 8470 0 Jul27 ? 00:01:02 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
<user> 8727 8729 0 Jul28 ? 00:31:52 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 8728 8729 0 Jul28 ? 00:32:15 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 8729 1 0 Jul28 ? 00:32:52 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
<user> 8731 8729 0 Jul28 ? 00:32:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 8732 8729 0 Jul28 ? 00:32:13 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 8802 8729 0 Jul28 ? 00:00:42 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
<user> 9109 1 0 Jul27 ? 00:38:53 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
<user> 9110 9109 0 Jul27 ? 00:00:00 [nagios] <defunct>
<user> 9111 9109 0 Jul27 ? 00:00:00 [nagios] <defunct>
<user> 9112 9109 0 Jul27 ? 00:00:00 [nagios] <defunct>
<user> 9113 9109 0 Jul27 ? 00:00:00 [nagios] <defunct>
<user> 9114 9109 0 Jul27 ? 00:01:08 /usr/local/nagios/bin/nagios -u /usr/local/nagios/etc/nagios.cfg
<user> 9460 1 0 Jul28 ? 00:47:26 ../bin/nagios -u ../etc/nagios.cfg
<user> 9503 9460 0 Jul28 ? 00:22:23 /usr/local/nagios-4.1.1/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 9504 9460 0 Jul28 ? 00:22:59 /usr/local/nagios-4.1.1/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 9505 9460 0 Jul28 ? 00:22:10 /usr/local/nagios-4.1.1/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 9506 9460 0 Jul28 ? 00:22:09 /usr/local/nagios-4.1.1/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 9778 9460 0 Jul28 ? 00:00:49 ../bin/nagios -u ../etc/nagios.cfg
<user> 11549 1 0 Jul28 ? 00:48:48 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
<user> 11564 11549 0 Jul28 ? 00:22:23 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 11565 11549 0 Jul28 ? 00:21:59 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 11566 11549 0 Jul28 ? 00:22:10 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 11567 11549 0 Jul28 ? 00:22:10 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 12441 11549 0 Jul28 ? 00:00:52 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
<user> 12782 8389 0 Jul27 ? 00:41:37 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 12783 8389 0 Jul27 ? 00:41:59 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 12784 8389 0 Jul27 ? 00:41:32 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 12785 8389 0 Jul27 ? 00:41:56 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 14695 9109 0 Jul27 ? 00:41:16 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 14696 9109 0 Jul27 ? 00:40:46 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 14697 9109 0 Jul27 ? 00:41:29 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 14698 9109 0 Jul27 ? 00:41:34 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 15940 8470 0 Jul27 ? 00:41:35 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 15941 8470 0 Jul27 ? 00:41:41 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 15942 8470 0 Jul27 ? 00:41:28 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 15943 8470 0 Jul27 ? 00:41:20 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 19780 5195 0 Jul28 ? 00:33:17 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 19781 5195 0 Jul28 ? 00:32:10 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 19782 5195 0 Jul28 ? 00:32:23 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 19783 5195 0 Jul28 ? 00:32:45 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 21725 3484 0 05:11 pts/10 00:00:00 grep nagios
<user> 26277 8729 0 Jul28 ? 00:00:02 [nagios] <defunct>
<user> 26278 8729 0 Jul28 ? 00:00:02 [nagios] <defunct>
<user> 26280 8729 0 Jul28 ? 00:00:02 [nagios] <defunct>

[sami21]

Hi,
Seems like problem is resolved.
I stopped nagios and then killed all the extra processes displayed by ps -elf | grep nagios.
Now there are two nagios -d processes running. and 4 worker processes are running.
following is the output.

<user> 12539 1 2 07:28 ? 00:02:29 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
<user> 12550 12539 0 07:28 ? 00:00:02 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
<user> 17887 12539 0 09:13 ? 00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 17888 12539 0 09:13 ? 00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 17889 12539 0 09:13 ? 00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 17890 12539 0 09:13 ? 00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
<user> 25115 30451 0 09:18 pts/13 00:00:00 grep nagios
is this okay? or still it is problematic??
But couldn't understand the reason why nagios was creating multiple -d processes.

[bwallace]

Yes multiple Nagios processes running will definitely cause a multitude of problems. The Nagios process forks itself to run checks therefore subprocesses are normal, as you now see:

Code: Select all

<user> 12539 1 2 07:28 ? 00:02:29 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
<user> 12550 12539 0 07:28 ? 00:00:02 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

-- this is normal (one parent and one child). But multiple parent processes as you saw earlier is most certainly a problem and a number of things could cause such.

[lmiltchev]

is this okay? or still it is problematic??

This output looks fine - there are no multiple instances of nagios running as there is only one "main" process with PID=12539 and the rest of them are "child" processes that spun from this "parent" (PPID=12539). The bottom line is for the grep command, which is irrelevant.

But couldn't understand the reason why nagios was creating multiple -d processes.

Probably nagios was not stopped "cleanly". Perhaps you have a check that takes a very long time to execute. It's hard to say after the fact.
Let us know if you have any more issues/questions. Is it all right if we lock this topic, and mark it as "resolved"?

[rkennedy]

It looks like you had a handful of processes running before. What OS are you running? It looks like the process isn't properly restarting, and you're then being left with old processes running.

[sami21]

Hi All,

Thanks for you kind replies.
you can mark it as resolved topic.
Thanks once again.