Nagios XI 5.4.4 Appluconfiguration taking forever

[emartine]

I see this in the log
tail -F /usr/local/nagiosxi/var/cmdsubsys.log
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...

PROCESSED 0 COMMANDS
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...

PROCESSED 0 COMMANDS
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...
.Another reconfigure process is still running, sleeping...

This goes on for a good 10 minute and eventually the configuration applies.

............................................................
PROCESSED 0 COMMANDS
.......................................................PROCESSING COMMAND ID 26361...
PROCESS COMMAND: CMD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
URL: https://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2017-08-25 13:09:57-- https://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:443... connected.
WARNING: cannot verify localhost’s certificate, issued by “/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA”:
Unable to locally verify the issuer’s authority.
WARNING: no certificate subject alternative name matches
requested host name “localhost”.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.login”

0K .......... .......... .......... ........ 1.25M=0.03s

2017-08-25 13:09:57 (1.25 MB/s) - “nagiosql.login” saved [39040]

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies https://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=t<removed for security>LOGIN SUCCESSFUL!
IMPORTING CONFIG FILES...URL: https://localhost/nagiosxi/includes/components/ccm/
Array
(
)

How can I make this apply quicker?

[dwhitfield]

To start off, edit /etc/php.ini and increase these values:
max_execution_time = 60
max_input_time = 60
memory_limit = 256


After making these changes you'll need to restart the httpd service: service httpd restart

If that does not resolve the issue, please attach your /etc/php.ini and PM me a profile. You can download the profile by going to Admin > System Config > System Profile and click the ***Download Profile*** button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+, Show Profile if older) in the thread (that will at least get us some info). This will give us access to many of the logs we would otherwise ask for individually. If security is a concern, you can unzip the profile take out what you like, and then zip it up again. We may end up needing something you remove, but we can ask for that specifically.

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

You may also want to take a look at https://assets.nagios.com/downloads/nag ... ios-XI.pdf

UPDATE: profile shared with techs

[emartine]

Most of those settings were already in place. I sent you the profile via PM.

I was just told that over the weekend at around 5AM on Saturday there were a few hundred alerts sent out regarding servers going down.

All them were flapping notifications with the message:
Info: (host check orphaned, is the mod-gearman worker on queue host running?)

This is the second time this has happened. How can I keep this from happening again?

[dwhitfield]

This is probably unrelated to your original question, but what is the output of gearman_top? It seems like there is some sort of problem with mod-gearman.

[emartine]

gearman had to be restarted when that happened. Some issues have been occurring where contacts that were fine before are being reported as missing notification commands but they are not missing anything when we reviewed them through the interface. We had also received errors regarding a host being in an escalation group but it wasn't. Modifications were not being made when we were reviewing the items.

[emartine]

BTW gearman is still supported with 5.4.4 ?

[scottwilkerson]

BTW gearman is still supported with 5.4.4 ?

Yes, as long as it was compiled with the nagios4 flag

[emartine]

Any idea what might be causing gearman to do that?

[tacolover101]

what version of gearman are you running?

[tgriep]

To troubleshoot the Gearman messages, you would have to look in the gearman server log files and the remote worker files to see what happened.

You said that the settings in the /etc/php.ini were already increased, increase them again, by doubling them.

Code: Select all

max_execution_time = 60
max_input_time = 60
memory_limit = 256M

Add this line to the php.ini file.

Code: Select all

max_input_vars=5000

Then restart Apache by running

Code: Select all

service httpd restart

Do this change to server as well to give it more time to apply the config before restarting the nagios process.
https://support.nagios.com/kb/article/n ... anner.html