Page 2 of 3

Re: Apply Configuration never completes

Posted: Wed Sep 26, 2012 6:04 pm
by mrochelle
YOU THE MAN!!! Thanks Scott, we are back in business!!!
There was some database corruption. After, that, I also restarted Nagios and the heavens opened up, light, angels, etc...

Re: Apply Configuration never completes

Posted: Wed Sep 26, 2012 6:13 pm
by mrochelle
I may have started cellebrating a little too soon. All group functionality is back, Google Maps fuctionality, but the write config data failed as before. Let me do some more checks and follow back up.

Re: Apply Configuration never completes

Posted: Wed Sep 26, 2012 6:38 pm
by mrochelle
Ok, we are still getting the following error when trying to apply configuration:

--2012-09-26 18:24:00-- (try: 2) http://localhost/nagiosql/admin/verify.php
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response...

Re: Apply Configuration never completes

Posted: Thu Sep 27, 2012 9:17 am
by mrochelle
Ok, let me provide some more information. The "Apply Configuration" never seems to complete. Even with an update to a single host. However, I can manually apply the changes using the buttons under the Write Config Files. It takes 30 or 40 minutes for the write monitoring data. Could we be running into some upper thresholds with the size of our configuration?

Re: Apply Configuration never completes

Posted: Thu Sep 27, 2012 2:38 pm
by lmiltchev
I wonder of you are exceeding the memory limit... In the php.ini you have:

Code: Select all

memory_limit = -1
How much physical memory do you have on this box?

Re: Apply Configuration never completes

Posted: Thu Sep 27, 2012 4:20 pm
by mrochelle
I had changed it to 1000M prior to the last set of testing. The box has a total of 16GB and the most I've seen used is 10GB from the server statistics.

Re: Apply Configuration never completes

Posted: Thu Sep 27, 2012 5:05 pm
by scottwilkerson
We actually may still have some database issues, lets run the following to see if there are any errors showing up

Code: Select all

tail -f /var/log/mysql.log
Additionally, lets return the output of the following

Code: Select all

df -h

Re: Apply Configuration never completes

Posted: Thu Sep 27, 2012 7:13 pm
by mrochelle
tail -f /var/log/mysqld.log

120925 21:51:31 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
120925 21:52:25 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
120925 21:52:25 InnoDB: Initializing buffer pool, size = 8.0M
120925 21:52:25 InnoDB: Completed initialization of buffer pool
120925 21:52:26 InnoDB: Started; log sequence number 0 44233
120925 21:52:26 [Note] Event Scheduler: Loaded 0 events
120925 21:52:26 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
120926 17:40:24 [Note] /usr/libexec/mysqld: Normal shutdown
120926 17:40:24 [Note] Event Scheduler: Purging the queue. 0 events
120926 17:40:26 InnoDB: Starting shutdown...
120926 17:40:29 InnoDB: Shutdown completed; log sequence number 0 44233
120926 17:40:29 [Note] /usr/libexec/mysqld: Shutdown complete

120926 17:40:29 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
120926 17:40:59 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
120926 17:40:59 InnoDB: Initializing buffer pool, size = 8.0M
120926 17:40:59 InnoDB: Completed initialization of buffer pool
120926 17:40:59 InnoDB: Started; log sequence number 0 44233
120926 17:40:59 [Note] Event Scheduler: Loaded 0 events
120926 17:40:59 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

df -h

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
1008G 14G 944G 2% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 97M 43M 50M 47% /boot

Re: Apply Configuration never completes

Posted: Fri Sep 28, 2012 10:41 am
by mguthrie
Can you run the following from the shell and see where the delay seems to occur?

Code: Select all

cd /usr/local/nagiosxi/scripts/
./reconfigure_nagios.sh
This runs the actual Apply Configuration Process from the command line with logging output.

Re: Apply Configuration never completes

Posted: Fri Sep 28, 2012 12:53 pm
by mrochelle
The CMD Line results are attached.