Apply Configuration never completes

[mrochelle]

YOU THE MAN!!! Thanks Scott, we are back in business!!!
There was some database corruption. After, that, I also restarted Nagios and the heavens opened up, light, angels, etc...

[mrochelle]

I may have started cellebrating a little too soon. All group functionality is back, Google Maps fuctionality, but the write config data failed as before. Let me do some more checks and follow back up.

[mrochelle]

Ok, we are still getting the following error when trying to apply configuration:

--2012-09-26 18:24:00-- (try: 2) http://localhost/nagiosql/admin/verify.php
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response...

[mrochelle]

Ok, let me provide some more information. The "Apply Configuration" never seems to complete. Even with an update to a single host. However, I can manually apply the changes using the buttons under the Write Config Files. It takes 30 or 40 minutes for the write monitoring data. Could we be running into some upper thresholds with the size of our configuration?

[lmiltchev]

I wonder of you are exceeding the memory limit... In the php.ini you have:

Code: Select all

memory_limit = -1

How much physical memory do you have on this box?

[mrochelle]

I had changed it to 1000M prior to the last set of testing. The box has a total of 16GB and the most I've seen used is 10GB from the server statistics.

[scottwilkerson]

We actually may still have some database issues, lets run the following to see if there are any errors showing up

Code: Select all

tail -f /var/log/mysql.log

Additionally, lets return the output of the following

Code: Select all

df -h

[mrochelle]

tail -f /var/log/mysqld.log

120925 21:51:31 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
120925 21:52:25 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
120925 21:52:25 InnoDB: Initializing buffer pool, size = 8.0M
120925 21:52:25 InnoDB: Completed initialization of buffer pool
120925 21:52:26 InnoDB: Started; log sequence number 0 44233
120925 21:52:26 [Note] Event Scheduler: Loaded 0 events
120925 21:52:26 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
120926 17:40:24 [Note] /usr/libexec/mysqld: Normal shutdown
120926 17:40:24 [Note] Event Scheduler: Purging the queue. 0 events
120926 17:40:26 InnoDB: Starting shutdown...
120926 17:40:29 InnoDB: Shutdown completed; log sequence number 0 44233
120926 17:40:29 [Note] /usr/libexec/mysqld: Shutdown complete

120926 17:40:29 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
120926 17:40:59 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
120926 17:40:59 InnoDB: Initializing buffer pool, size = 8.0M
120926 17:40:59 InnoDB: Completed initialization of buffer pool
120926 17:40:59 InnoDB: Started; log sequence number 0 44233
120926 17:40:59 [Note] Event Scheduler: Loaded 0 events
120926 17:40:59 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

df -h

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
1008G 14G 944G 2% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 97M 43M 50M 47% /boot

[mguthrie]

Can you run the following from the shell and see where the delay seems to occur?

Code: Select all

cd /usr/local/nagiosxi/scripts/
./reconfigure_nagios.sh

This runs the actual Apply Configuration Process from the command line with logging output.

[mrochelle]

The CMD Line results are attached.