Waiting for configuration verification.... spins forever

[timlwhite]

Hello!

64-Bit CentOS 6.4, Manual Install of 2012R2.3.

The fullinstall command completed successfully, however, when I add a host and do "Apply Configuration", the "Waiting for configuration verification...." line spins forever.

This is a fresh install, there is nothing other than the install defaults in the config - the only host listed is 'localhost' until I add one and try to Apply.

Running /nagiosql/admin/verify.php directly, I instantly see the output, and see any errors.

In addition, I see the errors in the "Configuration Snapshots" area, but when I try to delete the configs with errors, it says "Config Snapshot Deleted", but it still appears in both the list and on the filesystem - whereas on my other installs it does in fact delete it.

I've fun 'fix-nagiosadmin' from the install dir, and I see that I have a valid config, but I still can't apply it from the Core Config Manager.

It feels like there is a permissions problem somewhere.

I looked in /usr/local/nagiosxi/nom/checkpoints/nagioscore, and I can see the list of config snapshots and errors:

Code: Select all

664226 drwxr-xr-x 2 nagios nagios  4096 Sep  9 09:06 .
664225 drwxr-xr-x 3 nagios nagios  4096 Sep  9 09:08 ..
664167 -rw-r--r-- 1 nagios nagios 28399 Sep  9 08:58 1378738715.tar.gz
664166 -rw-r--r-- 1 nagios nagios  2799 Sep  9 08:58 1378738715.txt
664172 -rw-r--r-- 1 nagios nagios 28437 Sep  9 09:06 1378739165.tar.gz
664171 -rw-r--r-- 1 nagios nagios  2799 Sep  9 09:06 1378739165.txt

I ran 'check file permissions' from the Core Config Manager, and it shows green.

Any idea what I can check next for this one?

Thanks!

Here's the Install Profile Output:

Code: Select all

Nagios XI Installation Profile

Download Profile
System:

Nagios XI Version : 2012R2.3
ecca2 2.6.32-358.14.1.el6.x86_64 x86_64
CentOS release 6.4 (Final)
Gnome Installed
Apache Information

PHP Version: 5.3.3
Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36
Server Name: <redacted>
Server Address: <redacted>
Server Port: 80
Date/Time

PHP Timezone: America/Denver 
PHP Time: Mon, 09 Sep 2013 09:24:53 -0600
System Time: Mon, 09 Sep 2013 09:24:53 -0600
Nagios XI Data

nagios (pid 32440) is running...
NPCD running.
ndo2db (pid 15293) is running...
CPU Load 15: 0.81 
Total Hosts: 1 
Total Services: 8 
Function 'get_base_uri' returns: http://<redacted>/nagiosxi/
Function 'get_base_url' returns: http://<redacted>/nagiosxi/
Function 'get_backend_url(internal_call=false)' returns: http://<redacted>/nagiosxi/includes/components/profile/profile.php
Function 'get_backend_url(internal_call=true)' returns: http://localhost/nagiosxi/backend/
Ping Test localhost

Running:
/bin/ping -c 3 localhost 2>&1 
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.035 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.026 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.025 ms

--- localhost ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.025/0.028/0.035/0.007 ms
Test wget To locahost

WGET From URL: http://localhost/nagiosql/index.php 
Running:
/usr/bin/wget http://localhost/nagiosql/index.php 
--2013-09-09 09:24:55-- http://localhost/nagiosql/index.php
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5259 (5.1K) [text/html]
Saving to: `/tmp/nagiosql_index.tmp'

0K ..... 100% 368M=0s

2013-09-09 09:24:55 (368 MB/s) - `/tmp/nagiosql_index.tmp' saved [5259/5259]

[slansing]

Hmm, you may be running into some database issues here. Did you offload the mysql database to a remote server?

Please run through this document and then try to apply configuration again:

http://assets.nagios.com/downloads/nagi ... tabase.pdf

If it still hangs, run the following:

Code: Select all

tail -f /var/log/httpd/error_log

Then try to apply configuration, after it hangs, break the tail and send us the output here in addition to the output of the following command:

Code: Select all

tail -30 /var/log/mysqld.log

Thanks!

[timlwhite]

Ok - it looks like the error from the httpd error_log is:

Code: Select all

[Mon Sep 09 10:12:53 2013] [error] [client 155.70.39.45] PHP Warning:  mysql_pconnect(): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383, referer: http://<redacted ip>/nagiosxi//includes/components/nagioscorecfg/applyconfig.php?cmd=confirm

Any thoughts on where to start troubleshooting that socket connection?

Database is local on machine, no customizations...

I ran through all these fixes, to no avail (including just making sure it was running):

http://stackoverflow.com/questions/5376 ... ql-sock-38

Output of /var/log/mysqld.log:

Code: Select all

130909 10:29:02  InnoDB: Started; log sequence number 0 44243
130909 10:29:02 [Note] Event Scheduler: Loaded 0 events
130909 10:29:02 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.69'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
130909 10:31:36 [Note] /usr/libexec/mysqld: Normal shutdown

130909 10:31:36 [Note] Event Scheduler: Purging the queue. 0 events
130909 10:31:36  InnoDB: Starting shutdown...
130909 10:31:37  InnoDB: Shutdown completed; log sequence number 0 44243
130909 10:31:37 [Note] /usr/libexec/mysqld: Shutdown complete

130909 10:31:37 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130909 10:31:38 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130909 10:31:38  InnoDB: Initializing buffer pool, size = 8.0M
130909 10:31:38  InnoDB: Completed initialization of buffer pool
130909 10:31:38  InnoDB: Started; log sequence number 0 44243
130909 10:31:38 [Note] Event Scheduler: Loaded 0 events
130909 10:31:38 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.69'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

Thanks!

[abrist]

Are the disks full?

Code: Select all

df -i
df -h

What are you php.ini settings for memory, execution, etc?

Code: Select all

grep "memory_limit\|max_execution\|max_input" /etc/php.ini

[timlwhite]

Disks & PHP look ok:

Code: Select all

[root@ecca2 ~]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg_ecca2-lv_root
                     3276800  144002 3132798    5% /
tmpfs                 474103       4  474099    1% /dev/shm
/dev/sda1             128016      53  127963    1% /boot
/dev/mapper/vg_ecca2-lv_home
                     57499648    1383 57498265    1% /home

[root@ecca2 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_ecca2-lv_root
                       50G  4.6G   43G  10% /
tmpfs                 1.9G   12K  1.9G   1% /dev/shm
/dev/sda1             485M   93M  367M  21% /boot
/dev/mapper/vg_ecca2-lv_home
                      864G  339M  820G   1% /home

[root@ecca2 ~]# grep "memory_limit\|max_execution\|max_input" /etc/php.ini
; max_input_time
max_execution_time = 30
max_input_time = 60
;max_input_nesting_level = 64
memory_limit = 128M

[sreinhardt]

Well, just to be 100% sure, lets make sure you can access the socket file, and that mysql is started.

Code: Select all

service mysql status
ll /var/lib/mysql/mysql.sock

[timlwhite]

Ok - here's the output:

Code: Select all

[root@ecca2 etc]# service mysql status
mysql: unrecognized service

[root@ecca2 etc]# service mysqld status
mysqld (pid  10143) is running...

[root@ecca2 etc]# ll /var/lib/mysql/mysql.sock
srwxrwxrwx 1 mysql mysql 0 Sep  9 10:31 /var/lib/mysql/mysql.sock

[root@ecca2 etc]# mysql --version
mysql  Ver 14.14 Distrib 5.1.69, for redhat-linux-gnu (x86_64) using readline 5.1

[timlwhite]

Also:

Confirmed that /etc/sysconfig/selinux is set to disabled, and that php has --enable-sockets.

[lmiltchev]

Can you run the following command and show us the output?

Code: Select all

cat /etc/my.cnf

[timlwhite]

Code: Select all

[root@ecca2 network-scripts]# cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid