NagiosXI database error after repair

[syncbak]

I recently had to repair the Nagios database because of a full disk.
The repair seemed to complete successfully, but I am still unable to login to the /nagiosxi page, although the /nagios page is fine now.

When i hit the /nagiosxi page, Error message is this,
Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh

I have tried the above command with no luck.
I see other posts about this, have followed advice on the threads but cannot resolve.
Now I am following he 'please contact nagios suppor't directive in the error message.

Can anyone advise?

[lmiltchev]

What is the error that you see when you try to log in the Nagios XI web UI? Were all of the mysql errors fixed? Did you add more disk space? What is the output of the following commands?

Code: Select all

service mysqld status
service mysqld restart
tail -50 /var/log/mysqld.log
df -h
df -i

[syncbak]

The error from the Web is what was posted:

Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.
Run the following from the CLI as root to attempt to repair the DB
/usr/local/nagiosxi/scripts/repair_databases.sh

The tables appeared to be fixed, I got no error messages when running the repair script.
More disk space was added by removing old backup/log files.

Output of the commands are below:

service mysqld status
mysqld (pid 23197) is running...

service mysqld restart
Stopping mysqld: [ OK ]
Starting mysqld [ OK ]

tail -50 /var/log/mysqld.log

141124 10:14:03 [Note] Event Scheduler: Purging the queue. 0 events
141124 10:14:05 InnoDB: Starting shutdown...
141124 10:14:11 InnoDB: Shutdown completed; log sequence number 0 44233
141124 10:14:11 [Note] /usr/libexec/mysqld: Shutdown complete

141124 10:14:11 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
141124 10:16:06 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
141124 10:16:06 [Warning] /usr/libexec/mysqld: Option '--set-variable' is deprecated. Use --variable-name=value instead.
141124 10:16:06 [Warning] /usr/libexec/mysqld: Option '--set-variable' is deprecated. Use --variable-name=value instead.
141124 10:16:06 InnoDB: Initializing buffer pool, size = 8.0G
141124 10:16:07 InnoDB: Completed initialization of buffer pool
141124 10:16:07 InnoDB: Started; log sequence number 0 44233
141124 10:16:07 [Note] Event Scheduler: Loaded 0 events
141124 10:16:07 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
141124 10:16:44 [Note] /usr/libexec/mysqld: Normal shutdown

141124 10:16:44 [Note] Event Scheduler: Purging the queue. 0 events
141124 10:16:46 InnoDB: Starting shutdown...
141124 10:16:48 InnoDB: Shutdown completed; log sequence number 0 44233
141124 10:16:48 [Note] /usr/libexec/mysqld: Shutdown complete

141124 10:16:48 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
141124 10:35:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
141124 10:35:45 [Warning] /usr/libexec/mysqld: Option '--set-variable' is deprecated. Use --variable-name=value instead.
141124 10:35:45 [Warning] /usr/libexec/mysqld: Option '--set-variable' is deprecated. Use --variable-name=value instead.
141124 10:35:45 InnoDB: Initializing buffer pool, size = 8.0G
141124 10:35:46 InnoDB: Completed initialization of buffer pool
141124 10:35:49 InnoDB: Started; log sequence number 0 44233
141124 10:35:49 [Note] Event Scheduler: Loaded 0 events
141124 10:35:49 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
141124 11:36:25 [Note] /usr/libexec/mysqld: Normal shutdown

141124 11:36:25 [Note] Event Scheduler: Purging the queue. 0 events
141124 11:36:27 InnoDB: Starting shutdown...
141124 11:36:29 InnoDB: Shutdown completed; log sequence number 0 44233
141124 11:36:29 [Note] /usr/libexec/mysqld: Shutdown complete

141124 11:36:29 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
141124 11:36:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
141124 11:36:30 [Warning] /usr/libexec/mysqld: Option '--set-variable' is deprecated. Use --variable-name=value instead.
141124 11:36:30 [Warning] /usr/libexec/mysqld: Option '--set-variable' is deprecated. Use --variable-name=value instead.
141124 11:36:30 InnoDB: Initializing buffer pool, size = 8.0G
141124 11:36:31 InnoDB: Completed initialization of buffer pool
141124 11:36:31 InnoDB: Started; log sequence number 0 44233
141124 11:36:31 [Note] Event Scheduler: Loaded 0 events
141124 11:36:31 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 50G 18G 30G 38% /
tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 99M 57M 37M 61% /boot
tmpfs 50M 4.6M 46M 10% /var/nagiosramdisk
/dev/mapper/largeChunk-nagios--lv
99G 21G 73G 23% /usr/local/nagios
/dev/mapper/largeChunk-mysql--lv
394G 2.7G 372G 1% /var/lib/mysql


# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 3276800 94244 3182556 3% /
tmpfs 4105696 1 4105695 1% /dev/shm
/dev/sda1 26208 47 26161 1% /boot
tmpfs 4105696 8 4105688 1% /var/nagiosramdisk
/dev/mapper/largeChunk-nagios--lv
6553600 20328 6533272 1% /usr/local/nagios
/dev/mapper/largeChunk-mysql--lv
26214400 730 26213670 1% /var/lib/mysql

[sreinhardt]

Did you happen to change the various mysql passwords lately? Some files to make sure that all passwords match:

/usr/local/nagiosxi/html/config.inc.php - postgres and nagios\nagiosql mysql users
/usr/local/nagios/etc/ndo2db.cfg - nagios mysql user
/usr/local/nagiosxi/var/xi-sys.cfg - most of these same passwords, in a single location for upgrades.

I would suggest making sure you can login with the three db accounts as well

Code: Select all

psql -d nagiosxi -U nagiosxi -W
mysql -u nagios -p[nagiospass] -D nagios
mysql -u nagiosql -p[nagiosqlpass] -D nagiosql

[syncbak]

I had not changes the passwords, and was not able to login to postgresql. as suggested in the response.

After restarting postgresql (/etc/init.d/postgresql restart), I was able to reach the /nagiosxi page, and am able to login now.

I do not recall having to mess with postgresql while repairing the databases after the disk got full, but it must have hung it

The machine is now authenticating as it should and I can see the NagiosXI pages.




I

[tmcdonald]

Let's keep this topic open for a bit in case you run into more related DB issues. Otherwise we'll close it out if things look good.