Unable to login to nagiosxi after /var filesystem filled up

[skynardo]

Im running Nagios XI 2014 R2.5. When trying to login to the nagiosxi console, getting message to run repair_databases.sh script. I have a remote database server and when checking found /var filesystems were full on both my Nagios server and my remote database server. After expanding those and trying to run repair, was getting errors that it couldn't write to /usr/local/nagios/var/spool/checkresults, no space left, but when checking /usr filesystem has over 4Gb free space. There were thousands of c* c*.ok files in this directory. After removing some of them, I was able to get repair_databases script to run without errors, but I am still unable to login to nagios xi. It appears checks are being processed as I see them in /var/log/messages.

[ssax]

What error are you getting when you try to log in?

Have you restarted the services after free up the space?

Code: Select all

service nagios stop
service npcd stop
service ndo2db stop
service mysqld restart
service ndo2db start
service npcd start
service nagios start

Are you seeing anything in your /var/log/mysqld.log or /var/lib/pgsql/data/pg_log/postgresql-Mon.log?

[skynardo]

I had previously restarted nagios and remote mysqld services, but am not familiar with the other's you listed. I restarted all services in the order you provided but I am still unable to get the nagiosxi login screen. Insteady I get:

Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB

/usr/local/nagiosxi/scripts/repair_databases.sh

The above script starts mysqld on the nagiosxi server rather than my remote mysql database server. The remote mysql log looks OK:
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150518 10:55:04 [Note] /usr/libexec/mysqld: Normal shutdown

150518 10:55:04 [Note] Event Scheduler: Purging the queue. 0 events
150518 10:55:04 InnoDB: Starting shutdown...
150518 10:55:08 InnoDB: Shutdown completed; log sequence number 0 44233
150518 10:55:08 [Note] /usr/libexec/mysqld: Shutdown complete

150518 10:55:08 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150518 10:55:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150518 10:55:08 InnoDB: Initializing buffer pool, size = 8.0M
150518 10:55:08 InnoDB: Completed initialization of buffer pool
150518 10:55:08 InnoDB: Started; log sequence number 0 44233
150518 10:55:08 [Note] Event Scheduler: Loaded 0 events
150518 10:55:08 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

[skynardo]

After checking the postgres log, found postgres db was down. I restarted it and now I am able to login.