"Database Error" after upgrade.

[lee.krause]

After the upgrade I got the following error after login:

Database Error
A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.
Run the following from the CLI as root to attempt to repair the DB:

/usr/local/nagiosxi/scripts/repair_databases.sh

I ran the script suggested and got the following(after the repairs), but still get the same error:

=======================
nagios database repair succeeded
nagiosql database repair succeeded
nagiosxi database repair succeeded

[benjaminsmith]

Hi Lee,

You might be running out of disk space. What is the output of the following commands?

Code: Select all

df -h
df -i
service mysqld restart
service mysqld status
tail -n100 /var/log/mariadb/mariadb.log /var/log/mysqld.log

If that's not the case, please send us your system profile for us to review.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message and then reply to this post to bring it up in the queue.

[lee.krause]

Code: Select all

root@hostname ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_erpgrctemp-lv_root
                       57G   43G   12G  80% /
tmpfs                 2.9G   12K  2.9G   1% /dev/shm
/dev/sda1             477M   85M  367M  19% /boot
[root@hostname ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_erpgrctemp-lv_root
                       57G   43G   12G  80% /
tmpfs                 2.9G   12K  2.9G   1% /dev/shm
/dev/sda1             477M   85M  367M  19% /boot
[root@hostname ~]# df -i
Filesystem            Inodes  IUsed   IFree IUse% Mounted on
/dev/mapper/vg_erpgrctemp-lv_root
                     3778816 138062 3640754    4% /
tmpfs                 749091      4  749087    1% /dev/shm
/dev/sda1             128016     50  127966    1% /boot
[root@hostname ~]# service mysqld restart
Stopping mysqld:                                           [  OK  ]
Starting mysqld:                                           [  OK  ]
[root@hostname ~]# service mysqld status
mysqld (pid  21679) is running...
[root@hostname ~]# tail -n100 /var/log/mariadb/mariadb.log /var/log/mysqld.log
tail: cannot open `/var/log/mariadb/mariadb.log' for reading: No such file or directory
==> /var/log/mysqld.log <==
190918 10:04:12 [Note] /usr/libexec/mysqld: Shutdown complete

190918 10:04:12 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
190918 10:07:01 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190918 10:07:07  InnoDB: Initializing buffer pool, size = 8.0M
190918 10:07:07  InnoDB: Completed initialization of buffer pool
190918 10:07:08  InnoDB: Started; log sequence number 1 4286813225
190918 10:07:08 [Note] Event Scheduler: Loaded 0 events
190918 10:07:08 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
190918 10:07:09 [Note] /usr/libexec/mysqld: Normal shutdown

190918 10:07:10 [Note] Event Scheduler: Purging the queue. 0 events
190918 10:07:12  InnoDB: Starting shutdown...
190918 10:07:13  InnoDB: Shutdown completed; log sequence number 1 4286813225
190918 10:07:14 [Note] /usr/libexec/mysqld: Shutdown complete

190918 10:07:14 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
190918 10:11:37 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190918 10:11:40  InnoDB: Initializing buffer pool, size = 8.0M
190918 10:11:40  InnoDB: Completed initialization of buffer pool
190918 10:11:41  InnoDB: Started; log sequence number 1 4286813225
190918 10:11:41 [Note] Event Scheduler: Loaded 0 events
190918 10:11:41 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
190918 10:11:42 [Note] /usr/libexec/mysqld: Normal shutdown

190918 10:11:43 [Note] Event Scheduler: Purging the queue. 0 events
190918 10:11:45  InnoDB: Starting shutdown...
190918 10:11:46  InnoDB: Shutdown completed; log sequence number 1 4286813225
190918 10:11:47 [Note] /usr/libexec/mysqld: Shutdown complete

190918 10:11:47 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
190918 10:12:13 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190918 10:12:18  InnoDB: Initializing buffer pool, size = 8.0M
190918 10:12:18  InnoDB: Completed initialization of buffer pool
190918 10:12:18  InnoDB: Started; log sequence number 1 4286813225
190918 10:12:19 [Note] Event Scheduler: Loaded 0 events
190918 10:12:19 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
190918 14:08:19 [Note] /usr/libexec/mysqld: Normal shutdown

190918 14:08:19 [Note] Event Scheduler: Purging the queue. 0 events
190918 14:08:19  InnoDB: Starting shutdown...
190918 14:08:22  InnoDB: Shutdown completed; log sequence number 1 4287157085
190918 14:08:22 [Note] /usr/libexec/mysqld: Shutdown complete

190918 14:08:22 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
190918 14:09:52 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190918 14:09:52  InnoDB: Initializing buffer pool, size = 8.0M
190918 14:09:52  InnoDB: Completed initialization of buffer pool
190918 14:09:52  InnoDB: Started; log sequence number 1 4287157085
190918 14:09:52 [Note] Event Scheduler: Loaded 0 events
190918 14:09:52 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
190918 14:09:54 [Note] /usr/libexec/mysqld: Normal shutdown

190918 14:09:54 [Note] Event Scheduler: Purging the queue. 0 events
190918 14:09:54  InnoDB: Starting shutdown...
190918 14:09:57  InnoDB: Shutdown completed; log sequence number 1 4287157085
190918 14:09:57 [Note] /usr/libexec/mysqld: Shutdown complete

190918 14:09:57 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
190918 14:10:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190918 14:10:00  InnoDB: Initializing buffer pool, size = 8.0M
190918 14:10:00  InnoDB: Completed initialization of buffer pool
190918 14:10:01  InnoDB: Started; log sequence number 1 4287157085
190918 14:10:01 [Note] Event Scheduler: Loaded 0 events
190918 14:10:01 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
190918 14:10:04 [Note] /usr/libexec/mysqld: Normal shutdown

190918 14:10:04 [Note] Event Scheduler: Purging the queue. 0 events
190918 14:10:06  InnoDB: Starting shutdown...
190918 14:10:11  InnoDB: Shutdown completed; log sequence number 1 4287157095
190918 14:10:11 [Note] /usr/libexec/mysqld: Shutdown complete

190918 14:10:11 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
190918 14:10:14 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190918 14:10:14  InnoDB: Initializing buffer pool, size = 8.0M
190918 14:10:14  InnoDB: Completed initialization of buffer pool
190918 14:10:15  InnoDB: Started; log sequence number 1 4287157095
190918 14:10:15 [Note] Event Scheduler: Loaded 0 events
190918 14:10:15 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
190918 16:30:01 [Note] /usr/libexec/mysqld: Normal shutdown

190918 16:30:01 [Note] Event Scheduler: Purging the queue. 0 events
190918 16:30:01  InnoDB: Starting shutdown...
190918 16:30:03  InnoDB: Shutdown completed; log sequence number 1 4287745282
190918 16:30:03 [Note] /usr/libexec/mysqld: Shutdown complete

190918 16:30:03 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
190918 16:30:04 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190918 16:30:04  InnoDB: Initializing buffer pool, size = 8.0M
190918 16:30:04  InnoDB: Completed initialization of buffer pool
190918 16:30:04  InnoDB: Started; log sequence number 1 4287745282
190918 16:30:04 [Note] Event Scheduler: Loaded 0 events
190918 16:30:04 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

I can't get to the GUI, when to the login screen and put in my credentials, the error pops up.

[benjaminsmith]

Hi Lee,

Since the system is down and you're unable to access the GUI, please open support ticket on this issue for faster resolution.

https://support.nagios.com/tickets/

Also, if you can run a system profile from the terminal and post that to the ticket, it would help speed up the process. Thanks.

Code: Select all

rm -rf /usr/local/nagiosxi/var/components/profile*
/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh SUPPORT