[SOLVED] Installation Error - Database problems - Small VM

[Cyanelix]

Hi,

I've attempted to run the fullinstall.sh script from v.5.4.3 downloaded yesterday (27th March) on a completely clean build of CentOS 7.3.1611 64 bit (on a DigitalOcean VM), and it failed with:

Code: Select all

RESETTING PERMS
/bin/chown: cannot access ‘/usr/local/nagiosxi/nom/checkpoints/nagiosxi’: No such file or directory
PHP Warning:  mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /tmp/nagiosxi/nagiosxi/basedir/html/db/adodb/drivers/adodb-mysqli.inc.php on line 117
PHP Warning:  mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /tmp/nagiosxi/nagiosxi/basedir/html/db/adodb/drivers/adodb-mysqli.inc.php on line 117
PHP Warning:  mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /tmp/nagiosxi/nagiosxi/basedir/html/db/adodb/drivers/adodb-mysqli.inc.php on line 117
PHP Warning:  mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /tmp/nagiosxi/nagiosxi/basedir/html/db/adodb/drivers/adodb-mysqli.inc.php on line 117
PHP Warning:  mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /tmp/nagiosxi/nagiosxi/basedir/html/db/adodb/drivers/adodb-mysqli.inc.php on line 117
PHP Warning:  mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /tmp/nagiosxi/nagiosxi/basedir/html/db/adodb/drivers/adodb-mysqli.inc.php on line 117
PHP Warning:  mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /tmp/nagiosxi/nagiosxi/basedir/html/db/adodb/drivers/adodb-mysqli.inc.php on line 117
PHP Warning:  mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /tmp/nagiosxi/nagiosxi/basedir/html/db/adodb/drivers/adodb-mysqli.inc.php on line 117
PHP Warning:  mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /tmp/nagiosxi/nagiosxi/basedir/html/db/adodb/drivers/adodb-mysqli.inc.php on line 117
PHP Warning:  mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /tmp/nagiosxi/nagiosxi/basedir/html/db/adodb/drivers/adodb-mysqli.inc.php on line 117
<h3>Databse Error</h3>A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.<p>Run the following from the CLI as root to attempt to repair the DB:<br><pre>/usr/local/nagiosxi/scripts/repair_databases.sh</pre></p>URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'backend=1&cmd=apply&type=writeConfig' -O nagiosql.export.monitoring
--2017-03-27 21:38:16--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 297 [text/html]
Saving to: ‘nagiosql.export.monitoring’

     0K                                                       100% 44.2M=0s

2017-03-27 21:38:36 (44.2 MB/s) - ‘nagiosql.export.monitoring’ saved [297/297]

NAGIOSQL WRITE CONFIGS FAILED!\n
RESULT=1

===================
INSTALLATION ERROR!
===================
Installation step failed - exiting.
Check for error messages in the install log (install.log).

If you require assistance in resolving the issue, please include install.log
in your communications with Nagios XI technical support.

The script that failed was: './E-importnagiosql'

I've attached the full install.log for reference.

As advised in the output, I ran the /usr/local/nagiosxi/scripts/repair_databases.sh script, which seemed to have a couple of errors:

Code: Select all

DATABASE: nagiosql
TABLE:
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2017-03-28 05:24:19 UTC; 176ms ago
  Process: 32373 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
  Process: 32372 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 32345 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
 Main PID: 32372 (code=exited, status=0/SUCCESS)

Mar 28 05:24:18 cyanelix-monitor systemd[1]: Starting MariaDB database server...
Mar 28 05:24:18 cyanelix-monitor mysqld_safe[32372]: 170328 05:24:18 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Mar 28 05:24:18 cyanelix-monitor mysqld_safe[32372]: 170328 05:24:18 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Mar 28 05:24:19 cyanelix-monitor systemd[1]: mariadb.service: control process exited, code=exited status=1
Mar 28 05:24:19 cyanelix-monitor systemd[1]: Failed to start MariaDB database server.
Mar 28 05:24:19 cyanelix-monitor systemd[1]: Unit mariadb.service entered failed state.
Mar 28 05:24:19 cyanelix-monitor systemd[1]: mariadb.service failed.

and

Code: Select all

DATABASE: nagiosxi
TABLE:    
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2017-03-28 05:24:21 UTC; 304ms ago
  Process: 32677 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
  Process: 32676 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 32649 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
 Main PID: 32676 (code=exited, status=0/SUCCESS)

Mar 28 05:24:20 cyanelix-monitor systemd[1]: Starting MariaDB database server...
Mar 28 05:24:20 cyanelix-monitor mysqld_safe[32676]: 170328 05:24:20 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Mar 28 05:24:20 cyanelix-monitor mysqld_safe[32676]: 170328 05:24:20 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Mar 28 05:24:21 cyanelix-monitor systemd[1]: mariadb.service: control process exited, code=exited status=1
Mar 28 05:24:21 cyanelix-monitor systemd[1]: Failed to start MariaDB database server.
Mar 28 05:24:21 cyanelix-monitor systemd[1]: Unit mariadb.service entered failed state.
Mar 28 05:24:21 cyanelix-monitor systemd[1]: mariadb.service failed.

and that script ended with:

Code: Select all

---------

- recovering (with sort) MyISAM-table 'xi_users.MYI'
Data records: 1
- Fixing index 1
- Fixing index 2
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
/tmp/nagiosxi
 
===============
REPAIR COMPLETE
===============
Job for nagios.service failed because the control process exited with error code. See "systemctl status nagios.service" and "journalctl -xe" for details.

=======================
nagios database repair succeeded
nagiosql database repair succeeded
nagiosxi database repair succeeded

Finally, I attempted to re-run fullinstall.sh, as recommended in a recent post here: https://support.nagios.com/forum/viewto ... 10#p215115. This time it failed with the following at the end of the output (also in the attached install.log):

Code: Select all

NOTE: If prompted, enter the MySQL root password
ERROR: Cannot connect to NagiosQL database - exiting.  Were MySQL and NagiosQL installed?
RESULT=1

===================
INSTALLATION ERROR!
===================
Installation step failed - exiting.
Check for error messages in the install log (install.log).

If you require assistance in resolving the issue, please include install.log
in your communications with Nagios XI technical support.

The script that failed was: './E-importnagiosql'

The same thread above followed-up by asking to see the statuses of mysqld, mariadb and ndo2db, which look like they might be enlightening:

Code: Select all

$ service mysqld status
Redirecting to /bin/systemctl status  mysqld.service
Unit mysqld.service could not be found.

$ service mariadb status
Redirecting to /bin/systemctl status  mariadb.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Tue 2017-03-28 05:25:04 UTC; 32min ago
  Process: 814 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=0/SUCCESS)
  Process: 813 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 784 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
 Main PID: 813 (code=exited, status=0/SUCCESS)

Mar 28 05:24:22 cyanelix-monitor systemd[1]: Starting MariaDB database server...
Mar 28 05:24:23 cyanelix-monitor mysqld_safe[813]: 170328 05:24:23 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Mar 28 05:24:23 cyanelix-monitor mysqld_safe[813]: 170328 05:24:23 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Mar 28 05:24:24 cyanelix-monitor systemd[1]: Started MariaDB database server.
Mar 28 05:25:03 cyanelix-monitor mysqld_safe[813]: /usr/bin/mysqld_safe: line 183:  1048 Killed                  nohup /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log...
Mar 28 05:25:04 cyanelix-monitor mysqld_safe[813]: 170328 05:25:04 mysqld_safe Number of processes running now: 0
Mar 28 05:25:04 cyanelix-monitor mysqld_safe[813]: 170328 05:25:04 mysqld_safe mysqld restarted
Mar 28 05:25:04 cyanelix-monitor mysqld_safe[813]: 170328 05:25:04 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
Hint: Some lines were ellipsized, use -l to show in full.

$ service ndo2db status
ndo2db (pid 1084) is running...

So - I'm guessing all is not well in MariaDB / MySQL land, but I'm reluctant to just dive in and start trying to fix them, as I'm conscious that the instructions explicitly call for a clean CentOS install, which is what I started with.

Any advice gratefully received.

[avandemore]

What does /var/log/mariadb/mariadb.log say? You can try running through this doc:
https://assets.nagios.com/downloads/nag ... tabase.pdf

[Cyanelix]

Thanks for the suggestion - looks like mariadb was running out of memory on startup, which makes sense as I'm using the smallest Digital Ocean VM with only 512MB RAM. I've tried turning off performance_schema as suggested here https://mariadb.com/resources/blog/star ... l-machines, which seems to have allowed me to start mariadb successfully. I'll try running the Nagios install again later and see what happens.

[cdienger]

Thanks for the update. Let us know your results after running the install again.

[Cyanelix]

So, I experimented a lot with the memory settings in /etc/my.cnf, none of which really achieved anything - MariaDB was still crashing almost immediately after starting, and frequently complaining about a lack of memory. After a bit of Googling I found this: http://www.webtrafficexchange.com/solve ... uffer-pool, and tried adding swap space as described in point 2 there. (Note here that if you do follow that example, there's a typo as mentioned in the comments and count=512M should be count=512k).

Re-running the fullinstall.sh script continued to fail, but I figured I'd been messing around a fair bit with the box at that point, so decided to start with a completely clean VM. Of course, I forgot to set the swap space, and just ran the install, but for some reason it worked perfectly this time, despite the setup being identical to my first attempt! However, as soon as I tried to access the Nagios URL, it crashed with an error about the DB, which reminded me that I hadn't set the swap space. So, I did that on the new box, and everything is now working fine!

TL;DR - if you're trying to use a tiny VM with 512MB RAM, try adding an additional 512MB swap space.

[cdienger]

Thanks for the update! Did you have any other questions related to this thread or are we okay closing it?