Page 1 of 2
Nagios Crashed - Web won't Load
Posted: Wed Dec 04, 2013 12:23 pm
by karnaugh
I am trying to Pilot NagiosXI. I have installed the HyperV 64bit template.
When I initially set it up, I only provided 512 MB of RAM (I think that was an issue). It was working fine when I left for home last night. When I returned this morning there were errors stating that the box ran out of memory. I had to hard shut down the server to reboot it. Once rebooting it, I try to log into the webgui and I get a blank screen, I do not get my login screen or anything.
I have since then tried to safely reboot the server to no avail. How do I get my NagiosXI instance webUI to work again for me?
Any help appreciated.
Re: Nagios Crashed - Web won't Load
Posted: Wed Dec 04, 2013 12:26 pm
by tmcdonald
Hard shutdowns can cause issues with the DB. Read over this document and take the steps listed there.
http://assets.nagios.com/downloads/nagi ... tabase.pdf
Re: Nagios Crashed - Web won't Load
Posted: Wed Dec 04, 2013 1:09 pm
by karnaugh
I ran the required database repair script. When it got to the end this is what it said:
Another MySQL daemon already running with the same unix socket.
Starting mysqld: [FAILED]
~
===============
REPAIR COMPLETE
===============
When I tried to start the mysqld manually this is what I get:
[root@PENMS01 ~]# service mysqld start
Another MySQL daemon already running with the same unix socket.
Starting mysqld: [FAILED]
Re: Nagios Crashed - Web won't Load
Posted: Wed Dec 04, 2013 1:17 pm
by tmcdonald
Did you read the whole document? Specifically where it says:
Next, stop the MySQL database server with the following command:
before running the script?
Re: Nagios Crashed - Web won't Load
Posted: Wed Dec 04, 2013 1:19 pm
by karnaugh
I did that. It said that it stopped successfully
[root@PENMS01 ~]# service mysqld stop
Stopping mysqld: [ OK ]
[root@PENMS01 ~]# /usr/local/nagiosxi/scripts/repairmysql.sh nagios
DATABASE: nagios
TABLE:
/var/lib/mysql/nagios ~
Stopping mysqld: [ OK ]
- recovering (with sort) MyISAM-table 'nagios_acknowledgements.MYI'
Data records: 0
- Fixing index 1
Still get the same error at the bottom of the script
Re: Nagios Crashed - Web won't Load
Posted: Wed Dec 04, 2013 1:40 pm
by karnaugh
Is there anyway to tell what other MySQL daemons are running?
Sorry I am not a strong Linux admin.
I ran ps aux and I do not see anything relating to MySQL in the running processes. Would httpd or nagios be holding the MySQL daemon open?
Re: Nagios Crashed - Web won't Load
Posted: Wed Dec 04, 2013 1:45 pm
by karnaugh
Not sure if this helps:
[root@PENMS01 ~]# tail /var/log/httpd/error_log
[Wed Dec 04 12:22:00 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Dec 04 12:22:00 2013] [notice] Digest: generating secret for digest authentication ...
[Wed Dec 04 12:22:00 2013] [notice] Digest: done
[Wed Dec 04 12:22:00 2013] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips configured -- resuming normal operations
[Wed Dec 04 12:22:14 2013] [error] [client 172.16.3.23] PHP Warning: mysql_pconnect(): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383
[Wed Dec 04 12:22:14 2013] [error] [client 172.16.3.23] PHP Warning: mysql_pconnect(): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383
[Wed Dec 04 12:22:14 2013] [error] [client 172.16.3.23] PHP Warning: mysql_pconnect(): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383
[Wed Dec 04 12:22:14 2013] [error] [client 172.16.3.23] PHP Warning: mysql_pconnect(): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383
[Wed Dec 04 12:22:14 2013] [error] [client 172.16.3.23] PHP Warning: mysql_pconnect(): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383
[Wed Dec 04 12:22:14 2013] [error] [client 172.16.3.23] PHP Warning: mysql_pconnect(): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysql.inc.php on line 383
Re: Nagios Crashed - Web won't Load
Posted: Wed Dec 04, 2013 2:00 pm
by slansing
Please run through these commands and check the web interface:
Code: Select all
service mysqld stop
killall -9 mysqld
service mysqld start
Code: Select all
psql nagiosxi nagiosxi
VACUUM;
VACUUM ANALYZE;
VACUUM FULL;
\q
You may see warnings from the above and below postgres commands, you can ignore them and continue:
Code: Select all
psql postgres postgres
VACUUM;
VACUUM ANALYZE;
VACUUM FULL;
\q
When this is finished please run:
Then try to access your XI server's url, once the page loads "but still does not function," or displays a blank page, close the log tail out and post it here. Also please attach the output from each of the commands you ran above.
Re: Nagios Crashed - Web won't Load
Posted: Wed Dec 04, 2013 2:01 pm
by abrist
Try killing all mysql processes . . . {though this can be potentially dangerous to the db):
Code: Select all
killall mysqld
ps -aef | grep mysqld
/usr/local/nagiosxi/scripts/repairmysql.sh nagios
service mysqld restart
Re: Nagios Crashed - Web won't Load
Posted: Wed Dec 04, 2013 2:08 pm
by karnaugh
Thanks for all the replies.
I only had a handful of items configured for monitoring, and nothing to critical.
I deleted the VM and recreated it from scratch. This time I added 1GB of RAM. I hope that will be enough.
Thanks again