Page 1 of 1
Hanging XI interface after database server swap
Posted: Tue Sep 22, 2020 12:38 pm
by YannPingot
Hello,
Considering our old version of MySQL running on a RHEL 6, we want to upgrade our offloaded database as a try to improve our overall performances as we have recurrent issues with ipcs queues filling up.
A MariaDB 10.5 (latest version) has been installed on a RHEL 7.7, all 3 databases dumped and imported on the new one.
After changing variables in config files and everything restarted Nagios core seems to be working pretty well as well as NDO2DB, but the XI interface is completely unusable, showing partial interface and multiple databases errors if it respond after several seconds/minutes :
Select of database nagiosxi failed. Lost connection to MySQL server during query.
Select of database nagios failed. Lost connection to MySQL server during query.
Select of database nagiosql failed. Lost connection to MySQL server during query.
Also Mysql tuning parameters have been copied to the new database so it should at least run like the old one.
We're using version 5.7.3 with rollbacked NDO to ndo2db (as NDO3 wasn't working, hoping that this new Mariadb version would improve this).
Thanks for your help,
Yann
Re: Hanging XI interface after database server swap
Posted: Wed Sep 23, 2020 2:38 pm
by jdunitz
Re: Hanging XI interface after database server swap
Posted: Wed Sep 30, 2020 3:07 am
by YannPingot
Hello Jeffrey,
Unfortunately it didn't change anything so i had to rollback again, the UI is still completely unusable, but the core + ndo2db seems to be working fine as it was sending alarms and the ipcs queue unstacking.
The way it works appears to be some kind of timeout, but we can access databases perfectly fine with a mysql client and there is no firewall involved here.
I've tried activating PHP logs but nothing more here (btw we're using PHP 5.3.3).
Any thoughts ?
Thanks,
Yann
Re: Hanging XI interface after database server swap
Posted: Wed Sep 30, 2020 5:32 pm
by ssax
Yeah, for MariaDB 10 you'll need to set sql_mode="" from the guide jdunitz posted. Also, try setting max_allowed_packet=512M in your /etc/my.cnf as well.
It shows it's failing for all databases (nagios, nagiosql, nagiosxi) so it's likely the
/usr/local/nagiosxi/html/config.inc.php that wasn't proper or there's something else going on.
Actually, did you update this file:
Or this file?
XI 5.7.2 should use the new NDO by default which would be this one:
Do you have a copy of your profile from the upgraded state? If not, send a copy of your current one from Admin > System Profile by clicking the Download Profile button.
Re: Hanging XI interface after database server swap
Posted: Thu Oct 01, 2020 10:00 am
by YannPingot
I've updated ndo2db.cfg as we've rollbacked to NDO2 like is said in my first post.
I've indeed set the sql_mode in the cnf as modifying the parameter through the CLI is not persistent.
max_allowed_packet has been set to 256M as our current DB instance, are you sure increasing it to the double would help ?
I can't retrieve a profile since the UI is totally unresponsive once on the new DB...
Regarding the config.inc.php i've literally just modified the backend address since i've recreated users identically.
Re: Hanging XI interface after database server swap
Posted: Fri Oct 02, 2020 10:35 am
by ssax
I would increase it to 512M and see if it helps:
https://dev.mysql.com/doc/refman/5.7/en ... ction.html
Are you seeing any messages on /var/log/messages or /var/log/mariadb/mariadb.log on the offloaded DB server?
If you're unable to generate the the profile through the web interface, please try generating it from the command line by running these commands as root:
Code: Select all
rm -rf /usr/local/nagiosxi/var/components/profile*
/usr/local/nagiosxi/scripts/components/getprofile.sh SUPPORT
Then send me the resulting
/usr/local/nagiosxi/var/components/profile.zip file.
If the profile script fails, please include the ENTIRE output.
Re: Hanging XI interface after database server swap
Posted: Wed Oct 07, 2020 11:43 am
by YannPingot
Hello Ssax,
I've sent you a PM with the profile attached.
As expected unfortunately increasing the packet size didn't change anything.
Is there a possibility that we could have somewhere else on the web part a cache which could cause problems like that ?
Thanks,
Best regards,
Yann Pingot
Re: Hanging XI interface after database server swap
Posted: Thu Oct 08, 2020 11:51 am
by ssax
You currently are running the old NDO2DB but your
/usr/local/nagios/etc/ndo2db.cfg file does not match your
/usr/local/nagiosxi/html/config.inc.php for the IP address. Which IP is proper? You'll need to fix one or the other and make sure they all match, then restart the services:
-- I added some cleanup commands as well just in case
Code: Select all
systemctl stop httpd
systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
systemctl stop mod-gearman-worker
systemctl stop gearmand
pkill -9 -u nagios
pkill -9 -u apache
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -f /usr/local/nagiosxi/var/dbmaint.lock
rm -f /usr/local/nagiosxi/var/event_handler.lock
rm -f /usr/local/nagiosxi/var/reconfigure_nagios.lock
rm -f /usr/local/nagios/var/ndo2db.lock
rm -f /usr/local/nagios/var/ndo2db.pid
rm -f /usr/local/nagios/var/ndo2db.sock
rm -f /usr/local/nagios/var/ndo.sock
rm -f /us/local/nagiosxi/var/subsys/ndo2db
rm -f /var/run/nagios/nagios.lock
rm -f /var/run/nagios.lock
rm -f /usr/local/nagios/var/nagios.lock
rm -f /var/run/httpd/httpd.pid
rm -f /usr/local/nagiosxi/var/subsys/npcd.pid
systemctl start gearmand
systemctl start mod-gearman-worker
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond
systemctl restart httpd
systemctl restart snmptt
Re: Hanging XI interface after database server swap
Posted: Mon Oct 12, 2020 3:37 am
by YannPingot
My mistake when i created the profile zip, that's why you have a different IP indeed, but of course on my earlier tries it was matching and the behavior was the same (in that case the web part is correctly pointing to the new database).
I'll try again with all cleanup commands you mentioned (well, with their RHEL6 counterparts
), maybe this is linked to sockets or lock files
Re: Hanging XI interface after database server swap
Posted: Mon Oct 12, 2020 3:26 pm
by ssax
Let us know if that resolves it.
If it doesn't, please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:
https://support.nagios.com/tickets/
Thank you!