Message: A database connection error has been detected

[kendallchenoweth]

Hello,

I've followed the directions, http://assets.nagios.com/downloads/nagi ... Server.pdf, to use a remote mysql database. The mysql transation was complete and the services restarted, however the application is not working. I was prompted to repair the database (as prompted) and this doesn't make any difference to the application behavior. I am running the latest verison of the Nagios XI server and mysql is at version 5.1.73. My new mysql server is at version 5.6.21. I'm guessing that the dump produced by mysql 5.1 and/or the code in nagios XI is not compatible with mysql 5.6. Have you dealt with this before?

What is the latest version of mysql supported by Nagios XI 2014R2.0? My DBA isn't keen on using mysql 5.1 if he doesn't have to.

What is the best way to resolve this issue?

I'm getting the errors below in nagios.log

[1416849423] ndomod registered for adaptive contact data'
[1416849423] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1416849423] Successfully launched command file worker with pid 14186
[1416849439] ndomod: Successfully reconnected to data sink! 0 items lost, 3353 queued items to flush.
[1416849439] ndomod: Error writing to data sink! Some output may get lost. 3099 queued items to flush.


Thanks in advance!

-Kendall Chenoweth

[lmiltchev]

Can you go through the document again and make sure you didn't miss a step? Run the following command and show us the output:

Code: Select all

/sbin/sysctl -p

What is the latest version of mysql supported by Nagios XI 2014R2.0?

I am not sure about this one. I will talk to our devs and will try testing this in house.

[kendallchenoweth]

# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key
kernel.msgmnb = 393740288
kernel.msgmax = 131072000
kernel.shmmax = 4294967295
kernel.shmall = 268435456
kernel.randomize_va_space = 1
kernel.exec-shield = 1

I updated the kernel.msgmnb according to http://support.nagios.com/forum/viewtop ... 2&start=30 and http://support.nagios.com/forum/viewtop ... 7&start=10 and so far nothing has changed...

-Kendall Chenoweth

[abrist]

When the connection to the data sink is lost, is ndo2db still running?

Code: Select all

service ndo2db status
ps -aef | grep ndo 

[kendallchenoweth]

Code: Select all

[root@ip-10-154-25-117 etc]# tail -f /usr/local/nagios/var/nagios.log
[1416933483] ndomod registered for contact data'
[1416933483] ndomod registered for contact notification data'
[1416933483] ndomod registered for acknowledgement data'
[1416933483] ndomod registered for state change data'
[1416933483] ndomod registered for contact status data'
[1416933483] ndomod registered for adaptive contact data'
[1416933483] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1416933483] ndomod: Error writing to data sink!  Some output may get lost...
[1416933483] ndomod: Please check remote ndo2db log, database connection or SSL Parameters
[1416933483] Successfully launched command file worker with pid 1286
[1416933499] ndomod: Successfully reconnected to data sink!  0 items lost, 1837 queued items to flush.
[1416933499] ndomod: Error writing to data sink!  Some output may get lost.  1583 queued items to flush.
[1416933515] ndomod: Successfully reconnected to data sink!  0 items lost, 1675 queued items to flush.
[1416933515] ndomod: Error writing to data sink!  Some output may get lost.  1522 queued items to flush.
[1416933531] ndomod: Successfully reconnected to data sink!  0 items lost, 1554 queued items to flush.
[1416933531] ndomod: Error writing to data sink!  Some output may get lost.  1340 queued items to flush.

^C

Code: Select all

[root@ip-10-154-25-117 etc]# ps -eaf | grep ndo2db
nagios    1154     1  0 11:38 ?        00:00:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
root      1530  2878  0 11:39 pts/0    00:00:00 grep ndo2db
[root@ip-10-154-25-117 etc]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key
kernel.msgmnb = 393740288
kernel.msgmax = 131072000
kernel.shmmax = 4294967295
kernel.shmall = 268435456
kernel.randomize_va_space = 1
kernel.exec-shield = 1

<host>/nagios works okay, but <host>/nagiosxi returns

Code: Select all

Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.
Run the following from the CLI as root to attempt to repair the DB
/usr/local/nagiosxi/scripts/repair_databases.sh

...

Code: Select all

- recovering (with sort) MyISAM-table 'tbl_user.MYI'
Data records: 2
- Fixing index 1
- Fixing index 2
          
---------

- recovering (with sort) MyISAM-table 'tbl_variabledefinition.MYI'
Data records: 30
- Fixing index 1
Starting mysqld:                                           [  OK  ]
/usr/local/nagiosxi/scripts
 
===============
REPAIR COMPLETE
===============
Stopping ndo2db: done.
Starting ndo2db: done.
Running configuration check...done.
Stopping nagios: .done.
Starting nagios: done.

The repair doesn't do anything to fix the page loading problem and the nagios.log error persists...

Any ideas?

It looks like others may have run into issues with mysql 5.6. http://support.nagios.com/forum/viewtop ... =6&t=28352

What are the versions of mysql that are compatible with nagios XI? I haven't yet found anything in a log file that suggests a database error. I suspect that Nagios XI uses a mysql feature that's deprecated or changed in from mysql 5.1 to mysql 5.6. (I don't know this to be true; it's just a logical possibility.)

-Kendall Chenoweth

[abrist]

We will need to look at your config.inc.php and potentially look at the user permissions and iptables settings on the remote db. Could you send an email to [email protected] in order to open a ticket so that we can setup a remote session or at least investigate these files in confidence?

[kendallchenoweth]

I will arrange some time with our DBA and put some times available in the email I send you.

The remote database in an amazon RDS database. I don't think that this makes a difference, but I can tell you that the security group access is correct and verified by running mysql -u <user> -p for each user from the command line on the nagios server to the remote database. I think that should rule out iptables.

I think that the problem either lies with an incomplete update on my part for updating database host and password OR an application incompatibility with mysql 5.6. While we coordinate a remote session, can you provide me a complete list of files that should be updated for database host/password and I'll check those again. I'd also like confirmation that Nagios XI has been run and tested with mysql 5.6. If not, what is the latest version of mysql server that's been tested with Nagios XI? We can use a different version of mysql if necessary.

-Kendall Chenoweth

[abrist]

Could you send an email to [email protected] so that we can open a ticket and schedule a remote session?