Database Connection Error

[grenley]

tail -f shows that no new messages have been written to the mysql log since yesterday (probably about the time I ran the repair_databases script):

Code: Select all

[u@h W]$ tail -f /opt/app/mysql/log/mysqld.log
150216 20:23:14 InnoDB: Completed initialization of buffer pool
150216 20:23:14 InnoDB: highest supported file format is Barracuda.
150216 20:23:14  InnoDB: Waiting for the background threads to start
150216 20:23:15 InnoDB: 1.1.8 started; log sequence number 1595675
150216 20:23:15 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
150216 20:23:15 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
150216 20:23:15 [Note] Server socket created on IP: '0.0.0.0'.
150216 20:23:15 [Note] Event Scheduler: Loaded 0 events
150216 20:23:15 [Note] //opt/app/mysql/bin/mysqld: ready for connections.
Version: '5.5.25-log'  socket: '//opt/app/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)

Despite that, we continue to receive the messages to run that command and CCM says there are no hosts (though it does show services).
Could it be that the hosts table is corrupted?
Is there a way to reload it?

[abrist]

Could it be that the hosts table is corrupted?

Possibly, but you should get an error that the repair was unsuccessful. Try to repair the hosts table with:

Code: Select all

echo "repair table nagios.nagios_hosts;" | mysql -pnagiosxi

Is there a way to reload it?

Yes, though you could just truncate it and let ndo refill the table.

[grenley]

Okay. The repair came back with an OK status:

Code: Select all

[u@h W]$ echo "repair table nagios.nagios_hosts;" | mysql -u root -p******** nagios --socket=/opt/app/mysql/mysql.sock 
Table   Op      Msg_type        Msg_text
nagios.nagios_hosts     repair  status  OK

But CCM still shows "Error connecting to database." when I select Hosts under the Monitoring section.
It says "No results returned from host table"

When I select Services, a list of services is displayed (along with "Error connecting to database.").
It also asks me to apply a new configuration (which is odd since we haven't used CCM recently to manage the configuration).
As soon as I click "Apply Configuration", I get the following...

Code: Select all

An error occurred while attempting to apply your configuration to Nagios Core. Monitoring engine configuration files have been rolled back to their last known good checkpoint.

But they aren't really rolled back as it just recurs when I repeat the process.

[lmiltchev]

Run the following commands and show us the output in code wraps:

Code: Select all

service nagios restart
service nagios status
service ndo2db restart
service ndo2db status
service mysqld restart
service mysqld status
tail -20 /var/log/mysqld.log
grep ndo /var/log/messages | tail -20

[grenley]

It appears our installation process (not actually owned by the Nagios team) does not install these as services.
Here's what we get when I run the commands you provided:

Code: Select all

nagios: unrecognized service
nagios: unrecognized service
ndo2db: unrecognized service
ndo2db: unrecognized service
mysqld: unrecognized service
mysqld: unrecognized service

Here's what we use to stop these processes:

Code: Select all

su - nagios -c "/usr/local/nagios/etc/ndoutils/init.d/ndo2db-lrm-stop"
su - mysql -c "/opt/app/platform/init.d/mysql.server stop"
su - nagios -c "/usr/local/nagios/etc/init.d/nagios-lrm-stop"

(plus the similar start commands)

Will this suffice for your purposes?

[abrist]

They will indeed.

[grenley]

Code: Select all

[rg8239@zltc5114 ~]$ tail -20 /opt/app/mysql/log/mysqld.log

150218 16:32:30 mysqld_safe mysqld from pid file //opt/app/mysql/data/zltc5114.pid ended
150218 16:32:54 mysqld_safe Starting mysqld daemon with databases from //opt/app/mysql/data
150218 16:32:54 [Warning] Changed limits: max_open_files: 1024  max_connections: 214  table_cache: 400
150218 16:32:55 [Note] Plugin 'FEDERATED' is disabled.
150218 16:32:55 InnoDB: The InnoDB memory heap is disabled
150218 16:32:55 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150218 16:32:55 InnoDB: Compressed tables use zlib 1.2.3
150218 16:32:55 InnoDB: Using Linux native AIO
150218 16:32:55 InnoDB: Initializing buffer pool, size = 128.0M
150218 16:32:55 InnoDB: Completed initialization of buffer pool
150218 16:32:55 InnoDB: highest supported file format is Barracuda.
150218 16:32:55  InnoDB: Waiting for the background threads to start
150218 16:32:56 InnoDB: 1.1.8 started; log sequence number 1595675
150218 16:32:56 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
150218 16:32:56 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
150218 16:32:56 [Note] Server socket created on IP: '0.0.0.0'.
150218 16:32:56 [Note] Event Scheduler: Loaded 0 events
150218 16:32:56 [Note] //opt/app/mysql/bin/mysqld: ready for connections.
Version: '5.5.25-log'  socket: '//opt/app/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)

There are no ndo messages in /var/log/messages

[abrist]

Is it possible to move this too a ticket (for a profile.zip) and then possibly to a remote session? If so, send an email to [email protected] and lets resolve this.

[grenley]

sure. happy to do it.
what do you mean by profile.zip?

[abrist]

In XI, go to Admin --> System profile --> download. Send the profile.zip that you downloaded with your email to [email protected]. Also, include a link to this thread.
Thanks!