Nagios Support Forum

Well, now I have confirmation that Nagios is NOT working properly... we had a production problem yesterday that did not get caught for hours because no alerts fired. I attempted to call the tech support line but was put to voicemail.

So... I've been digging around in logs and whatnot (waiting for a return email or phone call from support) and I see this in /var/log/mysql.log:

===================================================
131002 09:08:27 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
131002 9:08:27 [Note] libgovernor.so not found
131002 9:08:27 [Note] Plugin 'FEDERATED' is disabled.
/usr/libexec/mysqld: Table 'mysql.plugin' doesn't exist
131002 9:08:27 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
131002 9:08:27 InnoDB: The InnoDB memory heap is disabled
131002 9:08:27 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131002 9:08:27 InnoDB: Compressed tables use zlib 1.2.3
131002 9:08:27 InnoDB: Using Linux native AIO
131002 9:08:27 InnoDB: Initializing buffer pool, size = 128.0M
131002 9:08:27 InnoDB: Completed initialization of buffer pool
131002 9:08:27 InnoDB: highest supported file format is Barracuda.
131002 9:08:27 InnoDB: Waiting for the background threads to start
131002 9:08:28 InnoDB: 5.5.33 started; log sequence number 1588761
131002 9:08:28 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
131002 9:08:28 [Note] - '0.0.0.0' resolves to '0.0.0.0';
131002 9:08:28 [Note] Server socket created on IP: '0.0.0.0'.
131002 9:08:28 [ERROR] Missing system table mysql.proxies_priv; please run mysql_upgrade to create it
131002 9:08:28 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
131002 9:08:28 [ERROR] Native table 'performance_schema'.'events_waits_current' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'events_waits_history' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'events_waits_history_long' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'setup_consumers' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'setup_instruments' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'setup_timers' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'performance_timers' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'threads' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_thread_by_event_name' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_instance' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'events_waits_summary_global_by_event_name' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'file_summary_by_event_name' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'file_summary_by_instance' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'mutex_instances' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'rwlock_instances' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'cond_instances' has the wrong structure
131002 9:08:28 [ERROR] Native table 'performance_schema'.'file_instances' has the wrong structure
131002 9:08:28 [ERROR] Column count of mysql.db is wrong. Expected 22, found 20. Created with MySQL 50077, now running 50533. Please use mysql_upgrade to fix this error.
131002 9:08:28 [ERROR] mysql.user has no `Event_priv` column at position 29
131002 9:08:28 [ERROR] Cannot open mysql.event
131002 9:08:28 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler.
131002 9:08:28 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.33-cll-lve' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Atomicorp
===================================================

I've run the repair tool, but this problem persists...

Was mysql updated as well?

It kind of looks like it did... though I'm not certain how/when. I typically push updates to all my servers via a Redhat Satellite server and the mysql I've got installed came from an atomic repo, which Satellite can't manage/update for me. However, it does look like I am running the very latest of mysql. (Note to self - exclude mysql* in the yum.conf file!).

# rpm -qa|grep mysql
mysqlclient15-5.0.90-2.el5.art
mysql-server-5.5.33-20.el5.art
mysql-libs-5.5.33-20.el5.art
mysql-5.5.33-20.el5.art
php-mysql-5.4.19-25.el5.art
mysql-devel-5.5.33-20.el5.art

I tried running the mysql_upgrade command but it gives the following:

# mysql_upgrade
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
FATAL ERROR: Upgrade failed


What's weird is that I can do manual writes to the db via the Nagios admin page and checks - everything returns successful:

Write host configurations ...
Host configuration files successfully written!

Write service configurations ...
Service configuration files successfully written!

Configuration file: hostgroups.cfg successfully written!
Configuration file: servicegroups.cfg successfully written!
Configuration file: hosttemplates.cfg successfully written!
Configuration file: servicetemplates.cfg successfully written!
Configuration file: timeperiods.cfg successfully written!
Configuration file: commands.cfg successfully written!
Configuration file: contacts.cfg successfully written!
Configuration file: contactgroups.cfg successfully written!
Configuration file: contacttemplates.cfg successfully written!
Configuration file: servicedependencies.cfg successfully written!
Configuration file: hostdependencies.cfg successfully written!
Configuration file: serviceescalations.cfg successfully written!
Configuration file: hostescalations.cfg successfully written!
Configuration file: serviceextinfo.cfg successfully written!
Configuration file: hostextinfo.cfg successfully written!

The verify also is successful:

Nagios Core 3.5.0
Copyright (c) 2009-2011 Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 03-15-2013
License: GPL

Website: http://www.nagios.org
Reading configuration data...
Read main config file okay...
Processing object config file '/usr/local/nagios/etc/objects/commands.cfg'...
Processing object config file '/usr/local/nagios/etc/objects/contacts.cfg'...
Processing object config file '/usr/local/nagios/etc/objects/timeperiods.cfg'...
Processing object config file '/usr/local/nagios/etc/objects/templates.cfg'...
Processing object config file '/usr/local/nagios/etc/objects/localhost.cfg'...
Read object config files okay...

Running pre-flight check on configuration data...

Checking services...
Checked 8 services.
Checking hosts...
Checked 1 hosts.
Checking host groups...
Checked 1 host groups.
Checking service groups...
Checked 0 service groups.
Checking contacts...
Checked 1 contacts.
Checking contact groups...
Checked 1 contact groups.
Checking service escalations...
Checked 0 service escalations.
Checking service dependencies...
Checked 0 service dependencies.
Checking host escalations...
Checked 0 host escalations.
Checking host dependencies...
Checked 0 host dependencies.
Checking commands...
Checked 24 commands.
Checking time periods...
Checked 5 time periods.
Checking for circular paths between hosts...
Checking for circular host and service dependencies...
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check

ya these errors you have in the mysql log are not related to any Nagios XI DB's, but mysql tabes, it actually looks like the mysql upgrade did not go right.

OK, so I fixed the mysql issues... I had to run the mysql_upgrade with a username/password for it to work. I then had to repair some of the mysql db tables and it restarts without complaint now:

131002 09:42:47 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
131002 9:42:47 [Note] libgovernor.so not found
131002 9:42:47 [Note] Plugin 'FEDERATED' is disabled.
131002 9:42:47 InnoDB: The InnoDB memory heap is disabled
131002 9:42:47 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131002 9:42:47 InnoDB: Compressed tables use zlib 1.2.3
131002 9:42:47 InnoDB: Using Linux native AIO
131002 9:42:47 InnoDB: Initializing buffer pool, size = 128.0M
131002 9:42:47 InnoDB: Completed initialization of buffer pool
131002 9:42:47 InnoDB: highest supported file format is Barracuda.
131002 9:42:47 InnoDB: Waiting for the background threads to start
131002 9:42:48 InnoDB: 5.5.33 started; log sequence number 1589260
131002 9:42:48 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
131002 9:42:48 [Note] - '0.0.0.0' resolves to '0.0.0.0';
131002 9:42:48 [Note] Server socket created on IP: '0.0.0.0'.
131002 9:42:48 [Note] Event Scheduler: Loaded 0 events
131002 9:42:48 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.33-cll-lve' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Atomicorp

That alone didn't seem to have any effect on Nagios. I did restart the database and restarted the monitoring engine, but no-go. I'm restarting the whole server on the off chance that does something.

You've opened a new support ticket in our ticketing system. We will continue communication via email. I am locking this topic.