All hosts grey since reboot (again)

[tmcdonald]

The only thing that looked off was your postgres. As long as you are able to recover this like you have been, would you mind rebooting and running "service psql status" right away?

[WillemDH]

Trevor,

As I'm planning on upgrading my production Nagios server today, I had to take a snapshot. I did as you asked:

Code: Select all

 service psql status
psql: unrecognized service

Code: Select all

service postgresql status
postmaster (pid  2200) is running...

He didn't recognize psql, but I presume you meant postgresql?

Grtz

Willem

[abrist]

Could you check the DB for crashed tables? This can sometimes cause issue wit hretention as well:

Code: Select all

tail -100 /var/log/mysqld.log | grep crashed

[WillemDH]

Andy,

The output of mysqld.log (I had to shutdown the server and after that revert to a VMWare snapshot taken during the shutdown, because update to 2014 failed)

Code: Select all

140205 11:34:33 [Note] /usr/libexec/mysqld: Shutdown complete

140205 11:34:33 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140205 11:35:20 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140205 11:35:20  InnoDB: Initializing buffer pool, size = 8.0M
140205 11:35:20  InnoDB: Completed initialization of buffer pool
140205 11:35:20  InnoDB: Started; log sequence number 0 44233
140205 11:35:20 [Note] Event Scheduler: Loaded 0 events
140205 11:35:20 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140428 10:29:03 [Note] /usr/libexec/mysqld: Normal shutdown

140428 10:29:03 [Note] Event Scheduler: Purging the queue. 0 events
140428 10:29:03  InnoDB: Starting shutdown...
140428 10:29:05  InnoDB: Shutdown completed; log sequence number 0 44233
140428 10:29:05 [Note] /usr/libexec/mysqld: Shutdown complete

140428 10:29:05 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140428 10:32:03 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140428 10:32:03  InnoDB: Initializing buffer pool, size = 8.0M
140428 10:32:03  InnoDB: Completed initialization of buffer pool
140428 10:32:03  InnoDB: Started; log sequence number 0 44233
140428 10:32:03 [Note] Event Scheduler: Loaded 0 events
140428 10:32:03 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140430 20:12:51 [Note] /usr/libexec/mysqld: Normal shutdown

140430 20:12:51 [Note] Event Scheduler: Purging the queue. 0 events
140430 20:12:51  InnoDB: Starting shutdown...
140430 20:12:55  InnoDB: Shutdown completed; log sequence number 0 44233
140430 20:12:55 [Note] /usr/libexec/mysqld: Shutdown complete

140430 20:12:55 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140430 20:14:10 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140430 20:14:10  InnoDB: Initializing buffer pool, size = 8.0M
140430 20:14:10  InnoDB: Completed initialization of buffer pool
140430 20:14:10  InnoDB: Started; log sequence number 0 44233
140430 20:14:10 [Note] Event Scheduler: Loaded 0 events
140430 20:14:10 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140506 10:25:18 [Note] /usr/libexec/mysqld: Normal shutdown

140506 10:25:18 [Note] Event Scheduler: Purging the queue. 0 events
140506 10:25:20  InnoDB: Starting shutdown...
140506 10:25:24  InnoDB: Shutdown completed; log sequence number 0 44233
140506 10:25:24 [Note] /usr/libexec/mysqld: Shutdown complete

140506 10:25:24 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140506 10:28:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140506 10:28:11  InnoDB: Initializing buffer pool, size = 8.0M
140506 10:28:11  InnoDB: Completed initialization of buffer pool
140506 10:28:11  InnoDB: Started; log sequence number 0 44233
140506 10:28:11 [Note] Event Scheduler: Loaded 0 events
140506 10:28:11 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140506 10:32:21 [Note] /usr/libexec/mysqld: Normal shutdown

140506 10:32:21 [Note] Event Scheduler: Purging the queue. 0 events
140506 10:32:23  InnoDB: Starting shutdown...
140506 10:32:26  InnoDB: Shutdown completed; log sequence number 0 44233
140506 10:32:26 [Note] /usr/libexec/mysqld: Shutdown complete

140506 10:32:26 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140506 10:33:09 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140506 10:33:09  InnoDB: Initializing buffer pool, size = 8.0M
140506 10:33:09  InnoDB: Completed initialization of buffer pool
140506 10:33:09  InnoDB: Started; log sequence number 0 44233
140506 10:33:09 [Note] Event Scheduler: Loaded 0 events
140506 10:33:09 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140528 11:55:50 [Note] /usr/libexec/mysqld: Normal shutdown

140528 11:55:51 [Note] Event Scheduler: Purging the queue. 0 events
140528 11:55:51  InnoDB: Starting shutdown...
140528 11:55:52  InnoDB: Shutdown completed; log sequence number 0 44233
140528 11:55:52 [Note] /usr/libexec/mysqld: Shutdown complete

140528 11:55:52 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140528 11:57:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140528 11:57:30  InnoDB: Initializing buffer pool, size = 8.0M
140528 11:57:30  InnoDB: Completed initialization of buffer pool
140528 11:57:30  InnoDB: Started; log sequence number 0 44233
140528 11:57:31 [Note] Event Scheduler: Loaded 0 events
140528 11:57:31 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140611 10:49:27 [Note] /usr/libexec/mysqld: Normal shutdown

140611 10:49:27 [Note] Event Scheduler: Purging the queue. 0 events
140611 10:49:27  InnoDB: Starting shutdown...
140611 10:49:30  InnoDB: Shutdown completed; log sequence number 0 44233
140611 10:49:30 [Note] /usr/libexec/mysqld: Shutdown complete

140611 10:49:30 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140611 11:07:27 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140611 11:07:27  InnoDB: Initializing buffer pool, size = 8.0M
140611 11:07:27  InnoDB: Completed initialization of buffer pool
140611 11:07:27  InnoDB: Started; log sequence number 0 44233
140611 11:07:27 [Note] Event Scheduler: Loaded 0 events
140611 11:07:27 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

Grtz

Willem

[lmiltchev]

Try running the upgrade script again. If you run into some config errors (where the output doesn't indicate what the actual error is), deactivate temporarily the service escalations and service dependencies, that you have defined, under the CCM, and try to upgrade again. Hope this helps.

[WillemDH]

Ludmill,

The update to 2014 was failing due to some error with mrtg, see this thread: http://support.nagios.com/forum/viewtop ... 16&t=27635

Grtz

[lmiltchev]

I'm sorry, WillemDH. I just saw your other post...

[WillemDH]

No problem Ludmill,

I'll focus on making this upgrade to 2014 work. maybe this issue is fixed after the upgrade. i'll let you know.

Grtz

Willem

[slansing]

Great, keep us updated on how it goes!

[WillemDH]

Just tested again and we still have the same issue. After a reboot, I need to manually apply configuration.