Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

All hosts grey since reboot (again)

https://support.nagios.com/forum/viewtopic.php?t=25432

Page 3 of 4

Re: All hosts grey since reboot (again)

Posted: Wed May 28, 2014 4:30 pm
by tmcdonald
The only thing that looked off was your postgres. As long as you are able to recover this like you have been, would you mind rebooting and running "service psql status" right away?

Re: All hosts grey since reboot (again)

Posted: Wed Jun 11, 2014 3:55 am
by WillemDH
Trevor,

As I'm planning on upgrading my production Nagios server today, I had to take a snapshot. I did as you asked:

Code: Select all

 service psql status
psql: unrecognized service

Code: Select all

service postgresql status
postmaster (pid  2200) is running...
He didn't recognize psql, but I presume you meant postgresql?

Grtz

Willem

Re: All hosts grey since reboot (again)

Posted: Wed Jun 11, 2014 10:56 am
by abrist
Could you check the DB for crashed tables? This can sometimes cause issue wit hretention as well:

Code: Select all

tail -100 /var/log/mysqld.log | grep crashed

Re: All hosts grey since reboot (again)

Posted: Wed Jun 11, 2014 12:47 pm
by WillemDH
Andy,

The output of mysqld.log (I had to shutdown the server and after that revert to a VMWare snapshot taken during the shutdown, because update to 2014 failed)

Code: Select all

140205 11:34:33 [Note] /usr/libexec/mysqld: Shutdown complete

140205 11:34:33 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140205 11:35:20 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140205 11:35:20  InnoDB: Initializing buffer pool, size = 8.0M
140205 11:35:20  InnoDB: Completed initialization of buffer pool
140205 11:35:20  InnoDB: Started; log sequence number 0 44233
140205 11:35:20 [Note] Event Scheduler: Loaded 0 events
140205 11:35:20 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140428 10:29:03 [Note] /usr/libexec/mysqld: Normal shutdown

140428 10:29:03 [Note] Event Scheduler: Purging the queue. 0 events
140428 10:29:03  InnoDB: Starting shutdown...
140428 10:29:05  InnoDB: Shutdown completed; log sequence number 0 44233
140428 10:29:05 [Note] /usr/libexec/mysqld: Shutdown complete

140428 10:29:05 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140428 10:32:03 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140428 10:32:03  InnoDB: Initializing buffer pool, size = 8.0M
140428 10:32:03  InnoDB: Completed initialization of buffer pool
140428 10:32:03  InnoDB: Started; log sequence number 0 44233
140428 10:32:03 [Note] Event Scheduler: Loaded 0 events
140428 10:32:03 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140430 20:12:51 [Note] /usr/libexec/mysqld: Normal shutdown

140430 20:12:51 [Note] Event Scheduler: Purging the queue. 0 events
140430 20:12:51  InnoDB: Starting shutdown...
140430 20:12:55  InnoDB: Shutdown completed; log sequence number 0 44233
140430 20:12:55 [Note] /usr/libexec/mysqld: Shutdown complete

140430 20:12:55 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140430 20:14:10 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140430 20:14:10  InnoDB: Initializing buffer pool, size = 8.0M
140430 20:14:10  InnoDB: Completed initialization of buffer pool
140430 20:14:10  InnoDB: Started; log sequence number 0 44233
140430 20:14:10 [Note] Event Scheduler: Loaded 0 events
140430 20:14:10 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140506 10:25:18 [Note] /usr/libexec/mysqld: Normal shutdown

140506 10:25:18 [Note] Event Scheduler: Purging the queue. 0 events
140506 10:25:20  InnoDB: Starting shutdown...
140506 10:25:24  InnoDB: Shutdown completed; log sequence number 0 44233
140506 10:25:24 [Note] /usr/libexec/mysqld: Shutdown complete

140506 10:25:24 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140506 10:28:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140506 10:28:11  InnoDB: Initializing buffer pool, size = 8.0M
140506 10:28:11  InnoDB: Completed initialization of buffer pool
140506 10:28:11  InnoDB: Started; log sequence number 0 44233
140506 10:28:11 [Note] Event Scheduler: Loaded 0 events
140506 10:28:11 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140506 10:32:21 [Note] /usr/libexec/mysqld: Normal shutdown

140506 10:32:21 [Note] Event Scheduler: Purging the queue. 0 events
140506 10:32:23  InnoDB: Starting shutdown...
140506 10:32:26  InnoDB: Shutdown completed; log sequence number 0 44233
140506 10:32:26 [Note] /usr/libexec/mysqld: Shutdown complete

140506 10:32:26 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140506 10:33:09 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140506 10:33:09  InnoDB: Initializing buffer pool, size = 8.0M
140506 10:33:09  InnoDB: Completed initialization of buffer pool
140506 10:33:09  InnoDB: Started; log sequence number 0 44233
140506 10:33:09 [Note] Event Scheduler: Loaded 0 events
140506 10:33:09 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140528 11:55:50 [Note] /usr/libexec/mysqld: Normal shutdown

140528 11:55:51 [Note] Event Scheduler: Purging the queue. 0 events
140528 11:55:51  InnoDB: Starting shutdown...
140528 11:55:52  InnoDB: Shutdown completed; log sequence number 0 44233
140528 11:55:52 [Note] /usr/libexec/mysqld: Shutdown complete

140528 11:55:52 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140528 11:57:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140528 11:57:30  InnoDB: Initializing buffer pool, size = 8.0M
140528 11:57:30  InnoDB: Completed initialization of buffer pool
140528 11:57:30  InnoDB: Started; log sequence number 0 44233
140528 11:57:31 [Note] Event Scheduler: Loaded 0 events
140528 11:57:31 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140611 10:49:27 [Note] /usr/libexec/mysqld: Normal shutdown

140611 10:49:27 [Note] Event Scheduler: Purging the queue. 0 events
140611 10:49:27  InnoDB: Starting shutdown...
140611 10:49:30  InnoDB: Shutdown completed; log sequence number 0 44233
140611 10:49:30 [Note] /usr/libexec/mysqld: Shutdown complete

140611 10:49:30 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140611 11:07:27 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140611 11:07:27  InnoDB: Initializing buffer pool, size = 8.0M
140611 11:07:27  InnoDB: Completed initialization of buffer pool
140611 11:07:27  InnoDB: Started; log sequence number 0 44233
140611 11:07:27 [Note] Event Scheduler: Loaded 0 events
140611 11:07:27 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
Grtz

Willem

Re: All hosts grey since reboot (again)

Posted: Wed Jun 11, 2014 1:51 pm
by lmiltchev
Try running the upgrade script again. If you run into some config errors (where the output doesn't indicate what the actual error is), deactivate temporarily the service escalations and service dependencies, that you have defined, under the CCM, and try to upgrade again. Hope this helps.

Re: All hosts grey since reboot (again)

Posted: Wed Jun 11, 2014 2:22 pm
by WillemDH
Ludmill,

The update to 2014 was failing due to some error with mrtg, see this thread: http://support.nagios.com/forum/viewtop ... 16&t=27635

Grtz

Re: All hosts grey since reboot (again)

Posted: Wed Jun 11, 2014 3:20 pm
by lmiltchev
I'm sorry, WillemDH. I just saw your other post...

Re: All hosts grey since reboot (again)

Posted: Thu Jun 12, 2014 5:59 am
by WillemDH
No problem Ludmill,

I'll focus on making this upgrade to 2014 work. maybe this issue is fixed after the upgrade. i'll let you know.

Grtz

Willem

Re: All hosts grey since reboot (again)

Posted: Wed Jun 18, 2014 9:40 am
by slansing
Great, keep us updated on how it goes!

Re: All hosts grey since reboot (again)

Posted: Fri Jun 20, 2014 3:49 am
by WillemDH
Just tested again and we still have the same issue. After a reboot, I need to manually apply configuration.
All times are UTC-05:00
Page 3 of 4

Powered by phpBB® Forum Software © phpBB Limited