Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

So our data center decided to update our servers

https://support.nagios.com/forum/viewtopic.php?t=9372

Page 1 of 3

So our data center decided to update our servers

Posted: Fri Feb 15, 2013 11:40 am
by disrael
We had our DC team update:

Code: Select all


Feb 15 01:31:40 usa7061lv981 yum[8291]: Updated: tzdata-2012j-1.el6.noarch
Feb 15 01:31:49 usa7061lv981 yum[8291]: Updated: glibc-2.12-1.80.el6_3.7.x86_64
Feb 15 01:31:56 usa7061lv981 yum[8291]: Updated: glibc-common-2.12-1.80.el6_3.7.x86_64
Feb 15 01:31:57 usa7061lv981 yum[8291]: Updated: krb5-libs-1.9-33.el6_3.3.x86_64
Feb 15 01:31:57 usa7061lv981 yum[8291]: Updated: openssl-1.0.0-25.el6_3.1.x86_64
Feb 15 01:31:57 usa7061lv981 yum[8291]: Updated: nspr-4.9.2-0.el6_3.1.x86_64
Feb 15 01:31:57 usa7061lv981 yum[8291]: Updated: nss-util-3.13.6-1.el6_3.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: nss-3.13.6-2.el6_3.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: nss-sysinit-3.13.6-2.el6_3.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: cyrus-sasl-lib-2.1.23-13.el6_3.1.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: 1:dbus-libs-1.2.24-7.el6_3.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: libreport-plugin-reportuploader-2.0.9-5.el6_3.2.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: libreport-2.0.9-5.el6_3.2.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: libreport-plugin-rhtsupport-2.0.9-5.el6_3.2.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: libreport-python-2.0.9-5.el6_3.2.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: freetype-2.3.11-14.el6_3.1.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: pango-1.28.1-7.el6_3.x86_64
Feb 15 01:31:59 usa7061lv981 yum[8291]: Updated: abrt-libs-2.0.8-6.el6_3.2.x86_64
Feb 15 01:31:59 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:31:59 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:31:59 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:31:59 usa7061lv981 yum[8291]: Updated: abrt-2.0.8-6.el6_3.2.x86_64
Feb 15 01:31:59 usa7061lv981 yum[8291]: Updated: 1:dbus-1.2.24-7.el6_3.x86_64
Feb 15 01:32:00 usa7061lv981 yum[8291]: Updated: mysql-libs-5.1.67-1.el6_3.x86_64
Feb 15 01:32:00 usa7061lv981 yum[8291]: Updated: mysql-5.1.67-1.el6_3.x86_64
Feb 15 01:32:00 usa7061lv981 yum[8291]: Updated: libproxy-python-0.3.0-3.el6_3.x86_64
Feb 15 01:32:00 usa7061lv981 yum[8291]: Updated: libproxy-0.3.0-3.el6_3.x86_64
Feb 15 01:32:00 usa7061lv981 yum[8291]: Updated: libproxy-bin-0.3.0-3.el6_3.x86_64
Feb 15 01:32:01 usa7061lv981 yum[8291]: Updated: systemtap-devel-1.7-5.el6_3.1.x86_64
Feb 15 01:32:01 usa7061lv981 yum[8291]: Updated: nss-tools-3.13.6-2.el6_3.x86_64
Feb 15 01:32:01 usa7061lv981 yum[8291]: Updated: openldap-2.4.23-26.el6_3.2.x86_64
Feb 15 01:32:01 usa7061lv981 yum[8291]: Updated: cyrus-sasl-2.1.23-13.el6_3.1.x86_64
Feb 15 01:32:01 usa7061lv981 yum[8291]: Updated: openssh-5.3p1-81.el6_3.x86_64
Feb 15 01:32:02 usa7061lv981 yum[8291]: Updated: nspr-devel-4.9.2-0.el6_3.1.x86_64
Feb 15 01:32:03 usa7061lv981 yum[8291]: Updated: 1:net-snmp-libs-5.5-41.el6_3.1.x86_64
Feb 15 01:32:03 usa7061lv981 yum[8291]: Updated: kpartx-0.4.9-56.el6_3.1.x86_64
Feb 15 01:32:03 usa7061lv981 yum[8291]: Updated: nss-util-devel-3.13.6-1.el6_3.x86_64
Feb 15 01:32:03 usa7061lv981 yum[8291]: Updated: openssh-clients-5.3p1-81.el6_3.x86_64
Feb 15 01:32:03 usa7061lv981 yum[8291]: Updated: cyrus-sasl-devel-2.1.23-13.el6_3.1.x86_64
Feb 15 01:32:04 usa7061lv981 yum[8291]: Updated: 1:dbus-x11-1.2.24-7.el6_3.x86_64
Feb 15 01:32:04 usa7061lv981 yum[8291]: Updated: abrt-addon-ccpp-2.0.8-6.el6_3.2.x86_64
Feb 15 01:32:04 usa7061lv981 yum[8291]: Updated: abrt-addon-python-2.0.8-6.el6_3.2.x86_64
Feb 15 01:32:04 usa7061lv981 yum[8291]: Updated: abrt-tui-2.0.8-6.el6_3.2.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: gnome-keyring-2.28.2-8.el6_3.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: libreport-plugin-kerneloops-2.0.9-5.el6_3.2.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: abrt-addon-kerneloops-2.0.8-6.el6_3.2.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: libreport-plugin-logger-2.0.9-5.el6_3.2.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: libreport-plugin-mailx-2.0.9-5.el6_3.2.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: libreport-cli-2.0.9-5.el6_3.2.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: cyrus-sasl-gssapi-2.1.23-13.el6_3.1.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: cyrus-sasl-md5-2.1.23-13.el6_3.1.x86_64
Feb 15 01:32:06 usa7061lv981 sshd[13404]: rexec line 51: Deprecated option RhostsAuthentication
Feb 15 01:32:06 usa7061lv981 yum[8291]: Updated: libvirt-client-0.9.10-21.el6_3.8.x86_64
Feb 15 01:32:06 usa7061lv981 yum[8291]: Updated: libvirt-java-0.4.7-1.el6_3.1.noarch
Feb 15 01:32:06 usa7061lv981 yum[8291]: Updated: systemtap-runtime-1.7-5.el6_3.1.x86_64
Feb 15 01:32:09 usa7061lv981 yum[8291]: Updated: systemtap-client-1.7-5.el6_3.1.x86_64
Feb 15 01:32:10 usa7061lv981 yum[8291]: Updated: 32:bind-libs-9.8.2-0.10.rc1.el6_3.6.x86_64
Feb 15 01:32:10 usa7061lv981 yum[8291]: Updated: 32:bind-utils-9.8.2-0.10.rc1.el6_3.6.x86_64
Feb 15 01:32:10 usa7061lv981 yum[8291]: Updated: krb5-workstation-1.9-33.el6_3.3.x86_64
Feb 15 01:32:10 usa7061lv981 yum[8291]: Updated: krb5-devel-1.9-33.el6_3.3.x86_64
Feb 15 01:32:13 usa7061lv981 yum[8291]: Updated: openssl-devel-1.0.0-25.el6_3.1.x86_64
Feb 15 01:32:13 usa7061lv981 yum[8291]: Updated: ipa-python-2.2.0-17.el6_3.1.x86_64
Feb 15 01:32:13 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:13 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:13 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:14 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:14 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:14 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:14 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:14 usa7061lv981 yum[8291]: Updated: subscription-manager-0.99.19.4-1.el6_3.x86_64
Feb 15 01:32:14 usa7061lv981 yum[8291]: Updated: subscription-manager-gnome-0.99.19.4-1.el6_3.x86_64
Feb 15 01:32:16 usa7061lv981 yum[8291]: Updated: glibc-headers-2.12-1.80.el6_3.7.x86_64
Feb 15 01:32:16 usa7061lv981 yum[8291]: Updated: device-mapper-multipath-libs-0.4.9-56.el6_3.1.x86_64
Feb 15 01:32:16 usa7061lv981 yum[8291]: Updated: nscd-2.12-1.80.el6_3.7.x86_64
Feb 15 01:32:16 usa7061lv981 yum[8291]: Updated: libtiff-3.9.4-9.el6_3.x86_64
Feb 15 01:32:16 usa7061lv981 yum[8291]: Updated: 1:cups-libs-1.4.2-48.el6_3.3.x86_64
Feb 15 01:32:17 usa7061lv981 yum[8291]: Updated: unixODBC-2.2.14-12.el6_3.x86_64
Feb 15 01:32:17 usa7061lv981 yum[8291]: Updated: unixODBC-devel-2.2.14-12.el6_3.x86_64
Feb 15 01:32:19 usa7061lv981 yum[8291]: Updated: ghostscript-8.70-14.el6_3.1.x86_64
Feb 15 01:32:19 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:19 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:19 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:21 usa7061lv981 yum[8291]: Updated: 1:cups-1.4.2-48.el6_3.3.x86_64
Feb 15 01:32:21 usa7061lv981 yum[8291]: Updated: nss-pam-ldapd-0.7.5-15.el6_3.2.x86_64
Feb 15 01:32:21 usa7061lv981 yum[8291]: Updated: device-mapper-multipath-0.4.9-56.el6_3.1.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: glibc-devel-2.12-1.80.el6_3.7.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: subscription-manager-firstboot-0.99.19.4-1.el6_3.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: ipa-client-2.2.0-17.el6_3.1.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: mysql-devel-5.1.67-1.el6_3.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: systemtap-1.7-5.el6_3.1.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: libvirt-java-devel-0.4.7-1.el6_3.1.noarch
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: libvirt-devel-0.9.10-21.el6_3.8.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: abrt-cli-2.0.8-6.el6_3.2.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: libreport-gtk-2.0.9-5.el6_3.2.x86_64
Feb 15 01:32:23 usa7061lv981 yum[8291]: Updated: openldap-devel-2.4.23-26.el6_3.2.x86_64
Feb 15 01:32:23 usa7061lv981 yum[8291]: Updated: nss-devel-3.13.6-2.el6_3.x86_64
Feb 15 01:32:23 usa7061lv981 yum[8291]: Updated: kexec-tools-2.0.0-245.el6_3.1.x86_64
Feb 15 01:32:24 usa7061lv981 yum[8291]: Updated: 1:net-snmp-5.5-41.el6_3.1.x86_64
Feb 15 01:32:24 usa7061lv981 yum[8291]: Updated: 1:net-snmp-utils-5.5-41.el6_3.1.x86_64
Feb 15 01:32:24 usa7061lv981 yum[8291]: Updated: openssh-server-5.3p1-81.el6_3.x86_64
Feb 15 01:32:25 usa7061lv981 yum[8291]: Updated: 1:autofs-5.0.5-55.el6_3.x86_64
Feb 15 01:32:25 usa7061lv981 yum[8291]: Updated: openldap-clients-2.4.23-26.el6_3.2.x86_64
Feb 15 01:32:26 usa7061lv981 yum[8291]: Updated: mysql-server-5.1.67-1.el6_3.x86_64
Feb 15 01:32:26 usa7061lv981 yum[8291]: Updated: 1:dbus-devel-1.2.24-7.el6_3.x86_64
Feb 15 01:32:28 usa7061lv981 yum[8291]: Updated: xulrunner-10.0.12-1.el6_3.x86_64
Feb 15 01:32:28 usa7061lv981 yum[8291]: Updated: freetype-devel-2.3.11-14.el6_3.1.x86_64
Feb 15 01:32:28 usa7061lv981 yum[8291]: Updated: libreport-newt-2.0.9-5.el6_3.2.x86_64
Feb 15 01:32:28 usa7061lv981 yum[8291]: Updated: pulseaudio-libs-0.9.21-14.el6_3.x86_64
Feb 15 01:32:28 usa7061lv981 yum[8291]: Updated: cyrus-sasl-plain-2.1.23-13.el6_3.1.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: biosdevname-0.3.11-1.el6_3.1.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: openjpeg-libs-1.3-9.el6_3.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: strace-4.5.19-1.11.el6_3.2.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: ftp-0.17-51.1.el6_3.1.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: psacct-6.3.2-63.el6_3.3.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: 2:irqbalance-0.55-35.el6_3.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: ORBit2-2.14.17-3.2.el6_3.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: apr-1.3.9-5.el6_2.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: libxslt-1.1.26-2.el6_3.1.x86_64
Feb 15 01:32:30 usa7061lv981 yum[8291]: Updated: perf-2.6.32-279.19.1.el6.x86_64
Feb 15 01:32:30 usa7061lv981 yum[8291]: Updated: xorg-x11-drv-qxl-0.0.14-14.el6_3.x86_64
Feb 15 01:32:31 usa7061lv981 yum[8291]: Updated: glibc-2.12-1.80.el6_3.7.i686
Feb 15 01:32:37 usa7061lv981 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_programstatus SET instance_id='1', status_update_time=FROM_UNIXTIME(1360909957), program_start_time=FROM_UNIXTIME(1360871543), is_currently_running='1', process_id='1280', daemon_mode='1', last_command_check=FROM_UNIXTIME(1360909957), last_log_rotation=FROM_UNIXTIME(1360904400), notifications_enabled='1', active_service_checks_enabled='1', passive_service_checks_enabled='1', active_host_checks_enabled='1', passive_host_checks_enabled='1', event_handlers_enabled='1', flap_detection_enabled='1', failure_prediction_enabled='1', process_performance_data='1', obsess_over_hosts='0', obsess_over_services='0', modified_host_attributes='0', modified_service_attributes='0', global_host_event_handler='xi_host_event_handler', global_service_event_handler='xi_service_event_handler' ON DUPLICATE KEY UPDATE instance_id='1', status_update_time=FROM_UNIXTIME(1360909957), program_start_time=FROM_UNIXTIME(1360871543), is_currently_running='1', process_id='1280', daemon_mode='1', last_command_check=FROM_UNIXTIME(1360909957), last_log_rotation=FROM_UNIXTIME(1360904400), notifications_enabled='1', active_service_checks_enabled='1', passive_service_checks_enabled='1', active_host_checks_enabled='1', passive_host_checks_enabled='1', event_handlers_enabled='1', flap_detection_enabled='1', failure_prediction_enabled='1', process_performance_data='1', obsess_over_hosts='0', obsess_over_services='0', modified_host_attributes='0', modified_service_attributes='0', global_host_event_handler='xi_host_event_handler', global_service_event_handler='xi_service_event_handler''
Feb 15 01:32:37 usa7061lv981 ndo2db: mysql_error: 'MySQL server has gone away'
Feb 15 01:32:37 usa7061lv981 ndo2db: Error: Connection to MySQL database has been lost!
Feb 15 01:32:40 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 01:32:40 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 01:32:40 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 01:32:40 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 01:32:45 usa7061lv981 kdump: kexec: unloaded kdump kernel
Feb 15 01:32:45 usa7061lv981 kdump: stopped
Feb 15 01:32:45 usa7061lv981 kdump: kexec: failed to load kdump kernel
Feb 15 01:32:45 usa7061lv981 kdump: failed to start up
Feb 15 01:32:47 usa7061lv981 abrtd: Got signal 15, exiting
Any idea on how to fix this, mysqld is starting up and running. I would like to get the expert opinion before I start trying to "fix" things. We can revert, though I would like to know how and why things went wrong. Should yum not be used after installing Nagios?

-Doron

Re: So our data center decided to update our servers

Posted: Fri Feb 15, 2013 11:46 am
by abrist
Have you tried restarting mysqld? Looks like a kexec command was sent to the kernel, was the attempted kernel switch intentional?

Mysqld will error out while yum is updating it as yum will bring the service down to install the new package.

Re: So our data center decided to update our servers

Posted: Fri Feb 15, 2013 12:50 pm
by disrael
Yes I tried restarting it repeatedly as well as postgresql, mysqld, nagios, and httpd.

-Doron

Re: So our data center decided to update our servers

Posted: Fri Feb 15, 2013 12:53 pm
by disrael
Just checked the /var/log/messags and saw this:

Code: Select all

Feb 15 12:49:06 usa7061lv981 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_systemcommands SET instance_id='1', start_time=FROM_UNIXTIME(1360950546), start_time_usec='200314', end_time=FROM_UNIXTIME(0), end_time_usec='0', command_line='/bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/xidpe/1360950546\.perfdata\.host', timeout='5', early_timeout='0', execution_time='0.000000', return_code='0', output='', long_output='' ON DUPLICATE KEY UPDATE instance_id='1', start_time=FROM_UNIXTIME(1360950546), start_time_usec='200314', end_time=FROM_UNIXTIME(0), end_time_usec='0', command_line='/bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/xidpe/1360950546\.perfdata\.host', timeout='5', early_timeout='0', execution_time='0.000000', return_code='0', output='', long_output='''
Feb 15 12:49:06 usa7061lv981 ndo2db: mysql_error: 'MySQL server has gone away'
Feb 15 12:49:06 usa7061lv981 ndo2db: Error: Connection to MySQL database has been lost!
Feb 15 12:49:06 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:06 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:06 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:06 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:06 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:12 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:18 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

Re: So our data center decided to update our servers

Posted: Fri Feb 15, 2013 2:42 pm
by abrist
I want you to check a few things:
Disk space and inode usage (can cause the mysql.sock file to not be created if the disk is full among other things):

Code: Select all

df -h
df -i
Check for the sock file:

Code: Select all

ls -la /var/lib/mysql/mysql.sock
Stop the mysqld service and check the sock file again (it should be missing):

Code: Select all

service stop mysqld
ls -la /var/lib/mysql/mysql.sock
service mysqld start
Check the mysqld log after a restart to make sure it comes up clean:

Code: Select all

tail -50 /var/log/mysqld.log
Check to make sure only 1 instance of a mysqld parent exists:

Code: Select all

ps -aef | grep mysql
Tables could be crashed if mysql did not shut down cleanly before the update:

Code: Select all

service mysqld stop
/usr/local/nagiosxi/scripts/repairmysql.sh nagios
service mysqld start

Re: So our data center decided to update our servers

Posted: Fri Feb 15, 2013 4:22 pm
by disrael
I want you to check a few things:
Disk space and inode usage (can cause the mysql.sock file to not be created if the disk is full among other things):

Code: Select all

df -h
df -i
Space is good, checked this first

Check for the sock file:

Code: Select all

ls -la /var/lib/mysql/mysql.sock
tmproot@usa7061lv981>ls -la /var/lib/mysql/mysql.sock
srwxrwxrwx 1 mysql mysql 0 Feb 15 12:59 /var/lib/mysql/mysql.sock


Stop the mysqld service and check the sock file again (it should be missing):

Code: Select all

service stop mysqld
ls -la /var/lib/mysql/mysql.sock
service mysqld start
ls -la /var/lib/mysql/mysql.sock
ls: cannot access /var/lib/mysql/mysql.sock: No such file or directory

Check the mysqld log after a restart to make sure it comes up clean:

Code: Select all

tail -50 /var/log/mysqld.log

Code: Select all

130215 12:49:19  InnoDB: Completed initialization of buffer pool
130215 12:49:19  InnoDB: Started; log sequence number 0 44233
130215 12:49:19 [Note] Event Scheduler: Loaded 0 events
130215 12:49:19 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
130215 12:54:37 [Note] /usr/libexec/mysqld: Normal shutdown

130215 12:54:37 [Note] Event Scheduler: Purging the queue. 0 events
130215 12:54:37  InnoDB: Starting shutdown...
130215 12:54:39  InnoDB: Shutdown completed; log sequence number 0 44233
130215 12:54:39 [Note] /usr/libexec/mysqld: Shutdown complete

130215 12:54:39 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130215 12:54:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130215 12:54:40  InnoDB: Initializing buffer pool, size = 8.0M
130215 12:54:40  InnoDB: Completed initialization of buffer pool
130215 12:54:40  InnoDB: Started; log sequence number 0 44233
130215 12:54:40 [Note] Event Scheduler: Loaded 0 events
130215 12:54:40 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
130215 12:59:33 [Note] /usr/libexec/mysqld: Normal shutdown

130215 12:59:33 [Note] Event Scheduler: Purging the queue. 0 events
130215 12:59:33  InnoDB: Starting shutdown...
130215 12:59:36  InnoDB: Shutdown completed; log sequence number 0 44233
130215 12:59:36 [Note] /usr/libexec/mysqld: Shutdown complete

130215 12:59:36 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130215 12:59:36 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130215 12:59:36  InnoDB: Initializing buffer pool, size = 8.0M
130215 12:59:36  InnoDB: Completed initialization of buffer pool
130215 12:59:36  InnoDB: Started; log sequence number 0 44233
130215 12:59:36 [Note] Event Scheduler: Loaded 0 events
130215 12:59:36 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
130215 16:18:41 [Note] /usr/libexec/mysqld: Normal shutdown

130215 16:18:41 [Note] Event Scheduler: Purging the queue. 0 events
130215 16:18:41  InnoDB: Starting shutdown...
130215 16:18:45  InnoDB: Shutdown completed; log sequence number 0 44233
130215 16:18:45 [Note] /usr/libexec/mysqld: Shutdown complete

130215 16:18:45 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130215 16:19:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130215 16:19:40  InnoDB: Initializing buffer pool, size = 8.0M
130215 16:19:40  InnoDB: Completed initialization of buffer pool
130215 16:19:40  InnoDB: Started; log sequence number 0 44233
130215 16:19:40 [Note] Event Scheduler: Loaded 0 events
130215 16:19:40 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
[/b]

Check to make sure only 1 instance of a mysqld parent exists:

Code: Select all

ps -aef | grep mysql

Code: Select all

root     28885     1  0 16:19 pts/1    00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql    28987 28885  0 16:19 pts/1    00:00:00 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
root     29237 28524  0 16:21 pts/1    00:00:00 grep mysql
Tables could be crashed if mysql did not shut down cleanly before the update:

Code: Select all

service mysqld stop
/usr/local/nagiosxi/scripts/repairmysql.sh nagios
service mysqld start
Done - issue is still there.

Re: So our data center decided to update our servers

Posted: Fri Feb 15, 2013 5:54 pm
by scottwilkerson
Can you verify that you can connect to the mysql server by running the following

Code: Select all

mysql -pnagiosxi nagios

Re: So our data center decided to update our servers

Posted: Mon Feb 18, 2013 9:22 am
by disrael

Code: Select all

>mysql -pnagiosxi nagios
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 49
Server version: 5.1.67 Source distribution

Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| nagios             |
| nagiosql           |
| test               |
+--------------------+
5 rows in set (0.00 sec)

Re: So our data center decided to update our servers

Posted: Mon Feb 18, 2013 10:15 am
by mguthrie
Lets make sure the credentials for the ndoutils DB user is also correct.

The places you'll want to verify the database credentials is:

/usr/local/nagiosxi/html/config.inc.php

Code: Select all

	"ndoutils" => array(
		"dbtype" => 'mysql',
		"dbserver" => 'localhost',
		"user" => 'ndoutils',
		"pwd" => 'n@gweb',
		"db" => 'nagios',
and /usr/local/nagios/etc/ndo2db.cfg


Unless you've updated the credentials from the default, you can test the connection with the following command.

Code: Select all

mysql -u ndoutils -pn@gweb nagios

Re: So our data center decided to update our servers

Posted: Mon Feb 18, 2013 10:26 am
by disrael
We didn't do anything to nagios, just the DC ran yum and updated a lot of packages. This is an non-internet connected system that we used a custom install created by Nagios support. I think the DC team used a tunnel or something to access yum.

Code: Select all

  "ndoutils" => array(
                "dbtype" => 'mysql',
                "dbserver" => 'localhost',
                "user" => 'ndoutils',
                "pwd" => 'n@gweb',
                "db" => 'nagios',
                "dbmaint" => array(             // variables affecting maintenance of ndoutils db

Code: Select all

>mysql -u ndoutils -pn@gweb nagios
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 74
Server version: 5.1.67 Source distribution

Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| nagios             |
| test               |
+--------------------+
3 rows in set (0.00 sec)
All times are UTC-05:00
Page 1 of 3

Powered by phpBB® Forum Software © phpBB Limited