So our data center decided to update our servers

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
disrael
Posts: 116
Joined: Tue Jun 19, 2012 3:16 pm

So our data center decided to update our servers

Post by disrael »

We had our DC team update:

Code: Select all


Feb 15 01:31:40 usa7061lv981 yum[8291]: Updated: tzdata-2012j-1.el6.noarch
Feb 15 01:31:49 usa7061lv981 yum[8291]: Updated: glibc-2.12-1.80.el6_3.7.x86_64
Feb 15 01:31:56 usa7061lv981 yum[8291]: Updated: glibc-common-2.12-1.80.el6_3.7.x86_64
Feb 15 01:31:57 usa7061lv981 yum[8291]: Updated: krb5-libs-1.9-33.el6_3.3.x86_64
Feb 15 01:31:57 usa7061lv981 yum[8291]: Updated: openssl-1.0.0-25.el6_3.1.x86_64
Feb 15 01:31:57 usa7061lv981 yum[8291]: Updated: nspr-4.9.2-0.el6_3.1.x86_64
Feb 15 01:31:57 usa7061lv981 yum[8291]: Updated: nss-util-3.13.6-1.el6_3.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: nss-3.13.6-2.el6_3.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: nss-sysinit-3.13.6-2.el6_3.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: cyrus-sasl-lib-2.1.23-13.el6_3.1.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: 1:dbus-libs-1.2.24-7.el6_3.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: libreport-plugin-reportuploader-2.0.9-5.el6_3.2.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: libreport-2.0.9-5.el6_3.2.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: libreport-plugin-rhtsupport-2.0.9-5.el6_3.2.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: libreport-python-2.0.9-5.el6_3.2.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: freetype-2.3.11-14.el6_3.1.x86_64
Feb 15 01:31:58 usa7061lv981 yum[8291]: Updated: pango-1.28.1-7.el6_3.x86_64
Feb 15 01:31:59 usa7061lv981 yum[8291]: Updated: abrt-libs-2.0.8-6.el6_3.2.x86_64
Feb 15 01:31:59 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:31:59 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:31:59 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:31:59 usa7061lv981 yum[8291]: Updated: abrt-2.0.8-6.el6_3.2.x86_64
Feb 15 01:31:59 usa7061lv981 yum[8291]: Updated: 1:dbus-1.2.24-7.el6_3.x86_64
Feb 15 01:32:00 usa7061lv981 yum[8291]: Updated: mysql-libs-5.1.67-1.el6_3.x86_64
Feb 15 01:32:00 usa7061lv981 yum[8291]: Updated: mysql-5.1.67-1.el6_3.x86_64
Feb 15 01:32:00 usa7061lv981 yum[8291]: Updated: libproxy-python-0.3.0-3.el6_3.x86_64
Feb 15 01:32:00 usa7061lv981 yum[8291]: Updated: libproxy-0.3.0-3.el6_3.x86_64
Feb 15 01:32:00 usa7061lv981 yum[8291]: Updated: libproxy-bin-0.3.0-3.el6_3.x86_64
Feb 15 01:32:01 usa7061lv981 yum[8291]: Updated: systemtap-devel-1.7-5.el6_3.1.x86_64
Feb 15 01:32:01 usa7061lv981 yum[8291]: Updated: nss-tools-3.13.6-2.el6_3.x86_64
Feb 15 01:32:01 usa7061lv981 yum[8291]: Updated: openldap-2.4.23-26.el6_3.2.x86_64
Feb 15 01:32:01 usa7061lv981 yum[8291]: Updated: cyrus-sasl-2.1.23-13.el6_3.1.x86_64
Feb 15 01:32:01 usa7061lv981 yum[8291]: Updated: openssh-5.3p1-81.el6_3.x86_64
Feb 15 01:32:02 usa7061lv981 yum[8291]: Updated: nspr-devel-4.9.2-0.el6_3.1.x86_64
Feb 15 01:32:03 usa7061lv981 yum[8291]: Updated: 1:net-snmp-libs-5.5-41.el6_3.1.x86_64
Feb 15 01:32:03 usa7061lv981 yum[8291]: Updated: kpartx-0.4.9-56.el6_3.1.x86_64
Feb 15 01:32:03 usa7061lv981 yum[8291]: Updated: nss-util-devel-3.13.6-1.el6_3.x86_64
Feb 15 01:32:03 usa7061lv981 yum[8291]: Updated: openssh-clients-5.3p1-81.el6_3.x86_64
Feb 15 01:32:03 usa7061lv981 yum[8291]: Updated: cyrus-sasl-devel-2.1.23-13.el6_3.1.x86_64
Feb 15 01:32:04 usa7061lv981 yum[8291]: Updated: 1:dbus-x11-1.2.24-7.el6_3.x86_64
Feb 15 01:32:04 usa7061lv981 yum[8291]: Updated: abrt-addon-ccpp-2.0.8-6.el6_3.2.x86_64
Feb 15 01:32:04 usa7061lv981 yum[8291]: Updated: abrt-addon-python-2.0.8-6.el6_3.2.x86_64
Feb 15 01:32:04 usa7061lv981 yum[8291]: Updated: abrt-tui-2.0.8-6.el6_3.2.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: gnome-keyring-2.28.2-8.el6_3.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: libreport-plugin-kerneloops-2.0.9-5.el6_3.2.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: abrt-addon-kerneloops-2.0.8-6.el6_3.2.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: libreport-plugin-logger-2.0.9-5.el6_3.2.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: libreport-plugin-mailx-2.0.9-5.el6_3.2.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: libreport-cli-2.0.9-5.el6_3.2.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: cyrus-sasl-gssapi-2.1.23-13.el6_3.1.x86_64
Feb 15 01:32:05 usa7061lv981 yum[8291]: Updated: cyrus-sasl-md5-2.1.23-13.el6_3.1.x86_64
Feb 15 01:32:06 usa7061lv981 sshd[13404]: rexec line 51: Deprecated option RhostsAuthentication
Feb 15 01:32:06 usa7061lv981 yum[8291]: Updated: libvirt-client-0.9.10-21.el6_3.8.x86_64
Feb 15 01:32:06 usa7061lv981 yum[8291]: Updated: libvirt-java-0.4.7-1.el6_3.1.noarch
Feb 15 01:32:06 usa7061lv981 yum[8291]: Updated: systemtap-runtime-1.7-5.el6_3.1.x86_64
Feb 15 01:32:09 usa7061lv981 yum[8291]: Updated: systemtap-client-1.7-5.el6_3.1.x86_64
Feb 15 01:32:10 usa7061lv981 yum[8291]: Updated: 32:bind-libs-9.8.2-0.10.rc1.el6_3.6.x86_64
Feb 15 01:32:10 usa7061lv981 yum[8291]: Updated: 32:bind-utils-9.8.2-0.10.rc1.el6_3.6.x86_64
Feb 15 01:32:10 usa7061lv981 yum[8291]: Updated: krb5-workstation-1.9-33.el6_3.3.x86_64
Feb 15 01:32:10 usa7061lv981 yum[8291]: Updated: krb5-devel-1.9-33.el6_3.3.x86_64
Feb 15 01:32:13 usa7061lv981 yum[8291]: Updated: openssl-devel-1.0.0-25.el6_3.1.x86_64
Feb 15 01:32:13 usa7061lv981 yum[8291]: Updated: ipa-python-2.2.0-17.el6_3.1.x86_64
Feb 15 01:32:13 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:13 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:13 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:14 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:14 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:14 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:14 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:14 usa7061lv981 yum[8291]: Updated: subscription-manager-0.99.19.4-1.el6_3.x86_64
Feb 15 01:32:14 usa7061lv981 yum[8291]: Updated: subscription-manager-gnome-0.99.19.4-1.el6_3.x86_64
Feb 15 01:32:16 usa7061lv981 yum[8291]: Updated: glibc-headers-2.12-1.80.el6_3.7.x86_64
Feb 15 01:32:16 usa7061lv981 yum[8291]: Updated: device-mapper-multipath-libs-0.4.9-56.el6_3.1.x86_64
Feb 15 01:32:16 usa7061lv981 yum[8291]: Updated: nscd-2.12-1.80.el6_3.7.x86_64
Feb 15 01:32:16 usa7061lv981 yum[8291]: Updated: libtiff-3.9.4-9.el6_3.x86_64
Feb 15 01:32:16 usa7061lv981 yum[8291]: Updated: 1:cups-libs-1.4.2-48.el6_3.3.x86_64
Feb 15 01:32:17 usa7061lv981 yum[8291]: Updated: unixODBC-2.2.14-12.el6_3.x86_64
Feb 15 01:32:17 usa7061lv981 yum[8291]: Updated: unixODBC-devel-2.2.14-12.el6_3.x86_64
Feb 15 01:32:19 usa7061lv981 yum[8291]: Updated: ghostscript-8.70-14.el6_3.1.x86_64
Feb 15 01:32:19 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:19 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:19 usa7061lv981 dbus-daemon: [system] Reloaded configuration
Feb 15 01:32:21 usa7061lv981 yum[8291]: Updated: 1:cups-1.4.2-48.el6_3.3.x86_64
Feb 15 01:32:21 usa7061lv981 yum[8291]: Updated: nss-pam-ldapd-0.7.5-15.el6_3.2.x86_64
Feb 15 01:32:21 usa7061lv981 yum[8291]: Updated: device-mapper-multipath-0.4.9-56.el6_3.1.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: glibc-devel-2.12-1.80.el6_3.7.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: subscription-manager-firstboot-0.99.19.4-1.el6_3.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: ipa-client-2.2.0-17.el6_3.1.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: mysql-devel-5.1.67-1.el6_3.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: systemtap-1.7-5.el6_3.1.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: libvirt-java-devel-0.4.7-1.el6_3.1.noarch
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: libvirt-devel-0.9.10-21.el6_3.8.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: abrt-cli-2.0.8-6.el6_3.2.x86_64
Feb 15 01:32:22 usa7061lv981 yum[8291]: Updated: libreport-gtk-2.0.9-5.el6_3.2.x86_64
Feb 15 01:32:23 usa7061lv981 yum[8291]: Updated: openldap-devel-2.4.23-26.el6_3.2.x86_64
Feb 15 01:32:23 usa7061lv981 yum[8291]: Updated: nss-devel-3.13.6-2.el6_3.x86_64
Feb 15 01:32:23 usa7061lv981 yum[8291]: Updated: kexec-tools-2.0.0-245.el6_3.1.x86_64
Feb 15 01:32:24 usa7061lv981 yum[8291]: Updated: 1:net-snmp-5.5-41.el6_3.1.x86_64
Feb 15 01:32:24 usa7061lv981 yum[8291]: Updated: 1:net-snmp-utils-5.5-41.el6_3.1.x86_64
Feb 15 01:32:24 usa7061lv981 yum[8291]: Updated: openssh-server-5.3p1-81.el6_3.x86_64
Feb 15 01:32:25 usa7061lv981 yum[8291]: Updated: 1:autofs-5.0.5-55.el6_3.x86_64
Feb 15 01:32:25 usa7061lv981 yum[8291]: Updated: openldap-clients-2.4.23-26.el6_3.2.x86_64
Feb 15 01:32:26 usa7061lv981 yum[8291]: Updated: mysql-server-5.1.67-1.el6_3.x86_64
Feb 15 01:32:26 usa7061lv981 yum[8291]: Updated: 1:dbus-devel-1.2.24-7.el6_3.x86_64
Feb 15 01:32:28 usa7061lv981 yum[8291]: Updated: xulrunner-10.0.12-1.el6_3.x86_64
Feb 15 01:32:28 usa7061lv981 yum[8291]: Updated: freetype-devel-2.3.11-14.el6_3.1.x86_64
Feb 15 01:32:28 usa7061lv981 yum[8291]: Updated: libreport-newt-2.0.9-5.el6_3.2.x86_64
Feb 15 01:32:28 usa7061lv981 yum[8291]: Updated: pulseaudio-libs-0.9.21-14.el6_3.x86_64
Feb 15 01:32:28 usa7061lv981 yum[8291]: Updated: cyrus-sasl-plain-2.1.23-13.el6_3.1.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: biosdevname-0.3.11-1.el6_3.1.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: openjpeg-libs-1.3-9.el6_3.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: strace-4.5.19-1.11.el6_3.2.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: ftp-0.17-51.1.el6_3.1.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: psacct-6.3.2-63.el6_3.3.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: 2:irqbalance-0.55-35.el6_3.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: ORBit2-2.14.17-3.2.el6_3.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: apr-1.3.9-5.el6_2.x86_64
Feb 15 01:32:29 usa7061lv981 yum[8291]: Updated: libxslt-1.1.26-2.el6_3.1.x86_64
Feb 15 01:32:30 usa7061lv981 yum[8291]: Updated: perf-2.6.32-279.19.1.el6.x86_64
Feb 15 01:32:30 usa7061lv981 yum[8291]: Updated: xorg-x11-drv-qxl-0.0.14-14.el6_3.x86_64
Feb 15 01:32:31 usa7061lv981 yum[8291]: Updated: glibc-2.12-1.80.el6_3.7.i686
Feb 15 01:32:37 usa7061lv981 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_programstatus SET instance_id='1', status_update_time=FROM_UNIXTIME(1360909957), program_start_time=FROM_UNIXTIME(1360871543), is_currently_running='1', process_id='1280', daemon_mode='1', last_command_check=FROM_UNIXTIME(1360909957), last_log_rotation=FROM_UNIXTIME(1360904400), notifications_enabled='1', active_service_checks_enabled='1', passive_service_checks_enabled='1', active_host_checks_enabled='1', passive_host_checks_enabled='1', event_handlers_enabled='1', flap_detection_enabled='1', failure_prediction_enabled='1', process_performance_data='1', obsess_over_hosts='0', obsess_over_services='0', modified_host_attributes='0', modified_service_attributes='0', global_host_event_handler='xi_host_event_handler', global_service_event_handler='xi_service_event_handler' ON DUPLICATE KEY UPDATE instance_id='1', status_update_time=FROM_UNIXTIME(1360909957), program_start_time=FROM_UNIXTIME(1360871543), is_currently_running='1', process_id='1280', daemon_mode='1', last_command_check=FROM_UNIXTIME(1360909957), last_log_rotation=FROM_UNIXTIME(1360904400), notifications_enabled='1', active_service_checks_enabled='1', passive_service_checks_enabled='1', active_host_checks_enabled='1', passive_host_checks_enabled='1', event_handlers_enabled='1', flap_detection_enabled='1', failure_prediction_enabled='1', process_performance_data='1', obsess_over_hosts='0', obsess_over_services='0', modified_host_attributes='0', modified_service_attributes='0', global_host_event_handler='xi_host_event_handler', global_service_event_handler='xi_service_event_handler''
Feb 15 01:32:37 usa7061lv981 ndo2db: mysql_error: 'MySQL server has gone away'
Feb 15 01:32:37 usa7061lv981 ndo2db: Error: Connection to MySQL database has been lost!
Feb 15 01:32:40 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 01:32:40 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 01:32:40 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 01:32:40 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 01:32:45 usa7061lv981 kdump: kexec: unloaded kdump kernel
Feb 15 01:32:45 usa7061lv981 kdump: stopped
Feb 15 01:32:45 usa7061lv981 kdump: kexec: failed to load kdump kernel
Feb 15 01:32:45 usa7061lv981 kdump: failed to start up
Feb 15 01:32:47 usa7061lv981 abrtd: Got signal 15, exiting
Any idea on how to fix this, mysqld is starting up and running. I would like to get the expert opinion before I start trying to "fix" things. We can revert, though I would like to know how and why things went wrong. Should yum not be used after installing Nagios?

-Doron
Top
abrist
Red Shirt
Posts: 8334
Joined: Thu Nov 15, 2012 1:20 pm

Re: So our data center decided to update our servers

Post by abrist »

Have you tried restarting mysqld? Looks like a kexec command was sent to the kernel, was the attempted kernel switch intentional?

Mysqld will error out while yum is updating it as yum will bring the service down to install the new package.
Former Nagios employee
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
Top
disrael
Posts: 116
Joined: Tue Jun 19, 2012 3:16 pm

Re: So our data center decided to update our servers

Post by disrael »

Yes I tried restarting it repeatedly as well as postgresql, mysqld, nagios, and httpd.

-Doron
Top
disrael
Posts: 116
Joined: Tue Jun 19, 2012 3:16 pm

Re: So our data center decided to update our servers

Post by disrael »

Just checked the /var/log/messags and saw this:

Code: Select all

Feb 15 12:49:06 usa7061lv981 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_systemcommands SET instance_id='1', start_time=FROM_UNIXTIME(1360950546), start_time_usec='200314', end_time=FROM_UNIXTIME(0), end_time_usec='0', command_line='/bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/xidpe/1360950546\.perfdata\.host', timeout='5', early_timeout='0', execution_time='0.000000', return_code='0', output='', long_output='' ON DUPLICATE KEY UPDATE instance_id='1', start_time=FROM_UNIXTIME(1360950546), start_time_usec='200314', end_time=FROM_UNIXTIME(0), end_time_usec='0', command_line='/bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/xidpe/1360950546\.perfdata\.host', timeout='5', early_timeout='0', execution_time='0.000000', return_code='0', output='', long_output='''
Feb 15 12:49:06 usa7061lv981 ndo2db: mysql_error: 'MySQL server has gone away'
Feb 15 12:49:06 usa7061lv981 ndo2db: Error: Connection to MySQL database has been lost!
Feb 15 12:49:06 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:06 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:06 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:06 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:06 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:12 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:16 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Feb 15 12:49:18 usa7061lv981 ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Top
abrist
Red Shirt
Posts: 8334
Joined: Thu Nov 15, 2012 1:20 pm

Re: So our data center decided to update our servers

Post by abrist »

I want you to check a few things:
Disk space and inode usage (can cause the mysql.sock file to not be created if the disk is full among other things):

Code: Select all

df -h
df -i
Check for the sock file:

Code: Select all

ls -la /var/lib/mysql/mysql.sock
Stop the mysqld service and check the sock file again (it should be missing):

Code: Select all

service stop mysqld
ls -la /var/lib/mysql/mysql.sock
service mysqld start
Check the mysqld log after a restart to make sure it comes up clean:

Code: Select all

tail -50 /var/log/mysqld.log
Check to make sure only 1 instance of a mysqld parent exists:

Code: Select all

ps -aef | grep mysql
Tables could be crashed if mysql did not shut down cleanly before the update:

Code: Select all

service mysqld stop
/usr/local/nagiosxi/scripts/repairmysql.sh nagios
service mysqld start
Former Nagios employee
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
Top
disrael
Posts: 116
Joined: Tue Jun 19, 2012 3:16 pm

Re: So our data center decided to update our servers

Post by disrael »

I want you to check a few things:
Disk space and inode usage (can cause the mysql.sock file to not be created if the disk is full among other things):

Code: Select all

df -h
df -i
Space is good, checked this first

Check for the sock file:

Code: Select all

ls -la /var/lib/mysql/mysql.sock
tmproot@usa7061lv981>ls -la /var/lib/mysql/mysql.sock
srwxrwxrwx 1 mysql mysql 0 Feb 15 12:59 /var/lib/mysql/mysql.sock


Stop the mysqld service and check the sock file again (it should be missing):

Code: Select all

service stop mysqld
ls -la /var/lib/mysql/mysql.sock
service mysqld start
ls -la /var/lib/mysql/mysql.sock
ls: cannot access /var/lib/mysql/mysql.sock: No such file or directory

Check the mysqld log after a restart to make sure it comes up clean:

Code: Select all

tail -50 /var/log/mysqld.log

Code: Select all

130215 12:49:19  InnoDB: Completed initialization of buffer pool
130215 12:49:19  InnoDB: Started; log sequence number 0 44233
130215 12:49:19 [Note] Event Scheduler: Loaded 0 events
130215 12:49:19 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
130215 12:54:37 [Note] /usr/libexec/mysqld: Normal shutdown

130215 12:54:37 [Note] Event Scheduler: Purging the queue. 0 events
130215 12:54:37  InnoDB: Starting shutdown...
130215 12:54:39  InnoDB: Shutdown completed; log sequence number 0 44233
130215 12:54:39 [Note] /usr/libexec/mysqld: Shutdown complete

130215 12:54:39 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130215 12:54:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130215 12:54:40  InnoDB: Initializing buffer pool, size = 8.0M
130215 12:54:40  InnoDB: Completed initialization of buffer pool
130215 12:54:40  InnoDB: Started; log sequence number 0 44233
130215 12:54:40 [Note] Event Scheduler: Loaded 0 events
130215 12:54:40 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
130215 12:59:33 [Note] /usr/libexec/mysqld: Normal shutdown

130215 12:59:33 [Note] Event Scheduler: Purging the queue. 0 events
130215 12:59:33  InnoDB: Starting shutdown...
130215 12:59:36  InnoDB: Shutdown completed; log sequence number 0 44233
130215 12:59:36 [Note] /usr/libexec/mysqld: Shutdown complete

130215 12:59:36 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130215 12:59:36 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130215 12:59:36  InnoDB: Initializing buffer pool, size = 8.0M
130215 12:59:36  InnoDB: Completed initialization of buffer pool
130215 12:59:36  InnoDB: Started; log sequence number 0 44233
130215 12:59:36 [Note] Event Scheduler: Loaded 0 events
130215 12:59:36 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
130215 16:18:41 [Note] /usr/libexec/mysqld: Normal shutdown

130215 16:18:41 [Note] Event Scheduler: Purging the queue. 0 events
130215 16:18:41  InnoDB: Starting shutdown...
130215 16:18:45  InnoDB: Shutdown completed; log sequence number 0 44233
130215 16:18:45 [Note] /usr/libexec/mysqld: Shutdown complete

130215 16:18:45 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130215 16:19:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130215 16:19:40  InnoDB: Initializing buffer pool, size = 8.0M
130215 16:19:40  InnoDB: Completed initialization of buffer pool
130215 16:19:40  InnoDB: Started; log sequence number 0 44233
130215 16:19:40 [Note] Event Scheduler: Loaded 0 events
130215 16:19:40 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
[/b]

Check to make sure only 1 instance of a mysqld parent exists:

Code: Select all

ps -aef | grep mysql

Code: Select all

root     28885     1  0 16:19 pts/1    00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql    28987 28885  0 16:19 pts/1    00:00:00 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
root     29237 28524  0 16:21 pts/1    00:00:00 grep mysql
Tables could be crashed if mysql did not shut down cleanly before the update:

Code: Select all

service mysqld stop
/usr/local/nagiosxi/scripts/repairmysql.sh nagios
service mysqld start
Done - issue is still there.
Top
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:
Contact scottwilkerson

Re: So our data center decided to update our servers

Post by scottwilkerson »

Can you verify that you can connect to the mysql server by running the following

Code: Select all

mysql -pnagiosxi nagios
Former Nagios employee
Creator:
Human Design Website
Get Your Human Design Chart
Top
disrael
Posts: 116
Joined: Tue Jun 19, 2012 3:16 pm

Re: So our data center decided to update our servers

Post by disrael »

Code: Select all

>mysql -pnagiosxi nagios
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 49
Server version: 5.1.67 Source distribution

Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| nagios             |
| nagiosql           |
| test               |
+--------------------+
5 rows in set (0.00 sec)
Top
mguthrie
Posts: 4380
Joined: Mon Jun 14, 2010 10:21 am

Re: So our data center decided to update our servers

Post by mguthrie »

Lets make sure the credentials for the ndoutils DB user is also correct.

The places you'll want to verify the database credentials is:

/usr/local/nagiosxi/html/config.inc.php

Code: Select all

	"ndoutils" => array(
		"dbtype" => 'mysql',
		"dbserver" => 'localhost',
		"user" => 'ndoutils',
		"pwd" => 'n@gweb',
		"db" => 'nagios',
and /usr/local/nagios/etc/ndo2db.cfg


Unless you've updated the credentials from the default, you can test the connection with the following command.

Code: Select all

mysql -u ndoutils -pn@gweb nagios
Top
disrael
Posts: 116
Joined: Tue Jun 19, 2012 3:16 pm

Re: So our data center decided to update our servers

Post by disrael »

We didn't do anything to nagios, just the DC ran yum and updated a lot of packages. This is an non-internet connected system that we used a custom install created by Nagios support. I think the DC team used a tunnel or something to access yum.

Code: Select all

  "ndoutils" => array(
                "dbtype" => 'mysql',
                "dbserver" => 'localhost',
                "user" => 'ndoutils',
                "pwd" => 'n@gweb',
                "db" => 'nagios',
                "dbmaint" => array(             // variables affecting maintenance of ndoutils db

Code: Select all

>mysql -u ndoutils -pn@gweb nagios
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 74
Server version: 5.1.67 Source distribution

Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| nagios             |
| test               |
+--------------------+
3 rows in set (0.00 sec)
Top
Locked

Return to “Nagios XI”