So our data center decided to update our servers

[scottwilkerson]

I think the DC team used a tunnel or something to access yum.

Do you know if they left something in place that could be redirecting localhost traffic?

Can you run

Code: Select all

cat /etc/hosts|grep localhost

also, can you attach your /usr/local/nagios/etc/ndo2db.cfg

thanks

[disrael]

I don't think so

Code: Select all

cat /etc/hosts|grep localhost
127.0.0.1       localhost.localdomain   localhost

Code: Select all

cat /usr/local/nagios/etc/ndo2db.cfg
#####################################################################
# NDO2DB DAEMON CONFIG FILE
#####################################################################


lock_file=/usr/local/nagios/var/ndo2db.lock

ndo2db_user=nagios
ndo2db_group=nagios

socket_type=unix

socket_name=/usr/local/nagios/var/ndo.sock

tcp_port=5668


db_servertype=mysql
db_host=localhost
db_port=3306

db_name=nagios
db_prefix=nagios_

db_user=ndoutils
db_pass=n@gweb



## TABLE TRIMMING OPTIONS
# Several database tables containing Nagios event data can become quite large
# over time.  Most admins will want to trim these tables and keep only a
# certain amount of data in them.  The options below are used to specify the
# age (in MINUTES) that data should be allowd to remain in various tables
# before it is deleted.  Using a value of zero (0) for any value means that
# that particular table should NOT be automatically trimmed.

# Keep timed events for 24 hours
max_timedevents_age=1440

# Keep system commands for 1 week
max_systemcommands_age=10080

# Keep service checks for 1 week
max_servicechecks_age=10080

# Keep host checks for 1 week
max_hostchecks_age=10080

# Keep event handlers for 31 days
max_eventhandlers_age=44640





# DEBUG LEVEL
# This option determines how much (if any) debugging information will
# be written to the debug file.  OR values together to log multiple
# types of information.
# Values: -1 = Everything
#          0 = Nothing
#          1 = Process info
#          2 = SQL queries

debug_level=0



# DEBUG VERBOSITY
# This option determines how verbose the debug log out will be.
# Values: 0 = Brief output
#         1 = More detailed
#         2 = Very detailed

debug_verbosity=1



# DEBUG FILE
# This option determines where the daemon should write debugging information.

debug_file=/usr/local/nagios/var/ndo2db.debug



# MAX DEBUG FILE SIZE
# This option determines the maximum size (in bytes) of the debug file.  If
# the file grows larger than this size, it will be renamed with a .old
# extension.  If a file already exists with a .old extension it will
# automatically be deleted.  This helps ensure your disk space usage doesn't
# get out of control when debugging.

max_debug_file_size=1000000

[lmiltchev]

See if restarting ndo2db will solve your issue:

Code: Select all

service ndo2db restart

[disrael]

Code: Select all

service ndo2db restart
Stopping ndo2db: head: cannot open `/usr/local/nagios/var/ndo2db.lock' for reading: No such file or directory
done.
Starting ndo2db: done.

[disrael]

I also see the error in /var/log/messages: eb 18 13:07:01 usa7061lv981 automount[5706]: key "nagios" not found in map source(s).

[disrael]

I see some new error messagews from ndo2db in /var/log/messages after I tried running ./fullinstall -o again. I saw it removed a lot of packages.

Code: Select all

pel-release RPM installed OK
Removing rpm php-pdo-5.3.3-14.el6_3.x86_64 from install
Removing rpm fontconfig-devel-2.8.0-3.el6.x86_64 from install
Removing rpm perl-Parse-RecDescent-1.965-1.el6.noarch from install
Removing rpm libXaw-1.0.6-4.1.el6.x86_64 from install
Removing rpm rrdtool-perl-1.3.8-6.el6.x86_64 from install
Removing rpm perl-Net-SMTP-TLS-0.12-1.el6.rf.noarch from install
Removing rpm php-pgsql-5.3.3-14.el6_3.x86_64 from install
Removing rpm php-xml-5.3.3-14.el6_3.x86_64 from install
Removing rpm libXau-devel-1.0.5-1.el6.x86_64 from install
Removing rpm libXpm-devel-3.5.8-2.el6.x86_64 from install
Removing rpm perl-DBD-MySQL-4.013-3.el6.x86_64 from install
Removing rpm dejavu-sans-mono-fonts-2.30-2.el6.noarch from install
Removing rpm perl-DateTime-0.5300-1.el6.x86_64 from install
Removing rpm php-gd-5.3.3-14.el6_3.x86_64 from install
Removing rpm fping-3.4-1.el6.rf.x86_64 from install
Removing rpm php-common-5.3.3-14.el6_3.x86_64 from install
Removing rpm postgresql-8.4.13-1.el6_3.x86_64 from install
Removing rpm perl-Net-SSLeay-1.35-9.el6.x86_64 from install
Removing rpm libmcrypt-2.5.8-9.el6.x86_64 from install
Removing rpm libxcb-devel-1.5-1.el6.x86_64 from install
Removing rpm perl-SNMP_Session-1.12-4.el6.noarch from install
Removing rpm perl-Net-LibIDN-0.12-3.el6.x86_64 from install
Removing rpm perl-Crypt-DES-2.05-9.el6.x86_64 from install
Removing rpm libjpeg-devel-6b-46.el6.x86_64 from install
Removing rpm dejavu-lgc-sans-mono-fonts-2.30-2.el6.noarch from install
Removing rpm rpmdevtools-7.5-1.el6.noarch from install
Removing rpm perl-Net-SNMP-5.2.0-4.el6.noarch from install
Removing rpm mrtg-2.16.2-5.el6.x86_64 from install
Removing rpm perl-Net-DNS-0.65-4.el6.x86_64 from install
Removing rpm php-ldap-5.3.3-14.el6_3.x86_64 from install
Removing rpm perl-XML-Simple-2.18-6.el6.noarch from install
Removing rpm expat-devel-2.0.1-11.el6_2.x86_64 from install
Removing rpm pymssql-1.0.2-4.el6.x86_64 from install
Removing rpm postgresql-libs-8.4.13-1.el6_3.x86_64 from install
Removing rpm php-cli-5.3.3-14.el6_3.x86_64 from install
Removing rpm perl-Digest-SHA1-2.12-2.el6.x86_64 from install
Removing rpm libXdmcp-devel-1.0.3-1.el6.x86_64 from install
Removing rpm mrtg-libs-2.16.2-5.el6.x86_64 from install
Removing rpm perl-IO-Socket-SSL-1.31-2.el6.noarch from install
Removing rpm perl-Class-Singleton-1.4-6.el6.noarch from install
Removing rpm fakeroot-libs-1.12.2-22.2.el6.x86_64 from install
Removing rpm rrdtool-1.3.8-6.el6.x86_64 from install
Removing rpm gd-devel-2.0.35-10.el6.x86_64 from install
Removing rpm libmcrypt-devel-2.5.8-9.el6.x86_64 from install
Removing rpm php-mysql-5.3.3-14.el6_3.x86_64 from install
Removing rpm postgresql-devel-8.4.13-1.el6_3.x86_64 from install
Removing rpm php-snmp-5.3.3-14.el6_3.x86_64 from install
Removing rpm perl-Mail-POP3Client-2.18-6.el6.noarch from install
Removing rpm php-mssql-5.3.3-1.el6.x86_64 from install
Removing rpm xorg-x11-proto-devel-7.6-13.el6.noarch from install
Removing rpm perl-Digest-HMAC-1.01-22.el6.noarch from install
Removing rpm libX11-devel-1.3-2.el6.x86_64 from install
Removing rpm perl-List-MoreUtils-0.22-10.el6.x86_64 from install
Removing rpm fakeroot-1.12.2-22.2.el6.x86_64 from install
Removing rpm graphviz-2.26.0-7.el6.x86_64 from install
Removing rpm php-mcrypt-5.3.3-1.el6.x86_64 from install
Removing rpm postgresql-server-8.4.13-1.el6_3.x86_64 from install
Removing rpm nmap-5.51-2.el6.x86_64 from install
Removing rpm perl-Socket6-0.23-3.el6.x86_64 from install
Removing rpm perl-IO-Socket-INET6-2.56-4.el6.noarch from install
Removing rpm perl-Params-Validate-0.92-3.el6.x86_64 from install
Removing rpm libpng-devel-1.2.49-1.el6_2.x86_64 from install
Removing rpm rrdtool-python-1.3.8-6.el6.x86_64 from install
Removing rpm perl-Mail-IMAPClient-3.32-1.el6.noarch from install
Removing rpm php-5.3.3-14.el6_3.x86_64 from install
Removing rpm xinetd-2.3.14-35.el6_3.x86_64 from install

Code: Select all

Feb 18 13:22:07 usa7061lv981 nagios: ndomod: Error writing to data sink!  Some output may get lost...
Feb 18 13:22:07 usa7061lv981 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
Feb 18 13:22:07 usa7061lv981 nagios: Successfully shutdown... (PID=9947)
Feb 18 13:22:07 usa7061lv981 nagios: ndomod: Shutdown complete.
Feb 18 13:22:07 usa7061lv981 nagios: Event broker module '/usr/local/nagios/bin/ndomod.o' deinitialized successfully.
Feb 18 13:22:07 usa7061lv981 nagios: Nagios 3.4.1 starting... (PID=15904)
Feb 18 13:22:07 usa7061lv981 nagios: Local time is Mon Feb 18 13:22:07 EST 2013
Feb 18 13:22:07 usa7061lv981 nagios: LOG VERSION: 2.0
Feb 18 13:22:07 usa7061lv981 nagios: ndomod: NDOMOD 1.5.1 (05-15-2012) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
Feb 18 13:22:07 usa7061lv981 nagios: ndomod: Successfully connected to data sink.  8 queued items to flush.
Feb 18 13:22:07 usa7061lv981 nagios: ndomod: Successfully flushed 8 queued items to data sink.
Feb 18 13:22:07 usa7061lv981 nagios: Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
Feb 18 13:22:07 usa7061lv981 nagios: Finished daemonizing... (New PID=15908)
Feb 18 13:22:07 usa7061lv981 ndo2db: Error: queue send error, retrying...
Feb 18 13:22:08 usa7061lv981 ndo2db: Message sent to queue
Feb 18 13:22:08 usa7061lv981 ndo2db: Error: queue send error, retrying...
Feb 18 13:22:09 usa7061lv981 ndo2db: Message sent to queue
Feb 18 13:22:09 usa7061lv981 ndo2db: Error: queue send error, retrying...
Feb 18 13:22:10 usa7061lv981 ndo2db: Message sent to queue
Feb 18 13:22:10 usa7061lv981 ndo2db: Error: queue send error, retrying...
Feb 18 13:22:11 usa7061lv981 ndo2db: Message sent to queue
Feb 18 13:23:01 usa7061lv981 automount[5706]: key "nagios" not found in map source(s).
Feb 18 13:23:38 usa7061lv981 nagios: Caught SIGTERM, shutting down...
Feb 18 13:23:38 usa7061lv981 nagios: ndomod: Error writing to data sink!  Some output may get lost...
Feb 18 13:23:38 usa7061lv981 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
Feb 18 13:23:38 usa7061lv981 nagios: Successfully shutdown... (PID=15908)
Feb 18 13:23:38 usa7061lv981 nagios: ndomod: Shutdown complete.
Feb 18 13:23:38 usa7061lv981 nagios: Event broker module '/usr/local/nagios/bin/ndomod.o' deinitialized successfully.
Feb 18 13:23:38 usa7061lv981 nagios: Nagios 3.4.1 starting... (PID=16657)
Feb 18 13:23:38 usa7061lv981 nagios: Local time is Mon Feb 18 13:23:38 EST 2013
Feb 18 13:23:38 usa7061lv981 nagios: LOG VERSION: 2.0
Feb 18 13:23:38 usa7061lv981 nagios: ndomod: NDOMOD 1.5.1 (05-15-2012) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
Feb 18 13:23:38 usa7061lv981 nagios: ndomod: Successfully connected to data sink.  8 queued items to flush.
Feb 18 13:23:38 usa7061lv981 nagios: ndomod: Successfully flushed 8 queued items to data sink.
Feb 18 13:23:38 usa7061lv981 nagios: Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
Feb 18 13:23:38 usa7061lv981 nagios: Finished daemonizing... (New PID=16661)
Feb 18 13:23:38 usa7061lv981 ndo2db: Error: queue send error, retrying...
Feb 18 13:23:39 usa7061lv981 ndo2db: Message sent to queue
Feb 18 13:23:39 usa7061lv981 ndo2db: Error: queue send error, retrying...
Feb 18 13:23:40 usa7061lv981 ndo2db: Message sent to queue
Feb 18 13:23:40 usa7061lv981 ndo2db: Error: queue send error, retrying...
Feb 18 13:23:41 usa7061lv981 ndo2db: Message sent to queue
Feb 18 13:23:41 usa7061lv981 ndo2db: Error: queue send error, retrying...
Feb 18 13:23:42 usa7061lv981 ndo2db: Message sent to queue
Feb 18 13:24:21 usa7061lv981 sshd[16875]: rexec line 51: Deprecated option RhostsAuthentication
Feb 18 13:25:01 usa7061lv981 automount[5706]: key "nagios" not found in map source(s).
Feb 18 13:25:29 usa7061lv981 rsyslogd-2177: imuxsock begins to drop messages from pid 11795 due to rate-limiting

[scottwilkerson]

Doran,

after I tried running ./fullinstall -o again.

Unfortunately, I do not have a fix for this... You should never run fullinstall on a system already running Nagios XI...

The only way I can see out of this would be to take your most recent backup move it off of the server, reinstall a fresh machine and then restore the backup.

[disrael]

Doran,

after I tried running ./fullinstall -o again.

Unfortunately, I do not have a fix for this... You should never run fullinstall on a system already running Nagios XI...

The only way I can see out of this would be to take your most recent backup move it off of the server, reinstall a fresh machine and then restore the backup.

We still have the same issue on a different server. I can have this one re-imaged, though i would like to know why it is doing what it is so we don't run into this issue when we upgrade via yum. Are we even supposed to do upgrade via yum?

[abrist]

Updating through yum should be reasonably safe, of course, it is always best to attempt the updates on a test box first. Concerning the second server: have you verified that all the previous attempts/suggestions in this thread result in the same behavior on the second server?

[disrael]

I tried everything you suggested and got all the same results as the other server.