Nagios database seems hosed

[tgriep]

Do the services and hosts show up in Core Config Manager?

[snapon_admin]

Yes.

[jdalrymple]

Code: Select all

[jdalrymple@localhost scripts]$ mysql -u root -pnagiosxi
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8232
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

To see if you've changed the password.

[snapon_admin]

Nope, doesn't appear to be changed:

Code: Select all

[root@keno-ngos-01-pv nagios]# mysql -u root -pnagiosxi
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3401
Server version: 5.1.71 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

[tgriep]

Run the following tail command and apply the configuration and post the output of the tail command.

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

[snapon_admin]

Code: Select all

[root@keno-ngos-01-pv ~]# tail -f /usr/local/nagiosxi/var/cmdsubsys.log
..........................
PROCESSED 0 COMMANDS
MD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
URL: https://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-08-31 15:50:36--  https://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:443... connected.
WARNING: cannot verify localhost's certificate, issued by `/C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA':
  Unable to locally verify the issuer's authority.
WARNING: certificate common name `*.snapon.com' doesn't match requested host name `localhost'.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `nagiosql.login'

     0K ..........                                              649K=0.02s

2015-08-31 15:50:36 (649 KB/s) - `nagiosql.login' saved [10904]

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies https://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=435vi4' -O nagiosql.loginLOGIN SUCCESSFUL!
IMPORTING CONFIG FILES...URL: https://localhost/nagiosxi/includes/components/ccm/
Array
(
)
RESETTING PERMS
SETUID ROOT OK
URL: https://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-08-31 15:50:39--  https://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:443... connected.
WARNING: cannot verify localhost's certificate, issued by `/C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA':
  Unable to locally verify the issuer's authority.
WARNING: certificate common name `*.snapon.com' doesn't match requested host name `localhost'.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `nagiosql.login'

     0K ..........                                              720K=0.01s

2015-08-31 15:50:40 (720 KB/s) - `nagiosql.login' saved [10904]

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies https://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=435vi4' -O nagiosql.loginLOGIN SUCCESSFUL!
URL: https://localhost/nagiosxi/includes/components/ccm/
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies https://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'cmd=apply&type=writeConfig' -O nagiosql.export.monitoring
--2015-08-31 15:50:40--  https://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:443... connected.
WARNING: cannot verify localhost's certificate, issued by `/C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA':
  Unable to locally verify the issuer's authority.
WARNING: certificate common name `*.snapon.com' doesn't match requested host name `localhost'.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `nagiosql.export.monitoring'

     0K .......... ...                                         8.28K=1.7s

2015-08-31 15:50:42 (8.28 KB/s) - `nagiosql.export.monitoring' saved [14317]

WRITE CONFIGS SUCCESSFUL!
OUTPUT: 
Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Warning: Service 'Zone CPU usage' on host 'kenachr01t on kentest12g' has no default contacts or contactgroups defined!
Warning: Service '/ Disk Usage' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service '/vendor/ctmagent7 Disk Usage' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service '/vendor/ctmlogs Disk Usage' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service '/vendor/nagios Disk Usage' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service '/workspace Disk Usage' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'Ping' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'Total Processes' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'Users' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'Zone CPU usage' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'Zone Memory usage' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'autofs process' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'autofs service' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'bind process' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'bind service' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'ctmagent process' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'ctmagent7 service' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'gss service' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'inetd process' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'inetd service' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'ldap client service' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'ldap process' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'local filesystem service' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'name-service-cache service' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'sendmail process' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'sendmail-client service' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'ssh process' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'ssh service' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'system-log service' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Service 'utmp service' on host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
        Checked 3825 services.
Warning: Host 'KenoshaWI-Core' has no default contacts or contactgroups defined!
Warning: Host 'LisleIL-Core' has no default contacts or contactgroups defined!
Warning: Host 'LisleIL-MPLS-1' has no default contacts or contactgroups defined!
Warning: Host 'ken01' has no default contacts or contactgroups defined!
Warning: Host 'ken01p' has no default contacts or contactgroups defined!
Warning: Host 'ken04' has no default contacts or contactgroups defined!
Warning: Host 'ken06p' has no default contacts or contactgroups defined!
Warning: Host 'ken10p' has no default contacts or contactgroups defined!
Warning: Host 'ken17p' has no default contacts or contactgroups defined!
Warning: Host 'ken19' has no default contacts or contactgroups defined!
Warning: Host 'ken23' has no default contacts or contactgroups defined!
Warning: Host 'ken23p' has no default contacts or contactgroups defined!
Warning: Host 'ken24' has no default contacts or contactgroups defined!
Warning: Host 'ken24p' has no default contacts or contactgroups defined!
Warning: Host 'ken9990' has no default contacts or contactgroups defined!
Warning: Host 'kendbms12t on kentest07g' has no default contacts or contactgroups defined!
Warning: Host 'kenibld01p' has no default contacts or contactgroups defined!
        Checked 219 hosts.
        Checked 85 host groups.
        Checked 3 service groups.
        Checked 158 contacts.
        Checked 82 contact groups.
        Checked 194 commands.
        Checked 162 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 219 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 162 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 47
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
OUTPUT=Starting nagios: done.
RETURNCODE=0
.................^C

[tgriep]

Funny that the mysql log file is clean but you are still seeing the errors in the GUI.
Let's restart everything and try again.

Code: Select all

    service nagios stop
    service npcd stop
    service ndo2db stop
    service nagios stop
    killall -9 nagios
    service cron stop
    service httpd restart
    service mysqld restart
    service ndo2db start
    service npcd start
    service nagios start
    service cron start

Wait a few minutes and login. Post any errors.

[snapon_admin]

Nope still borked.

DB Borked.png

[tgriep]

Before running the following command to repair the corrupt table, make sure you have a backup of the system.
This repair command could do more damage than good, so be warned.

Code: Select all

echo 'repair table nagios_hoststatus use_frm;' | mysql  -t -u root -pnagiosxi nagios

[snapon_admin]

Oh jeez, that sounds...interesting. K, config backup from last week is there as well as one from a couple days ago. I also had our VM admin take a snapshot, so here goes nothing...

hold onto your butts.jpg