Nagios suddenly (!) just shows 2 hosts/services!

[divB]

Hey all,

I just needed to rub my eyes in disbelief. I have nagios3 running ob Debian for 7 years without any problems!

Today, SUDDENLY without doing anything, nagios shows only 2 hosts and its services.
The others are visible when I select "View Host Status Detail For All Host Groups" and then every item has "No matching services".

I did one minor change in the configuration today (changed a timeout) but to make sure, I grabbed the completed /etc/nagios3 from last weeks backup and restored it completely. Still!

It is SO UNBELIEVABLE!!

Above all, I can't find any difference between the two hosts that work and the rest which used to work.

The log does not state anything useful:

Code: Select all

[1392802961] Nagios 3.4.1 starting... (PID=24677)
[1392802961] Local time is Wed Feb 19 10:42:41 CET 2014
[1392802961] LOG VERSION: 2.0
[1392802961] Warning: Ignoring lame service dependency (config file '/etc/nagios3/conf.d/private/server.cfg', line 8)
[1392802961] Warning: Ignoring lame service dependency (config file '/etc/nagios3/conf.d/private/intern.cfg', line 10)
[1392802961] Warning: Ignoring lame service dependency (config file '/etc/nagios3/conf.d/private/web.cfg', line 17)
[1392802961] Warning: Ignoring lame service dependency (config file '/etc/nagios3/conf.d/private/ns.cfg', line 17)
[1392802961] Warning: Ignoring lame service dependency (config file '/etc/nagios3/conf.d/private/mail.cfg', line 17)
[1392802961] Warning: Ignoring lame service dependency (config file '/etc/nagios3/conf.d/private/mail.cfg', line 30)
[1392802961] Warning: Ignoring lame service dependency (config file '/etc/nagios3/conf.d/private/core.cfg', line 10)
[1392802961] Warning: Ignoring lame service dependency (config file '/etc/nagios3/conf.d/private/gate.cfg', line 17)
[1392802961] Warning: Ignoring lame service dependency (config file '/etc/nagios3/conf.d/private/gate.cfg', line 30)
[1392802961] Warning: Host 'ap' has no services associated with it!
[1392802961] Warning: Host 'attx' has no services associated with it!
[1392802961] Warning: Host 'pwsw01' has no services associated with it!
[1392802961] Warning: Host 'pwsw02' has no services associated with it!
[1392802961] Warning: Host 'modem' has no services associated with it!
[1392802961] Warning: Host 'office-switch' has no services associated with it!
[1392802961] Warning: Host 'pi' has no services associated with it!
[1392802961] Finished daemonizing... (New PID=24678)
[1392802961] SERVICE FLAPPING ALERT: wireless;Softwareupdates;STARTED; Service appears to have started flapping (25.6% change >= 20.0% threshold)
[1392802961] SERVICE FLAPPING ALERT: intern;Softwareupdates;STARTED; Service appears to have started flapping (24.8% change >= 20.0% threshold)
[1392802961] SERVICE FLAPPING ALERT: server;Softwareupdates;STARTED; Service appears to have started flapping (20.7% change >= 20.0% threshold)

Apart from the fact that I cannot explain myself the "lame service dependency" (I don't think they're lame), all hosts have services associated and the ones which issue a warning here are not the ones I am looking for. For example, intern and server are not displayed in the web interface, yet, the issue a warning about a flapping state. Contradicting.

Any ideas?

Thanks

EDIT: I tried removing /var/cache/nagios3, /var/lib/nagios3 -> no change! Also the hint from the FAQ that multiple instances are running is wrong. Only one instance is (and was) running at a time.

[sreinhardt]

Your "lame" errors for services or hosts are actually getting this most likely due to missing configuration options. Are you the only person with access to this machine? Has nagios been restarted recently? Are the modified dates on your files very recent? Do you use any form of configuration management that might have overwritten them(puppet\chef style)?

[divB]

Thanks,

Yes, I am the one with access to the server (it's my private machine).
I have no version management.
I think I haven't changed anything in my configuration for 1 or 2 years. It just worked.
Of course, the process is "regularly" restarted.
Yesterday, it still worked but suddenly, without warning, these phenomena appeared.
The one thing I did yesterday is that I changed two timeouts (of check_apt and ping). I verified this by restoring last weeks backup and creating a diff -urN of /etc/nagios3. To be very sure, I even restored the old configuration.

Yes, at least the following time stamps are up to date (date of current nagios start):
/var/lib/nagios3/retention.dat
/var/cache/nagios3/status.dat
/var/cache/nagios3/objects.cache

The last file even contains all hosts and services!

This is so incredible!

Thanks...

[divB]

Also regarding the "lame services": I need to re-emphasize that this setup always worked for maybe 5 years now. Still it might be that something is wrong here causing nagios3 just having problems with this config now for some reason? In any case, I just don't see any issue ...

So all "lame" service dependencies refer to the same dependency which exists in the eqivalent form for all of those hosts. E.g.:

Code: Select all

define servicedependency{
        use                             internet
        host_name                       server
        service_description             Softwareupdates
}

The "master" template is defined as follows:

Code: Select all

define servicedependency{
        name                            internet
        dependent_host_name             internet
        dependent_service_description   PING
        register                        0
}

The host "internet" is defined as follows:

Code: Select all

define host{
        use                             generic-host
        host_name                       internet
        alias                           Internet (google)
        address                         173.194.35.184

        parents                         modem
}

It is tied to the service "PING" using the hostgroup "ping-servers". This is from the default nagios3 configuration!

Code: Select all

hostgroup_name  ping-servers
      alias           Pingable servers
      members         internet
}

define service {
        hostgroup_name                  ping-servers
        service_description             PING
        check_command                   check_ping!100.0,20%!1000.0,60%
        use                             generic-service
        notification_interval           0 ; set > 0 if you want to be renotified
}

[slansing]

Can you run a config verification and post it's output here? Thanks!

[divB]

Thanks!

The log output is not essentially different from the part I posted (I sent a copy via PM).

I just want to add yet another curiosity: I am getting notifications for one host which is not showing up in the web interface:

Code: Select all

***** Nagios *****

Notification Type: PROBLEM

Service: PING
Host: Internet (google)
Address: 173.194.35.184
State: WARNING

Date/Time: Fri Feb 21 06:44:29 CET 2014

Additional Info:

PING WARNING - Packet loss = 0%, RTA = 114.44 ms

[slansing]

Hmm, yeah the PM didn't help much, have you tried adding criteria lines?:

http://nagios.sourceforge.net/docs/3_0/ ... ncies.html

[divB]

I know these 'lame' messages are irritating.

But to make sure I removed all servicedependency objects from my config file. No lame messages any more.

But the problem is exactly the same! Only 2 hosts in the webinterface!
Log output is now exactly the same except the for the lame servicedependency

Thanks
divB

[divB]

OH MY GOD WHAT A PROBLEM!!

I have a single SSL domain and signed in with http auth for a different service. This wrong credentials were sent to nagios!

-> solved