Page 1 of 1

New hosts

Posted: Tue Oct 06, 2015 6:44 am
by sspaise
Hi All,

Quick question,

I'm currently writing an auto check in script, and would like to use some of the passive results to check that the host is who they say they are. As I'm sure you know when a new check is sent to Nagios, but the host configuration doesn't exist yet you receive the following in the nagios.log:

[1444131379] Warning: Passive check result was received for host 'testunit', but the host could not be found! Perhaps you forgot to define the host in your config files?

The actual result is:

[1444131379] EXTERNAL COMMAND: PROCESS_HOST_CHECK_RESULT;testunit;0;host OK on $IPADDR

But the result is not displayed in the log until the configuration is present, only the "host could not be found" message.

The question: Is there a way to see the result, even if the host/service configuration is not yet created?

(example:)
[1444131379] EXTERNAL COMMAND: PROCESS_HOST_CHECK_RESULT;testunit;0;host OK on $IPADDR
[1444131379] Warning: Passive check result was received for host 'testunit', but the host could not be found! Perhaps you forgot to define the host in your config files?

Thanks in advance

Re: New hosts

Posted: Tue Oct 06, 2015 10:16 am
by eloyd
No, there is not. You need to define the host and service so that Nagios knows what host and service to assign the results to. Otherwise, you could not send the data to Nagios and just send it to a database for further analysis on your own outside of Nagios.

Re: New hosts

Posted: Tue Oct 06, 2015 10:26 am
by sspaise
Hi eloyd,

Thanks for the response. I had thought this would be the case, however let me throw a spanner into the works here ;)

I administer another Nagios instance for one of our sister companies, who also have an auto check in script (designed by a previous sysadmin before me). The example I provided in my OP was taken from their nagios.log. Their script first gets the hostname, IP address and a UUID (custom identifier on their systems), and uses these three pieces of information to create the host configuration.

This is displayed in the nagios.log as follows:

[1444114790] EXTERNAL COMMAND: PROCESS_HOST_CHECK_RESULT;TINAVPN-AR;0;host OK on $IPADDR|8a69-ac02-8c79-4aa8-923c-61ef-6e91-cd3f
[1444114790] Warning: Passive check result was received for host 'TINAVPN-AR', but the host could not be found!

The only differences between the output shown above (sister company), and what I receive when I submit a new host check is that the actual result is not present, and the following is displayed:

[1444131379] Warning: Passive check result was received for host 'testunit', but the host could not be found! Perhaps you forgot to define the host in your config files?

I assume they have something custom running here to perform this, but cant figure out how.

Do you have any suggestions perhaps, I will be happy to share configurations if required?

Thanks,

Re: New hosts

Posted: Tue Oct 06, 2015 10:39 am
by eloyd
I guess I'm not sure what you're asking. Nagios won't process the service check if it doesn't recognize the host and service. Nagios XI, however, will keep track of a list of "unconfigured hosts" that it will let you import those unknown passive checks once the hosts and service are defined. Is it possible that the other site is running XI?

Re: New hosts

Posted: Tue Oct 06, 2015 12:55 pm
by sspaise
Sorry, let me try with a bit more depth.

It's not actually Nagios that will be processing this first check from a new host. I've written a script that runs via the crontab which keeps watch over the nagios.log. When the script detects the "Host not found" message, it checks to see if the host already exists or is part of a test network which shouldn't ever check in, and if not continues to create the host, passive host/service, contact, hostgroup and command definitions.

This first check is passed using Gearman, and will send "host OK | role=X | $ipaddr", which will be used for further checking to confirm its a host allowed to check in, and define the role which determines which service configurations to add to the host.

Now on the Nagios instance of the sister company, they are using a method similar (watching the nagios.log for specific strings), and some of the strings it is looking for are sent on the first host check. This is the part I am confused about, because as you rightly mentioned Nagios will not log the submitted data to nagios.log, only print a message stating the host is not defined.

Somehow on the sister company instance they have the data entering the nagios.log (as shown in the previous example) before host/service definitions are created, which is then used to check its a valid host, and create the configuration.

The other site is running core 3.5.1, as is my new instance.

Re: New hosts

Posted: Tue Oct 06, 2015 1:01 pm
by eloyd
I like the idea of what you're doing (which looks like automatic passive host & service additions). I'm guessing this is for an environment where virtual or development boxes are added/removed frequently?

At any rate, I'm afraid I can't help you any further as I've never seen any case where the nagios.log file is logging things that aren't actually nagios host and services. Maybe one of the development or support people can help you out better than I can.

Re: New hosts

Posted: Tue Oct 06, 2015 1:24 pm
by sspaise
Yep you guessed it, we have a number of roles that are rather modular for our dev and customer servers, each also having certain contacts, and commands. We are currently transitioning from being really manual with our builds, to automating as much as possible to prevent user error and add consistency, Nagios being one of them.

There are certainly other ways to do this that I could implement faster, but after seeing the sister company instance piping initial passive checks into the nagios.log before any configuration, I've grown to the idea of trying to keep it all logged there (that and the need to know how they are doing it).

No problem, I appreciate your input and thanks!

Re: New hosts

Posted: Tue Oct 06, 2015 1:31 pm
by eloyd
Running Nagios Core 4.something, I just submitted a passive check and got this:

Code: Select all

# ./submit_check_result testHost testService 1 output
[1444155623] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;testHost;testService;1;output
[1444155623] Warning:  Passive check result was received for service 'testService' on host 'testHost', but the host could not be found!
[1444155623] Error: External command failed -> PROCESS_SERVICE_CHECK_RESULT;testHost;testService;1;output
[1444155623] External command error: Command failed
So, to me, it looks like I'm getting the entire command line in the /usr/local/nagios/var/nagios.log log file, even though the host isn't identified. I could then cron something that parses the log file, looks for "Error: External command failed " and parse appropriately to automatically add it back into Nagios.

In other words, it seems to be doing what you want.

Re: New hosts

Posted: Tue Oct 06, 2015 1:50 pm
by sspaise
Hm thanks for trying that, I've just discovered that if you pipe into the nagios.cmd file, it will display the check:

[1444156976] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;Sam-Test;testservice;1;output
[1444156976] Warning: Passive check result was received for service 'testservice' on host 'Sam-Test', but the host could not be found!

So it would appear that piping it directly into the nagios.cmd file will display the check results, but when using gearman it does not.

Problem solved ;)

Re: New hosts

Posted: Tue Oct 06, 2015 1:54 pm
by eloyd
sspaise wrote:So it would appear that piping it directly into the nagios.cmd file will display the check results, but when using gearman it does not.

Problem solved ;)
You know, if you mentioned mod_gearman, I totally missed that. Glad we found a solution!