Volatile nrds Checks

[grenley]

Hi.

We have a log monitor that scans for multiple patterns in an application logfile and generates an alert for each one it finds in a given pass.
I have set this up as a passive (nrdp) plugin.

I just tested by echoing two matching patterns into the logfile.
Only one appeared on the XI server.

I believe the solution would be to set this up as a volatile service, but I see no option to do that in the nrds cfg file on the agent.
Perhaps this is controlled by the XI server?
If so, I don't know how the command on the agent and any service definition on the XI server would be tied together.
If not, is there another mechanism?

We're just getting started on a very large XI implementation and this is a fundamental issue that I need to resolve.

Thanks,
Rick

[lmiltchev]

You can modify any service in the CCM and set is_volatile=1 (CCM->Services->Modify->Check Settings).

[grenley]

Thanks. That addressed one issue.
Two more remain:

1) we will have 60,000+ servers with 100's of thousands of these log monitors
Clicking on the "volatile" button for each of these is not feasible.
a) Is there a way to do this at the plugin level rather than the service level?
b) Can it be automated as part of the nrds configuration on each agent?

2) The plugin/service generated two unique alarms on one pass, yet only one showed up on the XI server
How can I address this?

Thanks,
Rick

[tmcdonald]

Focusing on issue #1 first, volatile settings would not be related to the NRDS settings - that just affects what checks the remote host performs, the frequency, etc.

As for making 60,000+ changes, do the services have a common template they all share? If so you could set this at a template level, however if any other hosts/services share that template and you do *not* want them to be volatile, you would need to remember to specifically set them as such, otherwise the template will make them volatile.

[grenley]

Sounds reasonable.
So, now I'm trying to understand the mechanism.

The services appear to be created dynamically by virtue of the fact that they are instantiated (if that's a legit term in this context) by being in the nrds config on the nrdp agents.

Here's a sample nrds.cfg entry:

Code: Select all

command[Logmon_PM-LINUX-Messages-EMOC]=/usr/local/nagios/libexec/check_logmon -c PM-LINUX-Messages-EMOC.conf

And here's the service that is generated by default:

Code: Select all

define service {
        host_name                       123.45.67.89
        service_description             Logmon_PM-LINUX-Messages-EMOC
        use                             xiwizard_passive_service
        is_volatile                     1
        max_check_attempts              1
        check_interval                  1
        retry_interval                  1
        check_period                    xi_timeperiod_24x7
        notification_interval           60
        notification_period             xi_timeperiod_24x7
        contacts                        nagiosadmin
        _xiwizard                       passiveobject
        register                        1
        }

So, it appears xiwizard_passive_service is automatically assigned (as it is for all the services but Ping).

What I want is for all Services that begin with Logmon to use my own template where I set is_volatile.
Since there could be dozens on a given agent and hundreds of agents on a given XI server, this needs to be automatically defined.

Does this make sense?
I'm still trying to getting my head around the architecture.

Thanks,
Rick

[slansing]

Well, doing some digging around it looks like you will have to edit the "xiwizard_passive_service" template itself and add volatility there, unfortunately this will effect everything that uses this template. Otherwise I would copy the template, rename it, add the volatility settings, and then apply it over your services via the template itself, or individually on those services.

[grenley]

I'm not sure what you mean by
"Otherwise I would copy the template, rename it, add the volatility settings, and then apply it over your services via the template itself"

I'm trying to see if there is an automation opportunity in that statement.

Also, is there some place I could get into the code that builds these service definitions (that come in from nrdp agents)?
That way, i could just look for service names that start with "Logmon" and have it write out my own template name rather than xiwizard_passive_service.

It would be really nice if the service definitions supported a wildcard like this:

define service {
host_name 123.45.67.89
service_description Logmon_*
etc.

[slansing]

What I meant was you can create a template or copy the one you are already using and make the modifications you need, then add it to a servicegroup which contains all of the services you intend to apply it to. That would apply the template to all of those services and make changes you wish to add in the future easier than making them individually to those services. I believe the XI passive wizard is used when adding services that come in through the Unconfigured Objects list, since they are only looking for a host/service pair and directly push the incoming data to them.

[grenley]

Forgive me if I seem obtuse, but don't you have to apply service group through the XI Config Mgr?
That's what I need to bypass.
I won't know what the service names are or what systems they will need to apply to ahead of time.
All I will know is that they will be Logmon_*.
I need a way for this to be done automatically.
Since all these log checks will be passive and, therefore (I presume), will be going through the Unconfigured Objects path, it sounds like what I need to do is intercept that process and make the proper template assignment.

Ideally, there would be a wildcard capability in the Service definition like this:
service_description Logmon_*

Short of that, I'm thinking I need to mod the Nagios code that assigns "xiwizard_passive_service" and insert some code to look for service name of "^Logmon_".

If you agree that's my best option, any idea where in the code that assignment is made?
I'm guessing it's in some php module somewhere.

I'd prefer to not have usermods on the Nagios code so, if you've got a better idea, I'd love to hear it.

Thanks,
Rick

[tmcdonald]

Take a look at the following SQL command:

Code: Select all

echo 'use nagiosql; SELECT id, service_description, is_volatile FROM tbl_service WHERE service_description LIKE "Logmon_%"' | mysql -u root -pnagiosxi

This will select the id (used by the backend code), the service description and the volatility status from the CCM database, for any service whose service_description matches the Logmon_* pattern. You can further refine this to only show those that are already volatile.

In order to set all the is_volatile flags, you can do something like this:

Code: Select all

echo 'use nagiosql; UPDATE tbl_service SET is_volatile=1 WHERE service_description LIKE "Logmon_%"' | mysql -u root -pnagiosxi

then Apply Config.

Bear in mind, we do not recommend making manual changes to the database like this. Lots of damage can be done if mistakes are made. Also in this case it is not an automatic solution, so maybe sticking it in a cron would work.