Notifications sent when turned off in template

[tzimmermanFSU]

We have notifications turned off in a service template for Memory Usage, but some of the checks using the template will still send notifications unless we go in and turn it off in the actual service check. Some of these service checks were quiet until we did some maintenance and our ansible playbook ran the nagios module to re-enable the checks after maintenance was complete. It didn't happen to all the servers that had maintenance, just a handful. A co-worker added all the checks into a service group and turned off notifications, so far that seems to be keeping them quiet.

1. RHEL 7.9
2. 64bit
3. manual install of XI
4. NCPA on hosts, XI 5.8.5

[tgriep]

In Nagios, when you set an option in the Host or Service object, it overrides the settings from the template assigned to the object.
Setting the option in the template will not override the settings in the Host or Service object so what you are seeing is the desired effect and that is why you had to disable the Notifications in the Service objects to stop the notifications.

Thank you.

[tzimmermanFSU]

Our services had the notifications enabled option set to SKIP, so unless I'm misunderstanding how SKIP works that should mean it uses the value defined in the template, correct?
We didn't change the value in the service until notifications started sending even when it was set to SKIP in the service and OFF in the template.

[tgriep]

Yes, when the setting is set to skip, it will use the template settings.

But, if someone / something went to a service or host in the status menu and enabled notifications directly, that will bypass the settings in the Core Config Manager.

What I mean by something, notifications can be enabled through the API or the external command file.

Could you see if the Ansible playbook script did that or if it restored the retention.dat file where the statuses are stored with the service notifications enabled?

Thank you.

[tzimmermanFSU]

The Ansible task is calling a module, the documentation doesn't state if it explicitly turns on notifications. Checking the services afterwards still had notifications set to SKIP. Looking at the retention.dat, I just looked up one of the services in question and it was still set to notifications_enabled=0.

I think I may have muddied the timeline a bit here, let me see if I can fix that.
1) We have a number of servers with Java based apps and they are set to run around 90% Memory Usage, we recently switched from SNMP to NCPA and started getting more notifications even after setting check and notification times to match the SNMP setup that was previously fairly quiet.
2) We then adjusted the template for the Memory Usage service to turn notifications off.
3) After waiting about 30min to see if the template change had been applied across the board, we were still getting notifications from a couple of servers. So we set the Memory Usage service on those servers to notifications off.
4) All is quiet after that point.
5) We start doing quarterly patching (this is when Ansible comes in), at which point servers that previously hadn't needed to have their Memory Usage service adjusted started sending notifications.
6) At this point a coworker created a service group and tried disabling notifications that way. Quiet again.
7) Another batch of servers gets patched and again a couple servers start sending Memory Usage notifications, and everywhere I look in the configs it shows notifications_enabled=0.

[tgriep]

Without having access to the files before Ansible was ran and after it was ran to compare them, I do not know why that is happening.

Can you provide some details on what is patched by Ansible?

Could you post of PM me your Nagios XI System Profile so I can review it?
To get your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and upload it to the post or PM it to me so I can look at it.

I would also need to know the Host Names that were sending the Memory Usage notification.

Thank you.

[tzimmermanFSU]

I don't think Ansible is the actual problem, as the problem was happening before we started running playbooks. I'll PM the profile.

[tgriep]

Thanks for the profile.

The host name you PMed me, is it still sending Notifications for the Memory Usage service or where the settings in the CCM changed to disable the notifications for the service?

The service settings are set to get the settings from the template so it should not be sending them unless they are enabled in the Service Status of the check.

To view those settings I would need to see this file.

Code: Select all

/usr/local/nagios/var/status.dat

One thing I do see that has to be changed is the Broker module line in the /usr/local/nagios/etc/nagios.cfg file is commented out and that has to be put back.

Edit the /usr/local/nagios/etc/nagios.cfg and change this line at the bottom of the file from

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

to

Code: Select all

broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Save the file and run this to restart nagios

Code: Select all

systemctl restart nagios

This being comment out will not update the statuses in the XI interface.

Thank you.

[tzimmermanFSU]

Sorry for the long delay in response has been busy.

Thank you for spotting that commented out broker line, it would have taken us a while to find it. Nagios XI was taking a long time, more than an hour, to verify its configuration, so we restarted the nagios service from the command line, which apparently caused some files to go missing and for some reason that line got commented out in that config.

I'm sending you PM with a status.dat file from today. One of the memory usage checks sent a notification and it was enabled in status.dat but the written configs show that it should be off.

[tgriep]

In Nagios, if someone goes to the Service Status of a Service and enables the Notifications in the Advanced TAB or the Quick actions link.

That will enable the Notifications regardless of the settings in the configuration file in the /usr/local/nagios/etc/services file or the templates, etc.

The status.dat file gets written to the retention.dat file so when nagios is restarted, it will use those settings.
The status.dat settings override the configuration file settings and that is normal.

So, if someone enabled the Notifications in the Web interface, that is why that service sent the Notification.

It looks like the server has the Enterprise feature so you can check the audit.log file to see if someone enabled it.
Go to the Admin > Audit log menu and run it.

Check for a line with the following in it to see if the Notification was enabled for that Host and Service and it should show who enabled it.

Submitted a command to Nagios Core: ENABLE_SVC_NOTIFICATIONS;

Thank you.