Bulk Mod Tool Not Working

[mkeey]

Used the Bulk Modification Tool to add a contact to all hosts & services. When I checked the services I found that the new contact wasn't added to all services. Then I found that not all services had notifications_enabled set to "1". Used the bulk modification tool to update all the services to have notifications set to on. Then I noticed that not all services had the field in the config files. If it's not there, can't set it.

I've got a real mess on my XI's. This isn't just one having a problem but two so far. So, my notifications are incorrect on about 4000 services and I don't know how many hosts.

How can I fix this and why isn't the bulk modification tool working?

PS - I did try deleting all config files and then run the bulk mod tool, but that didn't help.

[mkeey]

Follow up, on the Linux server where we run our Nagios XI, I did a grep for notifications_enabled and found that some are completely missing from the CFG files. This is a real mess.

I'd like to share the spreadsheet but it has proprietary information I cannot post on the Forum's website.

[benjaminsmith]

Hi @mkeey,

I just tested this on my system it's working as expected. What version of Nagios Xi are you using? Also, did you Apply Configuration after making changes using the Bulk Modifications Tool?

Let us know.

[mkeey]

We are running Nagios XI version: 5.5.5. Yes, I did the apply config after each attempt.

[benjaminsmith]

Hi @mkeey,

I believe the configuration changes are not being applied and this is often due to permissions issues. Log in as root and run the following from the command line and try again.

Code: Select all

/usr/local/nagiosxi/scripts/reset_config_perms.sh

If this does not resolve the issue, please run the following command:

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

And then make those changes using the Bulk Mods Tool, select Apply Configuration and post the output errors for us to review.

Thanks.

[mkeey]

Do not have root authority access on our XI servers and I tried the reset with the Nagios account, but each reset failed with "Operation not permitted". I will have to enlist some assistance from our Linux Admins to try the reset.
So, I went to the log file and tried the tail command. Received nothing but "PROCESSED 0 COMMANDS". So, I copied the cmdsubsys.log file to my desktop and looked for anything other than "PROCESSED 0 COMMANDS". Last entry was...

.............PROCESSING COMMAND ID 1342...
PROCESS COMMAND: CMD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
No entry for terminal type "unknown";
using dumb terminal settings.

--- reset_config_perms.sh ------------
> Setting CCM script permissions
> Setting script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting NOM checkpoint user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------

--- ccm_import.php -------------------
> Setting import directory: /usr/local/nagios/etc/import/
> Importing config files into the CCM
No files to import
--------------------------------------

--- ccm_export.php -------------------
> Writing CCM configuration to Nagios files
Finished writing out configuraton
--------------------------------------

--------------------------------------
> Verifying configuration with Nagios Core
> Output:
Nagios Core 4.4.2
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2018-08-16
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 2425 services.
Checked 425 hosts.
Checked 2 host groups.
Checked 0 service groups.
Checked 40 contacts.
Checked 2 contact groups.
Checked 128 commands.
Checked 46 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 425 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 46 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
> Return Code: 0
--------------------------------------
OUTPUT=--------------------------------------
RETURNCODE=0
PROCESSING COMMAND ID 1343...
PROCESS COMMAND: CMD=1150, DATA=remove
CMDLINE=php /usr/local/nagiosxi/html/includes/components/nagiosbpi/api_tool.php --cmd=syncall
CMD: syncall
MSG: BPI configuration applied successfully! BPI configuration applied successfully!
OUTPUT=MSG: BPI configuration applied successfully! BPI configuration applied successfully!
RETURNCODE=0
.................PROCESSING COMMAND ID 1344...
PROCESS COMMAND: CMD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
No entry for terminal type "unknown";
using dumb terminal settings.

--- reset_config_perms.sh ------------
> Setting CCM script permissions
> Setting script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting NOM checkpoint user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------

--- ccm_import.php -------------------
> Setting import directory: /usr/local/nagios/etc/import/
> Importing config files into the CCM
No files to import
--------------------------------------

--- ccm_export.php -------------------
> Writing CCM configuration to Nagios files
Finished writing out configuraton
--------------------------------------

--------------------------------------
> Verifying configuration with Nagios Core
> Output:
Nagios Core 4.4.2
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2018-08-16
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 2425 services.
Checked 425 hosts.
Checked 2 host groups.
Checked 0 service groups.
Checked 40 contacts.
Checked 2 contact groups.
Checked 128 commands.
Checked 46 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 425 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 46 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
> Return Code: 0
--------------------------------------
OUTPUT=--------------------------------------
RETURNCODE=0
PROCESSING COMMAND ID 1345...
PROCESS COMMAND: CMD=1150, DATA=remove
CMDLINE=php /usr/local/nagiosxi/html/includes/components/nagiosbpi/api_tool.php --cmd=syncall
............CMD: syncall
MSG: BPI configuration applied successfully! BPI configuration applied successfully!
OUTPUT=MSG: BPI configuration applied successfully! BPI configuration applied successfully!
RETURNCODE=0

PROCESSED 4 COMMANDS

[mkeey]

Looked at that reset perm script and found you're just doing CHOWN on all the config files. So, I displayed the services config files to see what the owner is before we ran it. I changed the names to "HOSTNAME" to the names being proprietary but this shows you the owner of each config file...

[nagios@mlwnag25]:[/usr/local/nagios/etc/services]# ls -ltr
total 2188
-rw-rw-r-- 1 apache nagios 8041 Mar 28 09:49 HOSTNAME01.cfg
-rw-rw-r-- 1 apache nagios 8041 Mar 28 09:49 HOSTNAME02.cfg
-rw-rw-r-- 1 apache nagios 7442 Mar 28 09:49 HOSTNAME03.cfg
-rw-rw-r-- 1 apache nagios 7442 Mar 28 09:49 HOSTNAME04.cfg
-rw-rw-r-- 1 apache nagios 8053 Mar 28 09:49 HOSTNAME05.cfg
-rw-rw-r-- 1 apache nagios 8053 Mar 28 09:49 HOSTNAME06.cfg
-rw-rw-r-- 1 apache nagios ..................ETC, ETC, ETC.

Your reset script would reset each one to "root.$nagiosgroup". It also sets the read/write to 644. Are these Config Files incorrect?

[mkeey]

Got one of our Linux team members to run the reset script on all XI servers. Here's the output from that execution...

[root@TESTHOST01]:[/usr/local/nagiosxi/scripts]# ./reset_config_perms.sh
--- reset_config_perms.sh ------------
> Setting CCM script permissions
> Setting script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting NOM checkpoint user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------


[root@PRODHOST01]:[/root]# /usr/local/nagiosxi/scripts/reset_config_perms.sh
--- reset_config_perms.sh ------------
> Setting CCM script permissions
> Setting script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting NOM checkpoint user:group permissions
> + Setting CCM configuration file user:group permissionS
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------


[root@PRODHOST02]:[/root]# /usr/local/nagiosxi/scripts/reset_config_perms.sh
--- reset_config_perms.sh ------------
> Setting CCM script permissions
> Setting script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting NOM checkpoint user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------


[root@PRODHOST03]:[/root]# /usr/local/nagiosxi/scripts/reset_config_perms.sh
--- reset_config_perms.sh ------------
> Setting CCM script permissions
> Setting script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting NOM checkpoint user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------

I then went to the CCM and opened the Bulk Modification Tool. Selected "Add Contact" and selected the desired contact. Then clicked on "Select Services" and selected all services moving them over to the "Assigned" screen. Clicked "Close". The display changed from "0" to "2425" services selected. Then clicked on "Save Changes". Got a green box that said: "Updates saved successfully!". Then clicked on "Apply Configuration" on the same screen (Bulk Modification Screen). After about 15 seconds got a display "Configuration applied successfully." along with a green box that said "Nagios Core was restarted with an updated configuration."
I then went and checked a service and found that the new contact (CONTACT03) was NOT ADDED to the list...

define service {
host_name HOSTSERVER01
service_description CPU Usage
use xiwizard_windowsserver_nsclient_service
check_command check_xi_service_nsclient!!CPULOAD!-l 5,90,95
max_check_attempts 20
check_interval 5
retry_interval 3
check_period xi_timeperiod_24x7
notification_interval 0
notification_period xi_timeperiod_24x7
notifications_enabled 1
contacts CONTACT01,CONTACT02
_xiwizard windowsserver
register 1
}

[benjaminsmith]

Hi @mkeey,

Thanks for posting the logs. Those look normal, however, since you are making changes to over 2400 services, it's likely the server is maxing out the php limits and unable to process all of the changes.

Please perform the the following steps.
1. Backup Nagios XI
2. Run the database repair script as root from the command line:

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

3. Use the bulk modifications tool to makes changes to a smaller number for host and services ( 100 or less). If it's working, then you'll need to optimize the php settings (i.e. max_input_vars). Follow our guide: Nagios XI - Optimizing The PHP Settings File

4. If step 3 does not resolve the issue, please PM your system profile for us to review.
To Send Over a System Profile:
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and upload it to a cloud storage of your choice. You can share a link with me or attach the file in a personal message

Thank you.

[mkeey]

Thanks for the reply. I had thought about breaking the updates up into smaller chunks. Was hoping for a bit more than 100 though! That's 24 updates I'll have to perform for 2400 monitors.

I've got to get this fixed, so I'll do them in smaller chunks but then test the PHP fix process on our Test/UAT system. Please leave the thread open as this may take some time.


Regards,
Martin