Apply configuration takes more more to Apply

[ITOMB_IMT]

Hi,

whenever i make a change monitoring/alerting in CCM, i apply configuration. it shows configuration applied successfully. when i again go to the monitoring/alerting in CCM, its saying "Changes detected! Apply Configuration for new changes to take effect." After applying configuration for 5-7 times i don't see any error. Any ideas on why its taking several attempts to apply configuration?

Thanks,

[npolovenko]

@ISSB_MAOST, Please run the following script to check&repair the database:

/usr/local/nagiosxi/scripts/repair_databases.sh

Is your database offloaded? Please run the following commands to check your php and system time on the XI server(It needs to match):

grep "date.timezone" /etc/php.ini
ls -l /etc/localtime
php -r 'echo date("D M j G:i:s T Y")."\n";'
date
echo "SELECT NOW();" | mysql -u root -pnagiosxi -h my-RDS-FQDN
php -r "echo date('r').PHP_EOL;"
strings /etc/localtime | tail -1

[ITOMB_IMT]

@ISSB_MAOST, Please run the following script to check&repair the database:

/usr/local/nagiosxi/scripts/repair_databases.sh

Is your database offloaded? Please run the following commands to check your php and system time on the XI server(It needs to match):

grep "date.timezone" /etc/php.ini
ls -l /etc/localtime
php -r 'echo date("D M j G:i:s T Y")."\n";'
date
echo "SELECT NOW();" | mysql -u root -pnagiosxi -h my-RDS-FQDN
php -r "echo date('r').PHP_EOL;"
strings /etc/localtime | tail -1

Yes, our database is offloaded to a remote server.
#grep "date.timezone" /etc/php.ini
; http://php.net/date.timezone
date.timezone = America/New_York
# ls -l /etc/localtime
lrwxrwxrwx. 1 root root 38 Oct 15 10:15 /etc/localtime -> ../usr/share/zoneinfo/America/New_York
# php -r 'echo date("D M j G:i:s T Y")."\n";'
Thu Jan 17 10:06:37 EST 2019
# date
Thu Jan 17 10:06:37 EST 2019
# echo "SELECT NOW();" | mysql -u root -pnagiosxi -h my-RDS-FQDN
NOW()
2019-01-17 10:07:24
# php -r "echo date('r').PHP_EOL;"
Thu, 17 Jan 2019 10:06:37 -0500
# strings /etc/localtime | tail -1
EST5EDT,M3.2.0,M11.1.0

when i ran repair databases script the nagios db shows ok whereas the nagiosxi and nagiosql throws "nagiosxi.xi_usermeta
note : The storage engine for the table doesn't support repair" and completes without any errors.

nagios offloaded database repair succeeded
nagiosql offloaded database repair succeeded
nagiosxi offloaded database repair succeeded

still the problem persists.


Thanks,

[npolovenko]

@ISSB_MAOST, Can you run the same time commands on the offloaded database server? Timezone and time on the DB server needs to match the time on the main XI. I've seen this issue a lot.

[ITOMB_IMT]

@ISSB_MAOST, Can you run the same time commands on the offloaded database server? Timezone and time on the DB server needs to match the time on the main XI. I've seen this issue a lot.

Timezone and time on App and Db servers are the same
DB Server:
]# grep "date.timezone" /etc/php.ini
; http://php.net/date.timezone
;date.timezone =
# ls -l /etc/localtime
lrwxrwxrwx. 1 root root 38 Oct 4 07:54 /etc/localtime -> ../usr/share/zoneinfo/America/New_York
# php -r 'echo date("D M j G:i:s T Y")."\n";'
-bash: php: command not found
# date
Fri Jan 18 09:57:33 EST 2019
# echo "SELECT NOW();" | mysql -u root -proot
NOW()
2019-01-18 09:57:33
# php -r "echo date('r').PHP_EOL;"
-bash: php: command not found
# strings /etc/localtime | tail -1
EST5EDT,M3.2.0,M11.1.0


APP Server:
# grep "date.timezone" /etc/php.ini
; http://php.net/date.timezone
date.timezone = America/New_York
# ls -l /etc/localtime
lrwxrwxrwx. 1 root root 38 Oct 15 10:15 /etc/localtime -> ../usr/share/zoneinfo/America/New_York
# php -r 'echo date("D M j G:i:s T Y")."\n";'
Fri Jan 18 9:57:04 EST 2019
# date
Fri Jan 18 09:57:04 EST 2019
# echo "SELECT NOW();" | mysql -u root -proot -h Remote server
NOW()
2019-01-18 09:57:51
# php -r "echo date('r').PHP_EOL;"
Fri, 18 Jan 2019 09:57:04 -0500
# strings /etc/localtime | tail -1
EST5EDT,M3.2.0,M11.1.0

[ssax]

Please go to Configure > Core Config Manager > Tools > Config File Management:
- Click the Delete Files button (don't worry, it's safe, they will be rewritten)
- Then click the Write Configs button
- Then try to apply configuration and let us know if the "Changes detected" issue is resolved.

Additionally, please post the output of this command:

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagiosxi --table -h my-RDS-FQDN

Please PM me a copy of your profile as well so that I can look through your logs/configs, you can download it from Admin > System Profile > Download Profile.

Thank you

[ITOMB_IMT]

Please go to Configure > Core Config Manager > Tools > Config File Management:
- Click the Delete Files button (don't worry, it's safe, they will be rewritten)
- Then click the Write Configs button
- Then try to apply configuration and let us know if the "Changes detected" issue is resolved.

Additionally, please post the output of this command:

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagiosxi --table -h my-RDS-FQDN

Please PM me a copy of your profile as well so that I can look through your logs/configs, you can download it from Admin > System Profile > Download Profile.

Thank you

+--------------------------------------------+------------+
| Table | Size in MB |
+--------------------------------------------+------------+
| nagios_acknowledgements | 0.00 |
| nagios_commands | 0.02 |
| nagios_commenthistory | 2.63 |
| nagios_comments | 0.02 |
| nagios_configfiles | 0.00 |
| nagios_configfilevariables | 0.01 |
| nagios_conninfo | 0.17 |
| nagios_contact_addresses | 0.00 |
| nagios_contact_notificationcommands | 0.02 |
| nagios_contactgroup_members | 0.00 |
| nagios_contactgroups | 0.00 |
| nagios_contactnotificationmethods | 0.35 |
| nagios_contactnotifications | 0.38 |
| nagios_contacts | 0.01 |
| nagios_contactstatus | 0.00 |
| nagios_customvariables | 0.04 |
| nagios_customvariablestatus | 0.04 |
| nagios_dbversion | 0.00 |
| nagios_downtimehistory | 0.05 |
| nagios_eventhandlers | 0.02 |
| nagios_externalcommands | 0.06 |
| nagios_flappinghistory | 0.01 |
| nagios_host_contactgroups | 0.00 |
| nagios_host_contacts | 0.00 |
| nagios_host_parenthosts | 0.00 |
| nagios_hostchecks | 0.00 |
| nagios_hostdependencies | 0.00 |
| nagios_hostescalation_contactgroups | 0.00 |
| nagios_hostescalation_contacts | 0.00 |
| nagios_hostescalations | 0.00 |
| nagios_hostgroup_members | 0.00 |
| nagios_hostgroups | 0.00 |
| nagios_hosts | 0.01 |
| nagios_hoststatus | 0.03 |
| nagios_instances | 0.00 |
| nagios_logentries | 11.59 |
| nagios_notifications | 0.80 |
| nagios_objects | 0.54 |
| nagios_processevents | 0.35 |
| nagios_programstatus | 0.00 |
| nagios_runtimevariables | 0.00 |
| nagios_scheduleddowntime | 0.01 |
| nagios_service_contactgroups | 0.01 |
| nagios_service_contacts | 0.01 |
| nagios_service_parentservices | 0.00 |
| nagios_servicechecks | 0.00 |
| nagios_servicedependencies | 0.00 |
| nagios_serviceescalation_contactgroups | 0.00 |
| nagios_serviceescalation_contacts | 0.00 |
| nagios_serviceescalations | 0.00 |
| nagios_servicegroup_members | 0.00 |
| nagios_servicegroups | 0.00 |
| nagios_services | 0.10 |
| nagios_servicestatus | 0.25 |
| nagios_statehistory | 0.46 |
| nagios_systemcommands | 0.04 |
| nagios_timedeventqueue | 0.00 |
| nagios_timedevents | 0.00 |
| nagios_timeperiod_timeranges | 0.01 |
| nagios_timeperiods | 0.00 |
| tbl_command | 0.06 |
| tbl_contact | 0.03 |
| tbl_contactgroup | 0.03 |
| tbl_contacttemplate | 0.03 |
| tbl_domain | 0.03 |
| tbl_host | 0.03 |
| tbl_hostdependency | 0.03 |
| tbl_hostescalation | 0.03 |
| tbl_hostextinfo | 0.03 |
| tbl_hostgroup | 0.03 |
| tbl_hosttemplate | 0.03 |
| tbl_info | 0.17 |
| tbl_lnkContactToCommandHost | 0.02 |
| tbl_lnkContactToCommandService | 0.02 |
| tbl_lnkContactToContactgroup | 0.02 |
| tbl_lnkContactToContacttemplate | 0.02 |
| tbl_lnkContactToVariabledefinition | 0.02 |
| tbl_lnkContactgroupToContact | 0.02 |
| tbl_lnkContactgroupToContactgroup | 0.02 |
| tbl_lnkContacttemplateToCommandHost | 0.02 |
| tbl_lnkContacttemplateToCommandService | 0.02 |
| tbl_lnkContacttemplateToContactgroup | 0.02 |
| tbl_lnkContacttemplateToContacttemplate | 0.02 |
| tbl_lnkContacttemplateToVariabledefinition | 0.02 |
| tbl_lnkHostToContact | 0.02 |
| tbl_lnkHostToContactgroup | 0.02 |
| tbl_lnkHostToHost | 0.02 |
| tbl_lnkHostToHostgroup | 0.02 |
| tbl_lnkHostToHosttemplate | 0.02 |
| tbl_lnkHostToVariabledefinition | 0.02 |
| tbl_lnkHostdependencyToHost_DH | 0.02 |
| tbl_lnkHostdependencyToHost_H | 0.02 |
| tbl_lnkHostdependencyToHostgroup_DH | 0.02 |
| tbl_lnkHostdependencyToHostgroup_H | 0.02 |
| tbl_lnkHostescalationToContact | 0.02 |
| tbl_lnkHostescalationToContactgroup | 0.02 |
| tbl_lnkHostescalationToHost | 0.02 |
| tbl_lnkHostescalationToHostgroup | 0.02 |
| tbl_lnkHostgroupToHost | 0.02 |
| tbl_lnkHostgroupToHostgroup | 0.02 |
| tbl_lnkHosttemplateToContact | 0.02 |
| tbl_lnkHosttemplateToContactgroup | 0.02 |
| tbl_lnkHosttemplateToHost | 0.02 |
| tbl_lnkHosttemplateToHostgroup | 0.02 |
| tbl_lnkHosttemplateToHosttemplate | 0.02 |
| tbl_lnkHosttemplateToVariabledefinition | 0.02 |
| tbl_lnkServiceToContact | 0.02 |
| tbl_lnkServiceToContactgroup | 0.02 |
| tbl_lnkServiceToHost | 0.02 |
| tbl_lnkServiceToHostgroup | 0.02 |
| tbl_lnkServiceToServicegroup | 0.02 |
| tbl_lnkServiceToServicetemplate | 0.02 |
| tbl_lnkServiceToVariabledefinition | 0.02 |
| tbl_lnkServicedependencyToHost_DH | 0.02 |
| tbl_lnkServicedependencyToHost_H | 0.02 |
| tbl_lnkServicedependencyToHostgroup_DH | 0.02 |
| tbl_lnkServicedependencyToHostgroup_H | 0.02 |
| tbl_lnkServicedependencyToService_DS | 0.02 |
| tbl_lnkServicedependencyToService_S | 0.02 |
| tbl_lnkServiceescalationToContact | 0.02 |
| tbl_lnkServiceescalationToContactgroup | 0.02 |
| tbl_lnkServiceescalationToHost | 0.02 |
| tbl_lnkServiceescalationToHostgroup | 0.02 |
| tbl_lnkServiceescalationToService | 0.02 |
| tbl_lnkServicegroupToService | 0.02 |
| tbl_lnkServicegroupToServicegroup | 0.02 |
| tbl_lnkServicetemplateToContact | 0.02 |
| tbl_lnkServicetemplateToContactgroup | 0.02 |
| tbl_lnkServicetemplateToHost | 0.02 |
| tbl_lnkServicetemplateToHostgroup | 0.02 |
| tbl_lnkServicetemplateToServicegroup | 0.02 |
| tbl_lnkServicetemplateToServicetemplate | 0.02 |
| tbl_lnkServicetemplateToVariabledefinition | 0.02 |
| tbl_lnkTimeperiodToTimeperiod | 0.02 |
| tbl_logbook | 0.02 |
| tbl_mainmenu | 0.02 |
| tbl_permission | 0.02 |
| tbl_permission_inactive | 0.02 |
| tbl_service | 0.11 |
| tbl_servicedependency | 0.03 |
| tbl_serviceescalation | 0.03 |
| tbl_serviceextinfo | 0.03 |
| tbl_servicegroup | 0.03 |
| tbl_servicetemplate | 0.03 |
| tbl_session | 0.02 |
| tbl_session_locks | 0.02 |
| tbl_settings | 0.03 |
| tbl_submenu | 0.02 |
| tbl_timedefinition | 0.02 |
| tbl_timeperiod | 0.03 |
| tbl_user | 0.03 |
| tbl_variabledefinition | 0.06 |
| xi_auditlog | 2.23 |
| xi_auth_tokens | 0.42 |
| xi_cmp_trapdata | 0.03 |
| xi_cmp_trapdata_log | 0.03 |
| xi_commands | 0.06 |
| xi_eventqueue | 0.03 |
| xi_events | 0.09 |
| xi_incidents | 0.02 |
| xi_meta | 1.52 |
| xi_options | 0.03 |
| xi_sessions | 0.03 |
| xi_sysstat | 0.03 |
| xi_usermeta | 0.19 |
| xi_users | 0.03 |
+--------------------------------------------+------------+

Please see the pm for "System Profile"

Thanks,

[ssax]

Please send me the output of this command on your offloaded db:

Code: Select all

tail /var/log/messages /var/log/mariadb/mariadb.log /var/log/mysqld.log

Then run these commands on your XI server:

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

Then run this tail command (and leave it running):

Code: Select all

tail -F /usr/local/nagiosxi/var/cmdsubsys.log

Then try to apply configuration through the web interface and once it completes/errors send me the full output from the tail command above.

[ITOMB_IMT]

Please see the results

On Database server:
]# tail /var/log/messages /var/log/mariadb/mariadb.log /var/log/mysqld.log
==> /var/log/messages <==

Jan 25 12:47:25 APP_Server auditmanager: Received wakeup signal before sleep finished
Jan 25 12:47:25 APP_Server auditmanager: Received wakeup signal before sleep finished
Jan 25 12:47:25 APP_Server auditmanager: Received wakeup signal before sleep finished

==> /var/log/mariadb/mariadb.log <==
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/ ... oting.html
190125 10:15:48 InnoDB: Error: table `nagiosxi`.`xi_sysstat` does not exist in the InnoDB internal
InnoDB: data dictionary though MySQL is trying to drop it.
InnoDB: Have you copied the .frm file of the table to the
InnoDB: MySQL database directory from another database?
InnoDB: You can look for further help from
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/ ... oting.html
190125 10:20:34 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='Secondary_db_server', master_port='3306', master_log_file='mariadb-bin.000001', master_log_pos='46066554'.
190125 10:20:34 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000001' at position 46066554, relay log './mariadb-relay-bin.000001' position: 4
190125 10:20:34 [Note] Slave I/O thread: connected to master 'maost@ssc-dev-maost-nagiosdb-01.ba.ssa.gov:3306',replication started in log 'mariadb-bin.000001' at position 46066554
tail: cannot open '/var/log/mysqld.log' for reading: No such file or directory

On App Server:
tail -F /usr/local/nagiosxi/var/cmdsubsys.log
PROCESSED 0 COMMANDS
............................................................
PROCESSED 0 COMMANDS
.............PROCESSING COMMAND ID 4541...
PROCESS COMMAND: CMD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
No entry for terminal type "unknown";
using dumb terminal settings.

--- reset_config_perms.sh ------------
> Setting CCM script permissions
> Setting script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting NOM checkpoint user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------

--- ccm_import.php -------------------
> Setting import directory: /usr/local/nagios/etc/import/
> Importing config files into the CCM
No files to import
--------------------------------------

--- ccm_export.php -------------------
> Writing CCM configuration to Nagios files
Finished writing out configuraton
--------------------------------------

--------------------------------------
> Verifying configuration with Nagios Core
> Output:
Nagios Core 4.4.2
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2018-08-16
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Warning: Duplicate definition found for service 'Total Processes' on host 'Client_server' (config file '/usr/local/nagios/etc/services/client_server.cfg', starting on line 175)
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 430 services.
Warning: Host 'client_server' has no default contacts or contactgroups defined!
Checked 21 hosts.
Checked 16 host groups.
Checked 0 service groups.
Checked 19 contacts.
Checked 4 contact groups.
Checked 127 commands.
Checked 23 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 21 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 23 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 1
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
> Return Code: 0
--------------------------------------
OUTPUT=--------------------------------------
RETURNCODE=0
PROCESSING COMMAND ID 4543...
PROCESS COMMAND: CMD=1150, DATA=remove
CMDLINE=php /usr/local/nagiosxi/html/includes/components/nagiosbpi/api_tool.php --cmd=syncall
CMD: syncall
MSG: BPI configuration applied successfully! BPI configuration applied successfully!
OUTPUT=MSG: BPI configuration applied successfully! BPI configuration applied successfully!
RETURNCODE=0
..........................................
PROCESSED 2 COMMANDS

[cdienger]

Can you disable replication on the database temporarily as a test? I wonder if this may be preventing the database from being written to in a timely manner.