Page 1 of 1
XI 5.7.2 | CentOS 8 | Hosts failing to save to service check
Posted: Thu Jul 16, 2020 9:24 am
by danniiffxi
Hi Guys
OK this one has me stumped. I screen recorded a video and put it on YouTube so you can see what is happening..
The specs:
Nagios 5.7.2 on CentOS 8. Test config backup and restore from my old Nagios XI 5.7.2 CentOS 6 VM and then re-installed the tarball as recommended in the documentation due to jump in OS.
The Issue
The config in the video is untouched from the point of restore. I will try to explain.. So our original Test environment monitors our Prod Nagios server and alerts me if there is a problem with it. We have recently purchased a 2nd Unlimited licence for Nagios and built a second server. After transferring the CentOS 6 config backup and restoring it to the CentOS 8 and getting it all working I tried to add in the second nagios instance but noticed something stage when trying to add a host to a service.
You will see in the video, as I remove the host from the service (which was present on the old server and part of the restore), then go back in to add it and save it, when I go to check it has mysteriously vanished from the list, as if I had never added it in the first place.
https://youtu.be/M37sl529KAU
Do you have any idea what would cause this or how I can fix it please?
Re: XI 5.7.2 | CentOS 8 | Hosts failing to save to service c
Posted: Fri Jul 17, 2020 3:20 am
by danniiffxi
OK so to rule out an issue with the last backup i tried to restore, I reverted the VM snapshot back to XI 5.7.2 vanilla install on CentOS8, I then took a local config backup of one of our productions servers. I have restored said backup to the new CentOS 8 test server. I then installed the tarball to fix any binaries. I had no errors reported at all during the process but I still get the exact same problem.
Every host and service works perfectly, but if I try to add a host to the service check and save it, the hosts just vanish from the list like the video I posted above.
Re: XI 5.7.2 | CentOS 8 | Hosts failing to save to service c
Posted: Fri Jul 17, 2020 9:38 am
by ssax
Do you have any crashed tables?
Please send the output of these commands (as root):
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password
Code: Select all
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
Code: Select all
echo "SELECT relname as Table, pg_size_pretty(pg_total_relation_size(relid)) As Size, pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as ExternalSize FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;" | psql nagiosxi nagiosxi
If you run this command as root and leave it running:
Code: Select all
tail -Fn0 /var/log/httpd/error_log /var/log/httpd/ssl_error_log /usr/local/nagiosxi/var/cmdsubsys.log
Then replicate the issue again, do you see any errors that pop out?
I'm going to lab this up and see if I'm able to replicate it.
EDIT - I'm unable to replicate this on CentOS 8/XI 5.7.2.
Please send me a copy of your profile as well, you can download it from Admin > System Profile > Download Profile button.
Re: XI 5.7.2 | CentOS 8 | Hosts failing to save to service c
Posted: Fri Jul 17, 2020 10:12 am
by danniiffxi
Hi ssax
OK so I ran this as requested
Code: Select all
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
And it returned this.
Code: Select all
mysql: [Warning] Using a password on the command line interface can be insecure.
+--------------------------------------------+------------+
| Table | Size in MB |
+--------------------------------------------+------------+
| nagios_acknowledgements | 0.16 |
| nagios_commands | 0.02 |
| nagios_commenthistory | 13.51 |
| nagios_comments | 0.00 |
| nagios_configfiles | 0.01 |
| nagios_configfilevariables | 0.01 |
| nagios_conninfo | 0.44 |
| nagios_contact_addresses | 0.00 |
| nagios_contact_notificationcommands | 0.01 |
| nagios_contactgroup_members | 0.00 |
| nagios_contactgroups | 0.00 |
| nagios_contactnotificationmethods | 5.37 |
| nagios_contactnotifications | 3.77 |
| nagios_contacts | 0.01 |
| nagios_contactstatus | 0.00 |
| nagios_customvariables | 0.04 |
| nagios_customvariablestatus | 0.05 |
| nagios_dbversion | 0.00 |
| nagios_downtimehistory | 0.16 |
| nagios_eventhandlers | 0.24 |
| nagios_externalcommands | 0.15 |
| nagios_flappinghistory | 2.13 |
| nagios_host_contactgroups | 0.03 |
| nagios_host_contacts | 0.02 |
| nagios_host_parenthosts | 0.00 |
| nagios_hostchecks | 0.26 |
| nagios_hostdependencies | 0.00 |
| nagios_hostescalation_contactgroups | 0.00 |
| nagios_hostescalation_contacts | 0.00 |
| nagios_hostescalations | 0.00 |
| nagios_hostgroup_members | 0.02 |
| nagios_hostgroups | 0.00 |
| nagios_hosts | 0.09 |
| nagios_hoststatus | 0.32 |
| nagios_instances | 0.00 |
| nagios_logentries | 577.10 |
| nagios_notifications | 5.61 |
| nagios_objects | 6.41 |
| nagios_processevents | 0.45 |
| nagios_programstatus | 0.00 |
| nagios_runtimevariables | 0.00 |
| nagios_scheduleddowntime | 0.00 |
| nagios_service_contactgroups | 0.01 |
| nagios_service_contacts | 0.03 |
| nagios_service_parentservices | 0.00 |
| nagios_servicechecks | 0.21 |
| nagios_servicedependencies | 0.00 |
| nagios_serviceescalation_contactgroups | 0.00 |
| nagios_serviceescalation_contacts | 0.00 |
| nagios_serviceescalations | 0.00 |
| nagios_servicegroup_members | 0.00 |
| nagios_servicegroups | 0.00 |
| nagios_services | 0.06 |
| nagios_servicestatus | 0.31 |
| nagios_statehistory | 159.61 |
| nagios_systemcommands | 0.05 |
| nagios_timedeventqueue | 0.00 |
| nagios_timedevents | 0.00 |
| nagios_timeperiod_timeranges | 0.01 |
| nagios_timeperiods | 0.00 |
| tbl_command | 0.03 |
| tbl_contact | 0.01 |
| tbl_contactgroup | 0.01 |
| tbl_contacttemplate | 0.01 |
| tbl_domain | 0.01 |
| tbl_host | 0.17 |
| tbl_hostdependency | 0.00 |
| tbl_hostescalation | 0.00 |
| tbl_hostextinfo | 0.00 |
| tbl_hostgroup | 0.01 |
| tbl_hosttemplate | 0.01 |
| tbl_info | 0.13 |
| tbl_lnkContactToCommandHost | 0.00 |
| tbl_lnkContactToCommandService | 0.00 |
| tbl_lnkContactToContactgroup | 0.00 |
| tbl_lnkContactToContacttemplate | 0.00 |
| tbl_lnkContactToVariabledefinition | 0.00 |
| tbl_lnkContactgroupToContact | 0.00 |
| tbl_lnkContactgroupToContactgroup | 0.00 |
| tbl_lnkContacttemplateToCommandHost | 0.00 |
| tbl_lnkContacttemplateToCommandService | 0.00 |
| tbl_lnkContacttemplateToContactgroup | 0.00 |
| tbl_lnkContacttemplateToContacttemplate | 0.00 |
| tbl_lnkContacttemplateToVariabledefinition | 0.00 |
| tbl_lnkHostToContact | 0.01 |
| tbl_lnkHostToContactgroup | 0.02 |
| tbl_lnkHostToHost | 0.00 |
| tbl_lnkHostToHostgroup | 0.01 |
| tbl_lnkHostToHosttemplate | 0.01 |
| tbl_lnkHostToVariabledefinition | 0.01 |
| tbl_lnkHostdependencyToHost_DH | 0.00 |
| tbl_lnkHostdependencyToHost_H | 0.00 |
| tbl_lnkHostdependencyToHostgroup_DH | 0.00 |
| tbl_lnkHostdependencyToHostgroup_H | 0.00 |
| tbl_lnkHostescalationToContact | 0.00 |
| tbl_lnkHostescalationToContactgroup | 0.00 |
| tbl_lnkHostescalationToHost | 0.00 |
| tbl_lnkHostescalationToHostgroup | 0.00 |
| tbl_lnkHostgroupToHost | 0.00 |
| tbl_lnkHostgroupToHostgroup | 0.00 |
| tbl_lnkHosttemplateToContact | 0.00 |
| tbl_lnkHosttemplateToContactgroup | 0.00 |
| tbl_lnkHosttemplateToHost | 0.00 |
| tbl_lnkHosttemplateToHostgroup | 0.00 |
| tbl_lnkHosttemplateToHosttemplate | 0.00 |
| tbl_lnkHosttemplateToVariabledefinition | 0.00 |
| tbl_lnkServiceToContact | 0.01 |
| tbl_lnkServiceToContactgroup | 0.01 |
| tbl_lnkServiceToHost | 0.01 |
| tbl_lnkServiceToHostgroup | 0.00 |
| tbl_lnkServiceToServicegroup | 0.00 |
| tbl_lnkServiceToServicetemplate | 0.01 |
| tbl_lnkServiceToVariabledefinition | 0.01 |
| tbl_lnkServicedependencyToHost_DH | 0.00 |
| tbl_lnkServicedependencyToHost_H | 0.00 |
| tbl_lnkServicedependencyToHostgroup_DH | 0.00 |
| tbl_lnkServicedependencyToHostgroup_H | 0.00 |
| tbl_lnkServicedependencyToService_DS | 0.00 |
| tbl_lnkServicedependencyToService_S | 0.00 |
| tbl_lnkServicedependencyToServicegroup_DS | 0.02 |
| tbl_lnkServicedependencyToServicegroup_S | 0.02 |
| tbl_lnkServiceescalationToContact | 0.00 |
| tbl_lnkServiceescalationToContactgroup | 0.00 |
| tbl_lnkServiceescalationToHost | 0.00 |
| tbl_lnkServiceescalationToHostgroup | 0.00 |
| tbl_lnkServiceescalationToService | 0.00 |
| tbl_lnkServiceescalationToServicegroup | 0.02 |
| tbl_lnkServicegroupToService | 0.01 |
| tbl_lnkServicegroupToServicegroup | 0.00 |
| tbl_lnkServicetemplateToContact | 0.00 |
| tbl_lnkServicetemplateToContactgroup | 0.00 |
| tbl_lnkServicetemplateToHost | 0.00 |
| tbl_lnkServicetemplateToHostgroup | 0.00 |
| tbl_lnkServicetemplateToServicegroup | 0.00 |
| tbl_lnkServicetemplateToServicetemplate | 0.01 |
| tbl_lnkServicetemplateToVariabledefinition | 0.00 |
| tbl_lnkTimeperiodToTimeperiod | 0.00 |
| tbl_logbook | 1.60 |
| tbl_mainmenu | 0.00 |
| tbl_permission | 0.02 |
| tbl_permission_inactive | 0.02 |
| tbl_service | 0.08 |
| tbl_servicedependency | 0.00 |
| tbl_serviceescalation | 0.00 |
| tbl_serviceextinfo | 0.00 |
| tbl_servicegroup | 0.01 |
| tbl_servicetemplate | 0.02 |
| tbl_session | 0.00 |
| tbl_session_locks | 0.00 |
| tbl_settings | 0.00 |
| tbl_submenu | 0.00 |
| tbl_timedefinition | 0.01 |
| tbl_timeperiod | 0.01 |
| tbl_user | 0.01 |
| tbl_variabledefinition | 0.03 |
| xi_auditlog | 34.21 |
| xi_auth_tokens | 0.30 |
| xi_cmp_ccm_backups | 0.02 |
| xi_cmp_favorites | 0.03 |
| xi_cmp_nagiosbpi_backups | 0.02 |
| xi_cmp_trapdata | 0.03 |
| xi_cmp_trapdata_log | 0.03 |
| xi_commands | 0.03 |
| xi_deploy_agents | 0.02 |
| xi_deploy_jobs | 0.02 |
| xi_eventqueue | 0.10 |
| xi_events | 0.49 |
| xi_incidents | 0.00 |
| xi_meta | 27.94 |
| xi_mibs | 0.05 |
| xi_options | 0.04 |
| xi_sessions | 0.03 |
| xi_sysstat | 0.01 |
| xi_usermeta | 0.42 |
| xi_users | 0.02 |
+--------------------------------------------+------------+
[root@nagxit01 ~]#
Yep it failed, not running postgresql, this one is MySQL.
Code: Select all
echo "SELECT relname as Table, pg_size_pretty(pg_total_relation_size(relid)) As Size, pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as ExternalSize FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;" | psql nagiosxi nagiosxi
Ran this and replicated the error
Code: Select all
tail -Fn0 /var/log/httpd/error_log /var/log/httpd/ssl_error_log /usr/local/nagiosxi/var/cmdsubsys.log
This was the output.. I do not see any error as I am making the changes, only when making trying to apply the config which fails as the service check is no longer bound to a host.
Code: Select all
[root@nagxit01 ~]# tail -Fn0 /var/log/httpd/error_log /var/log/httpd/ssl_error_log /usr/local/nagiosxi/var/cmdsubsys.log
==> /var/log/httpd/error_log <==
==> /var/log/httpd/ssl_error_log <==
==> /usr/local/nagiosxi/var/cmdsubsys.log <==
...................................................
PROCESSED 0 COMMANDS
.PROCESSING COMMAND ID 9713...
PROCESS COMMAND: CMD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
--- reset_config_perms.sh ------------
> Setting script permissions
> Setting CCM script permissions
> Setting special script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting libexec directory permissions
> Setting Nagios XI config permissions
> Setting NOM checkpoint user:group permissions
> + Setting Nagios Core corelog.newobjects user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------
--- ccm_import.php -------------------
> Setting import directory: /usr/local/nagios/etc/import/
> Importing config files into the CCM
No files to import
--------------------------------------
--- ccm_export.php -------------------
> Writing CCM configuration to Nagios files
Finished writing out configuraton
--------------------------------------
--------------------------------------
> Verifying configuration with Nagios Core
> Output:
Nagios Core 4.4.6
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2020-04-28
License: GPL
Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Error: Service has no hosts and/or service_description (config file '/usr/local/nagios/etc/services/Admiralty Backup Router.cfg', starting on line 16)
Error processing object config files!
***> One or more problems was encountered while processing the config files...
Check your configuration file(s) to ensure that they contain valid
directives and data definitions. If you are upgrading from a previous
version of Nagios, you should be aware that some variables/definitions
may have been removed or modified in this version. Make sure to read
the HTML documentation regarding the config files, as well as the
'Whats New' section to find out what has changed.
> Return Code: 1
--------------------------------------
/usr/local/nagiosxi/nom/checkpoints/nagioscore/errors /usr/local/nagiosxi/scripts
tar: Removing leading `/' from member names
/usr/local/nagiosxi/scripts
LATEST NOM SNAPSHOT: /usr/local/nagiosxi/nom/checkpoints/nagioscore/1594997592.tar.gz
/ /usr/local/nagiosxi/scripts
RESTORING NOM SNAPSHOT : /usr/local/nagiosxi/nom/checkpoints/nagioscore/1594997592.tar.gz
tar: usr/local/nagios/etc/static: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/hosts: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/pnp/pages: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/pnp/check_commands: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/pnp: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/blib/arch/auto/TimePerf2: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/blib/arch/auto: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/blib/arch: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/blib/man3: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/blib/bin: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/blib/script: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/blib/man1: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/blib/lib/auto/TimePerf2: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/blib/lib/auto: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/blib/lib: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/blib: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/t: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2/lib: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import/TimePerf2: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/services: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors
/usr/local/nagiosxi/scripts
--- reset_config_perms.sh ------------
> Setting script permissions
> Setting CCM script permissions
> Setting special script permissions
> Setting special component script permissions
> Setting configuration file/directory permissions
> Setting perfdata directory and RRD permissions
> Setting libexec directory permissions
> Setting Nagios XI config permissions
> Setting NOM checkpoint user:group permissions
> + Setting Nagios Core corelog.newobjects user:group permissions
> + Setting CCM configuration file user:group permissions
> + Setting Recurring Downtime file user:group permissions
> + Setting BPI configuration file user:group permissions
--------------------------------------
OUTPUT=--------------------------------------
RETURNCODE=1
PROCESSING COMMAND ID 9714...
PROCESS COMMAND: CMD=1150, DATA=remove
CMDLINE=php /usr/local/nagiosxi/html/includes/components/nagiosbpi/api_tool.php --cmd=syncall
PHP Warning: count(): Parameter must be an array or an object that implements Countable in /usr/local/nagiosxi/html/includes/components/nagiosbpi/classes/BpGroup_class.php on line 837
PHP Warning: count(): Parameter must be an array or an object that implements Countable in /usr/local/nagiosxi/html/includes/components/nagiosbpi/classes/BpGroup_class.php on line 837
PHP Notice: Undefined variable: _SESSION in /usr/local/nagiosxi/html/includes/components/nagiosbpi/functions/manage_config.inc.php on line 192
fatal: invalid diff option/value: --ignore-cr-at-eol--ignore-blank-lines
fatal: invalid diff option/value: --ignore-cr-at-eol--ignore-blank-lines
PHP Notice: Undefined variable: changes in /usr/local/nagiosxi/html/includes/components/nagiosbpi/functions/manage_config.inc.php on line 275
PHP Notice: Undefined variable: old_config in /usr/local/nagiosxi/html/includes/components/nagiosbpi/functions/servicegroups.inc.php on line 161
PHP Notice: Undefined variable: _SESSION in /usr/local/nagiosxi/html/includes/components/nagiosbpi/functions/manage_config.inc.php on line 192
fatal: invalid diff option/value: --ignore-cr-at-eol--ignore-blank-lines
fatal: invalid diff option/value: --ignore-cr-at-eol--ignore-blank-lines
PHP Notice: Undefined variable: changes in /usr/local/nagiosxi/html/includes/components/nagiosbpi/functions/manage_config.inc.php on line 275
CMD: syncall
MSG: BPI configuration applied successfully! BPI configuration applied successfully!
OUTPUT=MSG: BPI configuration applied successfully! BPI configuration applied successfully!
RETURNCODE=0
..............
I will send both old and new server system profiles to you in the PM shortly.
Thank you.
Re: XI 5.7.2 | CentOS 8 | Hosts failing to save to service c
Posted: Fri Jul 17, 2020 5:17 pm
by ssax
Please edit your
/etc/my.cnf.d/mysql-server.conf and add this below your
[mysqld] section:
Then restart mysql:
Then apply configuration and test again.
Re: XI 5.7.2 | CentOS 8 | Hosts failing to save to service c
Posted: Mon Jul 20, 2020 2:31 am
by danniiffxi
"You're a wizard, Harry"
That worked. Thank you ssax
I have made a note of that and will apply to the other servers I have to migrate if it pops up again.
Re: XI 5.7.2 | CentOS 8 | Hosts failing to save to service c
Posted: Mon Jul 20, 2020 7:20 am
by scottwilkerson
danniiffxi wrote:"You're a wizard, Harry"
That worked. Thank you ssax
I have made a note of that and will apply to the other servers I have to migrate if it pops up again.
Great!
Locking thread