Page 2 of 5
Re: sucessful migration - mysql issues
Posted: Wed Jun 17, 2020 4:24 am
by veehexx
selinux is disabled. i beleive this was done by the install.sh XI installation script. i'm sure centos comes out of box with it enabled. It was actually my first thought on the problem.
the service check failure is another one on my list which i've now resolved with your sugestion. seems all our clients only had a 512 length dh key (which wasnt even referenced in nsclient.ini - maybe it's a fallback?) generated a 2048 length key and deployed to our windows hosts which is now working. fwiw now we have a 2048 DH key it doesn't seem to matter about -2 or -3 version enforcement.
Thats the first time i've seen DH key length as the reason for failure - maybe something to add to the 5.7.x release notes?
So thats the Service Alerts failures resolved, however we have ran both active & passive service checks Disabled for testing. it doesnt seem to matter enabled or disabled - CPU time is high and makes XI un-useable.
the new centos8 VM is on the same subnet but different IP so we currently have 2 XI servers running. our standard config allows the entire subnet, for exactly times like this where we're migrating and need both servers available concurrently.
I'll generate a new profile.zip and send it over to you, ssax, so you can see where we currently are... Hope it helps..
Re: sucessful migration - mysql issues
Posted: Wed Jun 17, 2020 3:16 pm
by ssax
There is not profile attached to the PM, please re-attach so I can investigate further.
Re: sucessful migration - mysql issues
Posted: Thu Jun 18, 2020 2:14 am
by veehexx
looks to be something wrong with PM's. resent a PM yesterday with attached file and it was stuck in outbox. retried with a new PM and it's doing the same.
I can PM myself with the file, just cant PM either yourself,
@ssax, or
@benjaminsmith.
Re: sucessful migration - mysql issues
Posted: Thu Jun 18, 2020 1:59 pm
by ssax
That's normal, it will stay in the outbox until they read it. Profile received, analyzing now. Will update shortly.
Re: sucessful migration - mysql issues
Posted: Thu Jun 18, 2020 2:18 pm
by ssax
Only the mysqld process is showing as consuming the CPU usage, let's check your DB table sizes:
Please send the output of this command (as root):
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the command if your DB is offloaded to another server and/or you've changed the root mysql password
Code: Select all
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
Please attach your
/etc/my.cnf and the output from this command:
Code: Select all
tail -n100 /var/log/mariadb/mariadb.log /var/log/mysqld.log
Also, I'm seeing this:
Code: Select all
[1592307517] NDO-3: ndo_handle_service_status(ndo-handlers.c:953): Unable to execute statement
[1592307537] NDO-3: ndo_return = 1 (Data too long for column 'check_command' at row 1)
You can resolve that by running this command:
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the command if your DB is offloaded to another server and/or you've changed the root mysql password
Code: Select all
mysql -h 127.0.0.1 -uroot -pnagiosxi nagios -e "ALTER TABLE nagios_servicestatus MODIFY check_command TEXT NOT NULL DEFAULT '';"
Re: sucessful migration - mysql issues
Posted: Fri Jun 19, 2020 10:03 am
by veehexx
Code: Select all
mysql: [Warning] Using a password on the command line interface can be insecure.
+--------------------------------------------+------------+
| Table | Size in MB |
+--------------------------------------------+------------+
| nagios_acknowledgements | 0.06 |
| nagios_commands | 0.02 |
| nagios_commenthistory | 455.09 |
| nagios_comments | 6.52 |
| nagios_configfiles | 0.01 |
| nagios_configfilevariables | 0.01 |
| nagios_conninfo | 0.10 |
| nagios_contact_addresses | 0.00 |
| nagios_contact_notificationcommands | 0.01 |
| nagios_contactgroup_members | 0.00 |
| nagios_contactgroups | 0.00 |
| nagios_contactnotificationmethods | 0.13 |
| nagios_contactnotifications | 0.09 |
| nagios_contacts | 0.00 |
| nagios_contactstatus | 0.00 |
| nagios_customvariables | 0.01 |
| nagios_customvariablestatus | 0.01 |
| nagios_dbversion | 0.00 |
| nagios_downtimehistory | 48.19 |
| nagios_eventhandlers | 0.00 |
| nagios_externalcommands | 7.87 |
| nagios_flappinghistory | 0.42 |
| nagios_host_contactgroups | 0.01 |
| nagios_host_contacts | 0.00 |
| nagios_host_parenthosts | 0.01 |
| nagios_hostchecks | 0.09 |
| nagios_hostdependencies | 0.00 |
| nagios_hostescalation_contactgroups | 0.00 |
| nagios_hostescalation_contacts | 0.00 |
| nagios_hostescalations | 0.00 |
| nagios_hostgroup_members | 0.01 |
| nagios_hostgroups | 0.00 |
| nagios_hosts | 0.03 |
| nagios_hoststatus | 0.10 |
| nagios_instances | 0.00 |
| nagios_logentries | 87.62 |
| nagios_notifications | 0.24 |
| nagios_objects | 0.65 |
| nagios_processevents | 0.12 |
| nagios_programstatus | 0.00 |
| nagios_runtimevariables | 0.00 |
| nagios_scheduleddowntime | 4.72 |
| nagios_service_contactgroups | 0.07 |
| nagios_service_contacts | 0.00 |
| nagios_service_parentservices | 0.00 |
| nagios_servicechecks | 0.00 |
| nagios_servicedependencies | 0.00 |
| nagios_serviceescalation_contactgroups | 0.00 |
| nagios_serviceescalation_contacts | 0.00 |
| nagios_serviceescalations | 0.00 |
| nagios_servicegroup_members | 0.03 |
| nagios_servicegroups | 0.00 |
| nagios_services | 0.47 |
| nagios_servicestatus | 0.81 |
| nagios_statehistory | 532.82 |
| nagios_systemcommands | 0.05 |
| nagios_timedeventqueue | 0.00 |
| nagios_timedevents | 0.00 |
| nagios_timeperiod_timeranges | 0.01 |
| nagios_timeperiods | 0.00 |
| tbl_command | 0.04 |
| tbl_contact | 0.01 |
| tbl_contactgroup | 0.01 |
| tbl_contacttemplate | 0.01 |
| tbl_domain | 0.01 |
| tbl_host | 0.02 |
| tbl_hostdependency | 0.01 |
| tbl_hostescalation | 0.00 |
| tbl_hostextinfo | 0.00 |
| tbl_hostgroup | 0.01 |
| tbl_hosttemplate | 0.01 |
| tbl_info | 0.13 |
| tbl_lnkContactToCommandHost | 0.00 |
| tbl_lnkContactToCommandService | 0.00 |
| tbl_lnkContactToContactgroup | 0.00 |
| tbl_lnkContactToContacttemplate | 0.00 |
| tbl_lnkContactToVariabledefinition | 0.00 |
| tbl_lnkContactgroupToContact | 0.00 |
| tbl_lnkContactgroupToContactgroup | 0.00 |
| tbl_lnkContacttemplateToCommandHost | 0.00 |
| tbl_lnkContacttemplateToCommandService | 0.00 |
| tbl_lnkContacttemplateToContactgroup | 0.00 |
| tbl_lnkContacttemplateToContacttemplate | 0.00 |
| tbl_lnkContacttemplateToVariabledefinition | 0.00 |
| tbl_lnkHostToContact | 0.00 |
| tbl_lnkHostToContactgroup | 0.00 |
| tbl_lnkHostToHost | 0.00 |
| tbl_lnkHostToHostgroup | 0.00 |
| tbl_lnkHostToHosttemplate | 0.01 |
| tbl_lnkHostToVariabledefinition | 0.00 |
| tbl_lnkHostdependencyToHost_DH | 0.00 |
| tbl_lnkHostdependencyToHost_H | 0.00 |
| tbl_lnkHostdependencyToHostgroup_DH | 0.00 |
| tbl_lnkHostdependencyToHostgroup_H | 0.00 |
| tbl_lnkHostescalationToContact | 0.00 |
| tbl_lnkHostescalationToContactgroup | 0.00 |
| tbl_lnkHostescalationToHost | 0.00 |
| tbl_lnkHostescalationToHostgroup | 0.00 |
| tbl_lnkHostgroupToHost | 0.01 |
| tbl_lnkHostgroupToHostgroup | 0.00 |
| tbl_lnkHosttemplateToContact | 0.00 |
| tbl_lnkHosttemplateToContactgroup | 0.00 |
| tbl_lnkHosttemplateToHost | 0.00 |
| tbl_lnkHosttemplateToHostgroup | 0.00 |
| tbl_lnkHosttemplateToHosttemplate | 0.00 |
| tbl_lnkHosttemplateToVariabledefinition | 0.00 |
| tbl_lnkServiceToContact | 0.00 |
| tbl_lnkServiceToContactgroup | 0.00 |
| tbl_lnkServiceToHost | 0.01 |
| tbl_lnkServiceToHostgroup | 0.01 |
| tbl_lnkServiceToServicegroup | 0.01 |
| tbl_lnkServiceToServicetemplate | 0.02 |
| tbl_lnkServiceToVariabledefinition | 0.00 |
| tbl_lnkServicedependencyToHost_DH | 0.00 |
| tbl_lnkServicedependencyToHost_H | 0.00 |
| tbl_lnkServicedependencyToHostgroup_DH | 0.00 |
| tbl_lnkServicedependencyToHostgroup_H | 0.00 |
| tbl_lnkServicedependencyToService_DS | 0.00 |
| tbl_lnkServicedependencyToService_S | 0.00 |
| tbl_lnkServicedependencyToServicegroup_DS | 0.02 |
| tbl_lnkServicedependencyToServicegroup_S | 0.02 |
| tbl_lnkServiceescalationToContact | 0.00 |
| tbl_lnkServiceescalationToContactgroup | 0.00 |
| tbl_lnkServiceescalationToHost | 0.00 |
| tbl_lnkServiceescalationToHostgroup | 0.00 |
| tbl_lnkServiceescalationToService | 0.00 |
| tbl_lnkServiceescalationToServicegroup | 0.02 |
| tbl_lnkServicegroupToService | 0.00 |
| tbl_lnkServicegroupToServicegroup | 0.00 |
| tbl_lnkServicetemplateToContact | 0.00 |
| tbl_lnkServicetemplateToContactgroup | 0.00 |
| tbl_lnkServicetemplateToHost | 0.00 |
| tbl_lnkServicetemplateToHostgroup | 0.00 |
| tbl_lnkServicetemplateToServicegroup | 0.00 |
| tbl_lnkServicetemplateToServicetemplate | 0.01 |
| tbl_lnkServicetemplateToVariabledefinition | 0.00 |
| tbl_lnkTimeperiodToTimeperiod | 0.00 |
| tbl_logbook | 0.00 |
| tbl_mainmenu | 0.00 |
| tbl_permission | 0.02 |
| tbl_permission_inactive | 0.02 |
| tbl_service | 0.11 |
| tbl_servicedependency | 0.01 |
| tbl_serviceescalation | 0.00 |
| tbl_serviceextinfo | 0.00 |
| tbl_servicegroup | 0.01 |
| tbl_servicetemplate | 0.02 |
| tbl_session | 0.00 |
| tbl_session_locks | 0.00 |
| tbl_settings | 0.00 |
| tbl_submenu | 0.00 |
| tbl_timedefinition | 0.00 |
| tbl_timeperiod | 0.01 |
| tbl_user | 0.01 |
| tbl_variabledefinition | 0.00 |
| xi_auditlog | 0.78 |
| xi_auth_tokens | 0.03 |
| xi_cmp_ccm_backups | 0.02 |
| xi_cmp_favorites | 0.03 |
| xi_cmp_nagiosbpi_backups | 0.02 |
| xi_cmp_trapdata | 0.03 |
| xi_cmp_trapdata_log | 0.03 |
| xi_commands | 0.00 |
| xi_deploy_agents | 0.02 |
| xi_deploy_jobs | 0.02 |
| xi_eventqueue | 0.64 |
| xi_events | 0.00 |
| xi_incidents | 0.00 |
| xi_meta | 0.02 |
| xi_mibs | 0.05 |
| xi_options | 0.04 |
| xi_sessions | 0.03 |
| xi_sysstat | 0.01 |
| xi_usermeta | 0.13 |
| xi_users | 0.01 |
+--------------------------------------------+------------+
my.cnf
Code: Select all
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
Ithe mysql/mariadb logs do not exist. closest i can find is /var/log/mysqld/mysqld.log which is empty.
the NDO-3 fix, completes but gets the following error
Code: Select all
ERROR 1101 (42000) at line 1: BLOB, TEXT, GEOMETRY or JSON column 'check_command' can't have a default value
no change anything unfortunately - issues still remain
Re: sucessful migration - mysql issues
Posted: Fri Jun 19, 2020 4:14 pm
by ssax
What is the output of these commands?
Code: Select all
rpm -qa | grep -i mysql
rpm -qa | grep -i mariadb
You may need to do this so set sql_mode to blank:
https://support.nagios.com/kb/article/n ... n-822.html
Try doing this:
Please run these commands from the server and see if it resolves your issue:
Code: Select all
wget https://assets.nagios.com/downloads/nagiosxi/patches/patch-ndo.sh
sh patch-ndo.sh
Once that's done, apply configuration and see if the problems are resolved.
Re: sucessful migration - mysql issues
Posted: Mon Jun 22, 2020 7:14 am
by veehexx
Code: Select all
# rpm -qa | grep -i mysql
php-mysqlnd-7.2.24-1.module_el8.1.0+276+68c728ae.x86_64
mysql-8.0.17-3.module_el8.0.0+181+899d6349.x86_64
mysql-common-8.0.17-3.module_el8.0.0+181+899d6349.x86_64
perl-DBD-MySQL-4.046-3.module_el8.1.0+203+e45423dc.x86_64
mysql-devel-8.0.17-3.module_el8.0.0+181+899d6349.x86_64
mysql-errmsg-8.0.17-3.module_el8.0.0+181+899d6349.x86_64
mysql-server-8.0.17-3.module_el8.0.0+181+899d6349.x86_64
mysql-libs-8.0.17-3.module_el8.0.0+181+899d6349.x86_64
Code: Select all
# rpm -qa | grep -i mariadb
mariadb-connector-c-config-3.0.7-1.el8.noarch
mariadb-connector-c-3.0.7-1.el8.x86_64
sql_mode:
Code: Select all
# mysql -u root -p'nagiosxi' -e "SELECT @@GLOBAL.sql_mode;"
mysql: [Warning] Using a password on the command line interface can be insecure.
+-----------------------------------------------------------------------------------------------------------------------+
| @@GLOBAL.sql_mode
|
+-----------------------------------------------------------------------------------------------------------------------+
| ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION |
+-----------------------------------------------------------------------------------------------------------------------+
edited /etc/my.cnf.d/mysql-server.cnf (since /etc/my.cnf has includedir) to add the sql_mode="", rechecked above command and now the output table is empty.
still the same.
ndo patch output, after the file extraction just incase it might be of use...
Code: Select all
...SNIPPED...
UPGRADE: NDO is being upgraded...
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for a BSD-compatible install... /usr/bin/install -c
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for stdlib.h... (cached) yes
checking stdio.h usability... yes
checking stdio.h presence... yes
checking for stdio.h... yes
checking for string.h... (cached) yes
checking mysql/mysql.h usability... yes
checking mysql/mysql.h presence... yes
checking for mysql/mysql.h... yes
checking mysql/errmsg.h usability... yes
checking mysql/errmsg.h presence... yes
checking for mysql/errmsg.h... yes
checking for stdlib.h... (cached) yes
checking for GNU libc compatible malloc... yes
/usr/bin/mysql_config
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating test/Makefile
config.status: creating include/config.h
*** Configuration summary for ***:
----------------
User: nagios
Group: nagios
Bin directory: /usr/local/nagios/bin
Config directory: /usr/local/nagios/etc
Nagios config: /usr/local/nagios/etc/nagios.cfg
----------------
Review the options above for accuracy. If they look okay,
type 'make all' to compile NDOUtils, or type 'make' to get
a list of make options.
cd src/ && make ndo.so
make[1]: Entering directory '/tmp/nagiosxi/subcomponents/ndo/ndo-3.0.1.patched/src'
gcc -fPIC -shared -g -O2 -I/usr/include/mysql -m64 ndo.c -o ndo.so -L/usr/lib64/mysql -lmysqlclient -lpthread -lz -lm -lrt -lssl -lcrypto -ldl -pthread
In file included from ndo.c:143:
ndo-startup.c: In function ‘ndo_startup_thread’:
ndo-startup.c:97:9: warning: implicit declaration of function ‘ndo_deregister_queue_functions’; did you mean ‘ndo_register_queue_callbacks’? [-Wimplicit-function-declaration]
ndo_deregister_queue_functions();
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ndo_register_queue_callbacks
ndo-startup.c:98:9: warning: ‘return’ with no value, in function returning non-void
return;
^~~~~~
ndo-startup.c:87:8: note: declared here
void * ndo_startup_thread(void * args)
^~~~~~~~~~~~~~~~~~
ndo-startup.c:104:9: warning: ‘return’ with no value, in function returning non-void
return;
^~~~~~
ndo-startup.c:87:8: note: declared here
void * ndo_startup_thread(void * args)
^~~~~~~~~~~~~~~~~~
ndo-startup.c:109:9: warning: ‘return’ with no value, in function returning non-void
return;
^~~~~~
ndo-startup.c:87:8: note: declared here
void * ndo_startup_thread(void * args)
^~~~~~~~~~~~~~~~~~
ndo.c: In function ‘ndo_log’:
ndo.c:153:5: warning: implicit declaration of function ‘asprintf’; did you mean ‘vsprintf’? [-Wimplicit-function-declaration]
asprintf(&tmp, "NDO-3: %s", buffer);
^~~~~~~~
vsprintf
make[1]: Leaving directory '/tmp/nagiosxi/subcomponents/ndo/ndo-3.0.1.patched/src'
cd src/ && make ndo.so
make[1]: Entering directory '/tmp/nagiosxi/subcomponents/ndo/ndo-3.0.1.patched/src'
make[1]: 'ndo.so' is up to date.
make[1]: Leaving directory '/tmp/nagiosxi/subcomponents/ndo/ndo-3.0.1.patched/src'
/usr/bin/install -c -m 774 -o nagios -g nagios src/ndo.so /usr/local/nagios/bin/ndo.so
Shared module ndo.so installed
/usr/local/nagios/etc/nagios.cfg configured to use NDO
UPGRADE: POST-UPGRADE: NDO post upgrade started...
Performing upgrade...
> Updating dbversion table
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
Database upgrade is complete!
UPGRADE: POST-UPGRADE: NDO post upgraded OK.
UPGRADE: NDO upgraded OK.
no fix - mysqld process still high cpu, 'apply configuration' took an excessively long time (around 90secs), and while the top bar system status icon is green and the 'system status' page is all green, the 'Monitoring Engine Status' page shows the process stopped (red blip) and refuses to start manually or after a reboot.
edit: ndo2db.service does not exist on my install. is it mean to?
Re: sucessful migration - mysql issues
Posted: Mon Jun 22, 2020 5:38 pm
by ssax
The ndo2db service no longer exists in XI 5.7 (on purpose).
Did you restart the nagios services after making those changes? If not, please do.
Please send me a FRESH copy of your profile so I can view the latest logs, you can download it from Admin > System Profile > Download Profile button.
Re: sucessful migration - mysql issues
Posted: Tue Jun 23, 2020 2:44 am
by veehexx
Hi ssax,
nagios service was restarted just before i posted my last message. VM was also rebooted.
another profile.zip has been sent to you.
thanks!