WARNING: RLIMIT_NPROC is 47285, total max estimated processe

[ScottMc]

We started noticing some intermittent weirdness with our Nagios XI 5.6.4 on CentOS 7 installation the other day. Pages won't completely load. Service checks won't show, but then will later on. General slowness.

Checking the event log I see this repeat every minute:

Code: Select all

2019-07-25 16:30:12	WARNING: RLIMIT_NPROC is 47285, total max estimated processes is 91767! You should increase your limits (ulimit -u, or limits.conf)
Information	2019-07-25 16:30:12	Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
Information	2019-07-25 16:30:12	ndomod registered for adaptive contact data'
Information	2019-07-25 16:30:12	ndomod registered for contact status data'
Information	2019-07-25 16:30:12	ndomod registered for state change data'
Information	2019-07-25 16:30:12	ndomod registered for acknowledgement data'
Information	2019-07-25 16:30:12	ndomod registered for contact notification data'
Information	2019-07-25 16:30:12	ndomod registered for contact data'
Information	2019-07-25 16:30:12	ndomod registered for retention data'
Information	2019-07-25 16:30:12	ndomod registered for aggregated status data'
Information	2019-07-25 16:30:12	ndomod registered for external command data'
Information	2019-07-25 16:30:12	ndomod registered for adaptive service data'
Information	2019-07-25 16:30:12	ndomod registered for adaptive host data'
Information	2019-07-25 16:30:12	ndomod registered for adaptive program data'
Information	2019-07-25 16:30:12	ndomod registered for service status data'
Information	2019-07-25 16:30:12	ndomod registered for host status data'
Information	2019-07-25 16:30:12	ndomod registered for program status data'
Information	2019-07-25 16:30:12	ndomod registered for flapping data'
Information	2019-07-25 16:30:12	ndomod registered for downtime data'
Information	2019-07-25 16:30:12	ndomod registered for comment data'
Information	2019-07-25 16:30:12	ndomod registered for notification data'
Information	2019-07-25 16:30:12	ndomod registered for event handler data'
Information	2019-07-25 16:30:12	ndomod registered for system command data'
Information	2019-07-25 16:30:12	ndomod registered for log data'

I can't find that limit specified anywhere, and setting a higher limit in /etc/security/limits.conf and/or /etc/security/limits.d/20-nproc.conf don't seem to change that number, though the changes are reflected in ulimit -a when run for both root and nagios users. Any idea what I'm missing?

Scott

[tgriep]

I found this post that you can take a look at for increasing the NPROC setting in the systemd service file.
https://serverfault.com/questions/62861 ... n-centos-7

That warning may be a symptom of an another issue in the system from your description on how the interface is acting.

I would guess that there is a MYSQL database issue and to fully troubleshoot this, we would need to see the System Profile from the server.

To get your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and upload it to the forum post.

Also, run the following commands as root and post the output here.

Code: Select all

echo "SELECT table_schema as 'Database', table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES ORDER BY (data_length + index_length) DESC;" |mysql -t -u root -pnagiosxi
mysql -u root -pnagiosxi -e "show global status like '%used_connections%'; show variables like 'max_connections';"

Check the MariaDB log file to see if there are any errors at the bottom and post them here.

Code: Select all

/var/log/mariadb/mariadb.log

[ScottMc]

I'll have to wait on data extraction request from the security team since the server is in a secure environment. I did do a tail -f /usr/local/nagiosxi/var/cmdsubsys.log and noticed that it keeps cycling through applying configurations. It never finds any problems with the configuration but reapplies every minutes or so:

Code: Select all

[root@cdeatlnagios01 ~]# tail -f /usr/local/nagiosxi/var/cmdsubsys.log
--------------------------------------

--- ccm_import.php -------------------
> Setting import directory: /usr/local/nagios/etc/import/
> Importing config files into the CCM
  No files to import
--------------------------------------

--- ccm_export.php -------------------
> Writing CCM configuration to Nagios files
  Finished writing out configuraton
--------------------------------------

--------------------------------------
> Verifying configuration with Nagios Core
> Output:
Nagios Core 4.4.3
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2019-01-15
License: GPL

Website: https://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
        Checked 10906 services.
        Checked 914 hosts.
        Checked 54 host groups.
        Checked 21 service groups.
        Checked 51 contacts.
        Checked 8 contact groups.
        Checked 193 commands.
        Checked 62 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 914 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 62 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
> Return Code: 0
--------------------------------------
OUTPUT=--------------------------------------
RETURNCODE=0
..........................................^C

echo "SELECT table_schema as 'Database', table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES ORDER BY (data_length + index_length) DESC;" |mysql -t -u root -pnagiosxi
mysql -u root -pnagiosxi -e "show global status like '%used_connections%'; show variables like 'max_connections';"

Code: Select all

[root@cdeatlnagios01 ~]# echo "SELECT table_schema as 'Database', table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size i                                                                                                                                                      n MB' FROM information_schema.TABLES ORDER BY (data_length + index_length) DESC;" |mysql -t -u root -pnagiosxi
+--------------------+----------------------------------------------+------------+
| Database           | Table                                        | Size in MB |
+--------------------+----------------------------------------------+------------+
| nagios             | nagios_statehistory                          |    1750.59 |
| nagios             | nagios_commenthistory                        |     711.06 |
| nagios             | nagios_logentries                            |     376.16 |
| nagios             | nagios_downtimehistory                       |      20.28 |
| nagios             | nagios_contactnotifications                  |       9.42 |
| nagios             | nagios_contactnotificationmethods            |       8.87 |
| nagios             | nagios_servicestatus                         |       6.36 |
| nagiosxi           | xi_auditlog                                  |       4.99 |
| nagiosxi           | xi_meta                                      |       3.30 |
| nagios             | nagios_notifications                         |       3.20 |
| nagios             | nagios_flappinghistory                       |       2.43 |
| nagios             | nagios_services                              |       2.29 |
| nagios             | nagios_objects                               |       1.76 |
| nagios             | nagios_processevents                         |       1.03 |
| nagios             | nagios_customvariablestatus                  |       0.74 |
| nagios             | nagios_conninfo                              |       0.73 |
| nagios             | nagios_customvariables                       |       0.68 |
| nagiosxi           | xi_usermeta                                  |       0.63 |
| nagios             | nagios_hoststatus                            |       0.54 |
| nagiosql           | tbl_service                                  |       0.52 |
| nagios             | nagios_service_contactgroups                 |       0.51 |
| nagiosxi           | xi_eventqueue                                |       0.45 |
| mysql              | help_topic                                   |       0.45 |
| nagiosxi           | xi_auth_tokens                               |       0.42 |
| nagios             | nagios_externalcommands                      |       0.33 |
| nagios             | nagios_hosts                                 |       0.29 |
| nagiosql           | tbl_variabledefinition                       |       0.20 |
| nagiosql           | tbl_host                                     |       0.16 |
| nagios             | nagios_comments                              |       0.14 |
| nagiosql           | tbl_info                                     |       0.13 |
| nagios             | nagios_acknowledgements                      |       0.13 |
| nagiosxi           | xi_events                                    |       0.12 |
| nagios             | nagios_servicegroup_members                  |       0.11 |
| mysql              | help_keyword                                 |       0.10 |
| nagios             | nagios_hostgroup_members                     |       0.09 |
| nagiosql           | tbl_lnkServiceToContactgroup                 |       0.09 |
| nagiosql           | tbl_lnkServiceToServicetemplate              |       0.08 |
| nagiosql           | tbl_lnkServiceToHost                         |       0.08 |
| nagiosql           | tbl_lnkServiceToVariabledefinition           |       0.07 |
| nagiosxi           | xi_options                                   |       0.06 |
| nagios             | nagios_scheduleddowntime                     |       0.06 |
| nagiosxi           | xi_mibs                                      |       0.05 |
| nagiosql           | tbl_lnkHostgroupToHost                       |       0.05 |
| nagiosql           | tbl_command                                  |       0.04 |
| nagios             | nagios_host_contactgroups                    |       0.04 |
| nagios             | nagios_contact_notificationcommands          |       0.03 |
| nagiosxi           | xi_commands                                  |       0.03 |
| nagios             | nagios_systemcommands                        |       0.03 |
| nagiosxi           | xi_sessions                                  |       0.03 |
| nagiosxi           | xi_cmp_trapdata_log                          |       0.03 |
| nagiosxi           | xi_cmp_trapdata                              |       0.03 |
| nagios             | nagios_eventhandlers                         |       0.03 |
| nagiosql           | tbl_lnkHostToHosttemplate                    |       0.03 |
| nagios             | nagios_commands                              |       0.03 |
| mysql              | help_relation                                |       0.03 |
| nagios             | nagios_timeperiod_timeranges                 |       0.02 |
| nagiosql           | tbl_timedefinition                           |       0.02 |
| nagiosxi           | xi_users                                     |       0.02 |
| nagiosql           | tbl_lnkHostToVariabledefinition              |       0.02 |
| nagiosql           | tbl_lnkHostToContactgroup                    |       0.02 |
| nagiosql           | tbl_servicetemplate                          |       0.02 |
| nagiosql           | tbl_lnkServiceescalationToServicegroup       |       0.02 |
| nagiosql           | tbl_permission_inactive                      |       0.02 |
| nagiosql           | tbl_permission                               |       0.02 |
| nagiosql           | tbl_lnkServicedependencyToServicegroup_S     |       0.02 |
| nagiosql           | tbl_lnkServicedependencyToServicegroup_DS    |       0.02 |
| nagiosql           | tbl_hosttemplate                             |       0.02 |
| nagiosql           | tbl_contact                                  |       0.01 |
| nagiosql           | tbl_lnkServicegroupToService                 |       0.01 |
| nagiosql           | tbl_timeperiod                               |       0.01 |
| nagios             | nagios_configfilevariables                   |       0.01 |
| nagiosql           | tbl_hostgroup                                |       0.01 |
| nagiosql           | tbl_servicegroup                             |       0.01 |
| nagiosql           | tbl_contactgroup                             |       0.01 |
| nagiosql           | tbl_domain                                   |       0.01 |
| nagiosql           | tbl_contacttemplate                          |       0.01 |
| nagiosql           | tbl_user                                     |       0.01 |
| nagios             | nagios_contacts                              |       0.01 |
| nagiosxi           | xi_sysstat                                   |       0.01 |
| information_schema | COLUMNS                                      |       0.01 |
| information_schema | PROCESSLIST                                  |       0.01 |
| information_schema | VIEWS                                        |       0.01 |
| information_schema | PLUGINS                                      |       0.01 |
| information_schema | PARTITIONS                                   |       0.01 |
| information_schema | TRIGGERS                                     |       0.01 |
| information_schema | PARAMETERS                                   |       0.01 |
| information_schema | EVENTS                                       |       0.01 |
| information_schema | ROUTINES                                     |       0.01 |
| nagiosql           | tbl_lnkServiceToServicegroup                 |       0.01 |
| nagios             | nagios_timeperiods                           |       0.01 |
| mysql              | db                                           |       0.01 |
| nagiosql           | tbl_lnkServiceToHostgroup                    |       0.01 |
| mysql              | proxies_priv                                 |       0.01 |
| nagios             | nagios_contactgroup_members                  |       0.01 |
| nagios             | nagios_hostgroups                            |       0.01 |
| nagios             | nagios_service_contacts                      |       0.01 |
| nagios             | nagios_contactstatus                         |       0.01 |
| nagiosql           | tbl_lnkServicetemplateToServicetemplate      |       0.01 |
| nagios             | nagios_configfiles                           |       0.00 |
| mysql              | proc                                         |       0.00 |
| nagios             | nagios_servicegroups                         |       0.00 |
| mysql              | help_category                                |       0.00 |
| nagios             | nagios_runtimevariables                      |       0.00 |
| nagiosql           | tbl_hostdependency                           |       0.00 |
| mysql              | columns_priv                                 |       0.00 |
| mysql              | tables_priv                                  |       0.00 |
| nagiosql           | tbl_serviceescalation                        |       0.00 |
| nagiosql           | tbl_servicedependency                        |       0.00 |
| mysql              | procs_priv                                   |       0.00 |
| nagiosql           | tbl_hostescalation                           |       0.00 |
| nagiosql           | tbl_logbook                                  |       0.00 |
| nagiosql           | tbl_submenu                                  |       0.00 |
| nagios             | nagios_contactgroups                         |       0.00 |
| nagiosql           | tbl_settings                                 |       0.00 |
| nagios             | nagios_programstatus                         |       0.00 |
| nagios             | nagios_host_contacts                         |       0.00 |
| nagios             | nagios_host_parenthosts                      |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToHosttemplate            |       0.00 |
| nagiosql           | tbl_lnkContactToContacttemplate              |       0.00 |
| mysql              | user                                         |       0.00 |
| nagiosql           | tbl_session                                  |       0.00 |
| nagiosql           | tbl_lnkServiceToContact                      |       0.00 |
| nagiosql           | tbl_mainmenu                                 |       0.00 |
| nagiosql           | tbl_lnkContactgroupToContact                 |       0.00 |
| nagiosql           | tbl_lnkHostToHostgroup                       |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToContactgroup            |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToContactgroup         |       0.00 |
| nagiosql           | tbl_lnkHostgroupToHostgroup                  |       0.00 |
| nagiosql           | tbl_session_locks                            |       0.00 |
| nagiosql           | tbl_lnkContactToCommandService               |       0.00 |
| nagiosql           | tbl_lnkContactToCommandHost                  |       0.00 |
| nagiosql           | tbl_lnkHostToHost                            |       0.00 |
| nagiosql           | tbl_lnkHostToContact                         |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToCommandService       |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToCommandHost          |       0.00 |
| nagios             | nagios_instances                             |       0.00 |
| nagiosql           | tbl_lnkServicegroupToServicegroup            |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToHostgroup               |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToContactgroup         |       0.00 |
| nagiosql           | tbl_lnkContactgroupToContactgroup            |       0.00 |
| mysql              | host                                         |       0.00 |
| mysql              | event                                        |       0.00 |
| nagios             | nagios_dbversion                             |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToService_S          |       0.00 |
| nagios             | nagios_hostescalation_contacts               |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToVariabledefinition   |       0.00 |
| nagios             | nagios_servicechecks                         |       0.00 |
| nagios             | nagios_hostescalation_contactgroups          |       0.00 |
| nagios             | nagios_service_parentservices                |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToService_DS         |       0.00 |
| nagios             | nagios_hostdependencies                      |       0.00 |
| mysql              | func                                         |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToServicegroup         |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToHostgroup_H        |       0.00 |
| nagiosql           | tbl_hostextinfo                              |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToHostgroup            |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToHostgroup_DH       |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToVariabledefinition   |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToHost_H             |       0.00 |
| mysql              | time_zone_transition                         |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToHost                 |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToHost_DH            |       0.00 |
| mysql              | time_zone_name                               |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToContacttemplate      |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToVariabledefinition      |       0.00 |
| mysql              | time_zone_leap_second                        |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToContact              |       0.00 |
| nagiosxi           | xi_incidents                                 |       0.00 |
| mysql              | time_zone                                    |       0.00 |
| mysql              | time_zone_transition_type                    |       0.00 |
| nagiosql           | tbl_lnkHostescalationToContactgroup          |       0.00 |
| nagiosql           | tbl_serviceextinfo                           |       0.00 |
| nagios             | nagios_timedevents                           |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToHost                    |       0.00 |
| nagiosql           | tbl_lnkHostescalationToContact               |       0.00 |
| nagios             | nagios_timedeventqueue                       |       0.00 |
| mysql              | servers                                      |       0.00 |
| nagiosql           | tbl_lnkServiceescalationToService            |       0.00 |
| nagiosql           | tbl_lnkHostdependencyToHostgroup_H           |       0.00 |
| nagiosql           | tbl_lnkContactToVariabledefinition           |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToContact                 |       0.00 |
| nagiosql           | tbl_lnkHostdependencyToHostgroup_DH          |       0.00 |
| nagiosql           | tbl_lnkServiceescalationToHost               |       0.00 |
| nagiosql           | tbl_lnkServiceescalationToHostgroup          |       0.00 |
| nagiosql           | tbl_lnkHostdependencyToHost_H                |       0.00 |
| nagios             | nagios_serviceescalations                    |       0.00 |
| nagiosql           | tbl_lnkServiceescalationToContactgroup       |       0.00 |
| nagiosql           | tbl_lnkHostdependencyToHost_DH               |       0.00 |
| nagios             | nagios_serviceescalation_contacts            |       0.00 |
| nagiosql           | tbl_lnkServiceescalationToContact            |       0.00 |
| nagiosql           | tbl_lnkContactToContactgroup                 |       0.00 |
| nagios             | nagios_serviceescalation_contactgroups       |       0.00 |
| nagios             | nagios_hostchecks                            |       0.00 |
| mysql              | plugin                                       |       0.00 |
| nagiosql           | tbl_lnkHostescalationToHostgroup             |       0.00 |
| nagios             | nagios_contact_addresses                     |       0.00 |
| mysql              | ndb_binlog_index                             |       0.00 |
| nagios             | nagios_hostescalations                       |       0.00 |
| nagiosql           | tbl_lnkTimeperiodToTimeperiod                |       0.00 |
| nagiosql           | tbl_lnkHostescalationToHost                  |       0.00 |
| nagios             | nagios_servicedependencies                   |       0.00 |
| information_schema | COLUMN_PRIVILEGES                            |       0.00 |
| information_schema | PROFILING                                    |       0.00 |
| information_schema | INNODB_CMP_RESET                             |       0.00 |
| information_schema | INNODB_CMPMEM_RESET                          |       0.00 |
| performance_schema | performance_timers                           |       0.00 |
| performance_schema | events_waits_current                         |       0.00 |
| mysql              | general_log                                  |       0.00 |
| information_schema | INNODB_CMP                                   |       0.00 |
| performance_schema | mutex_instances                              |       0.00 |
| performance_schema | cond_instances                               |       0.00 |
| information_schema | USER_STATISTICS                              |       0.00 |
| information_schema | COLLATION_CHARACTER_SET_APPLICABILITY        |       0.00 |
| information_schema | USER_PRIVILEGES                              |       0.00 |
| information_schema | COLLATIONS                                   |       0.00 |
| information_schema | INNODB_SYS_INDEXES                           |       0.00 |
| information_schema | CLIENT_STATISTICS                            |       0.00 |
| information_schema | INNODB_SYS_FOREIGN                           |       0.00 |
| information_schema | TABLE_STATISTICS                             |       0.00 |
| information_schema | CHARACTER_SETS                               |       0.00 |
| information_schema | INNODB_BUFFER_PAGE                           |       0.00 |
| information_schema | INNODB_SYS_STATS                             |       0.00 |
| information_schema | TABLE_PRIVILEGES                             |       0.00 |
| information_schema | INNODB_BUFFER_POOL_STATS                     |       0.00 |
| information_schema | TABLESPACES                                  |       0.00 |
| information_schema | INNODB_SYS_COLUMNS                           |       0.00 |
| information_schema | TABLE_CONSTRAINTS                            |       0.00 |
| performance_schema | file_summary_by_instance                     |       0.00 |
| information_schema | INNODB_BUFFER_PAGE_LRU                       |       0.00 |
| information_schema | KEY_COLUMN_USAGE                             |       0.00 |
| information_schema | TABLES                                       |       0.00 |
| information_schema | INNODB_SYS_FIELDS                            |       0.00 |
| information_schema | INNODB_SYS_FOREIGN_COLS                      |       0.00 |
| information_schema | KEY_CACHES                                   |       0.00 |
| mysql              | slow_log                                     |       0.00 |
| information_schema | STATISTICS                                   |       0.00 |
| information_schema | INNODB_SYS_TABLES                            |       0.00 |
| performance_schema | file_summary_by_event_name                   |       0.00 |
| information_schema | INNODB_TABLE_STATS                           |       0.00 |
| information_schema | INDEX_STATISTICS                             |       0.00 |
| information_schema | SESSION_VARIABLES                            |       0.00 |
| information_schema | INNODB_BUFFER_POOL_PAGES_BLOB                |       0.00 |
| information_schema | INNODB_INDEX_STATS                           |       0.00 |
| performance_schema | file_instances                               |       0.00 |
| information_schema | SESSION_STATUS                               |       0.00 |
| information_schema | INNODB_LOCKS                                 |       0.00 |
| information_schema | INNODB_LOCK_WAITS                            |       0.00 |
| performance_schema | threads                                      |       0.00 |
| performance_schema | events_waits_summary_global_by_event_name    |       0.00 |
| information_schema | XTRADB_ADMIN_COMMAND                         |       0.00 |
| information_schema | GLOBAL_VARIABLES                             |       0.00 |
| information_schema | INNODB_BUFFER_POOL_PAGES_INDEX               |       0.00 |
| information_schema | INNODB_SYS_TABLESTATS                        |       0.00 |
| performance_schema | setup_timers                                 |       0.00 |
| performance_schema | events_waits_summary_by_thread_by_event_name |       0.00 |
| information_schema | GLOBAL_STATUS                                |       0.00 |
| information_schema | SCHEMA_PRIVILEGES                            |       0.00 |
| information_schema | INNODB_CMPMEM                                |       0.00 |
| performance_schema | setup_instruments                            |       0.00 |
| performance_schema | events_waits_summary_by_instance             |       0.00 |
| information_schema | FILES                                        |       0.00 |
| information_schema | SCHEMATA                                     |       0.00 |
| information_schema | INNODB_TRX                                   |       0.00 |
| information_schema | INNODB_UNDO_LOGS                             |       0.00 |
| performance_schema | setup_consumers                              |       0.00 |
| performance_schema | events_waits_history_long                    |       0.00 |
| information_schema | INNODB_BUFFER_POOL_PAGES                     |       0.00 |
| performance_schema | rwlock_instances                             |       0.00 |
| performance_schema | events_waits_history                         |       0.00 |
| information_schema | ENGINES                                      |       0.00 |
| information_schema | REFERENTIAL_CONSTRAINTS                      |       0.00 |
| information_schema | INNODB_CHANGED_PAGES                         |       0.00 |
| information_schema | INNODB_RSEG                                  |       0.00 |
+--------------------+----------------------------------------------+------------+
[root@cdeatlnagios01 ~]# mysql -u root -pnagiosxi -e "show global status like '%used_connections%'; show variables like 'max_connections';"
+----------------------+-------+
| Variable_name        | Value |
+----------------------+-------+
| Max_used_connections | 113   |
+----------------------+-------+
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 151   |

cat /var/log/mariadb/mariadb.log

Code: Select all

190726 12:59:13 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
190726 12:59:44 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190726 12:59:44 [Note] /usr/libexec/mysqld (mysqld 5.5.60-MariaDB) starting as process 2249 ...
190726 12:59:44 InnoDB: The InnoDB memory heap is disabled
190726 12:59:44 InnoDB: Mutexes and rw_locks use GCC atomic builtins
190726 12:59:44 InnoDB: Compressed tables use zlib 1.2.7
190726 12:59:44 InnoDB: Using Linux native AIO
190726 12:59:44 InnoDB: Initializing buffer pool, size = 128.0M
190726 12:59:44 InnoDB: Completed initialization of buffer pool
190726 12:59:44 InnoDB: highest supported file format is Barracuda.
190726 12:59:44  InnoDB: Waiting for the background threads to start
190726 12:59:45 Percona XtraDB (http://www.percona.com) 5.5.59-MariaDB-38.11 started; log sequence number 1773850689
190726 12:59:45 [Note] Plugin 'FEEDBACK' is disabled.
190726 12:59:45 [Note] Server socket created on IP: '0.0.0.0'.
190726 12:59:45 [Note] Event Scheduler: Loaded 0 events
190726 12:59:45 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.60-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
190726 13:43:11 [Note] /usr/libexec/mysqld: Normal shutdown
190726 13:43:11 [Note] Event Scheduler: Purging the queue. 0 events
190726 13:43:11  InnoDB: Starting shutdown...
190726 13:43:15  InnoDB: Shutdown completed; log sequence number 1774484134
190726 13:43:15 [Note] /usr/libexec/mysqld: Shutdown complete

190726 13:43:16 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
190726 13:43:59 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190726 13:43:59 [Note] /usr/libexec/mysqld (mysqld 5.5.60-MariaDB) starting as process 2242 ...
190726 13:43:59 InnoDB: The InnoDB memory heap is disabled
190726 13:43:59 InnoDB: Mutexes and rw_locks use GCC atomic builtins
190726 13:43:59 InnoDB: Compressed tables use zlib 1.2.7
190726 13:43:59 InnoDB: Using Linux native AIO
190726 13:43:59 InnoDB: Initializing buffer pool, size = 128.0M
190726 13:43:59 InnoDB: Completed initialization of buffer pool
190726 13:43:59 InnoDB: highest supported file format is Barracuda.
190726 13:44:00  InnoDB: Waiting for the background threads to start
190726 13:44:01 Percona XtraDB (http://www.percona.com) 5.5.59-MariaDB-38.11 started; log sequence number 1774484134
190726 13:44:01 [Note] Plugin 'FEEDBACK' is disabled.
190726 13:44:01 [Note] Server socket created on IP: '0.0.0.0'.
190726 13:44:01 [Note] Event Scheduler: Loaded 0 events
190726 13:44:01 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.60-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server

[tgriep]

Try this, go to this menu Admin > Deadpool Settings, is it enabled?
If so disable it and see if the /usr/local/nagiosxi/var/cmdsubsys.log stops cycling through the Apply Configs.

If it is enabled, it is probably removing a host or service which is causing a bad configuration, then the system detects it, restores a good one.
Then the next time the Deadpool component runs, is does the same thing, removes the host or service and the process repeats forever.

The tables sizes don't look to bad. The statehistory table is large but on a larger busy system, that could be normal.

One thing I suggest, the Max Connections should be adjusted to a larger value to be safe in case that is the issue.
See this article for instructions.
https://support.nagios.com/kb/article/n ... s-513.html

[ScottMc]

That deadpool setting did the trick! Such a strange issue. I went ahead and make the recommended settings change to the max connections as a preventative measure. Thanks!

[mbellerue]

Glad to hear it's working!