Nagios Hangs when using Hostgroups for downtime

[brucej543]

This is an on-going issue with our Nagios system. When going into or coming out of a schedule downtime for a group of hosts, the system locks up with 100% CPU usage in which to clear requires a force reboot or cancelling nagios. Can we please set up someone to review the complete Nagios and MariaDB configurations and settings to determine what this issue is. We have tried multiple things including creating smaller hostgroups but get the same results. The system/application/DB can't seem to handle hundreds of hosts and thousands of services going into a downtime at one time. There are multiple post with various issue on the performance issues.

Please advise on getting a complete review system.

Thank you

[cdienger]

Let's start with getting a profile from the system. A profile can be generated under Admin > System Config > System Profile > Download Profile, or from the command line with:

Code: Select all

/usr/local/nagiosxi/scripts/components/getprofile.sh 59913

The profile is then saved to:

Code: Select all

/usr/local/nagiosxi/var/components/profile.zip

I'd also like to get a copy of the /etc/my.cnf.

Please PM both of them to me.

[brucej543]

Profile and my.cnf files sent in PM

[brucej543]

Update to this issue. Worked by my manager who has DB backgound and made several changes to the my.cnf file to increase the amount of memory that mariadb uses and I also updated the system to increase the number of files that can opened by mariadb to 1024000
Below is the revised my.cnf file. Ran full test of downtime with 1203 hosts and 7620 services and the system did not lock up or hang

[root@bcnagios01 etc]# cat my.cnf
[mysqld]
innodb_file_per_table=1

query_cache_size=16M
query_cache_limit=4M
tmp_table_size=64M
max_heap_table_size=64M
key_buffer_size=32M
table_open_cache=32
#Below settings added 08/31/20
max_connections=1000
log_warnings=4
# Changes added 9/4/2020
innodb_buffer_pool_instances=4
innodb_buffer_pool_size=4294967296

datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

[mysqld_safe]
# Changes added 9/4/2020
open_files_limit=1024000
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

[scottwilkerson]

Update to this issue. Worked by my manager who has DB backgound and made several changes to the my.cnf file to increase the amount of memory that mariadb uses and I also updated the system to increase the number of files that can opened by mariadb to 1024000
Below is the revised my.cnf file. Ran full test of downtime with 1203 hosts and 7620 services and the system did not lock up or hang

[root@bcnagios01 etc]# cat my.cnf
[mysqld]
innodb_file_per_table=1

query_cache_size=16M
query_cache_limit=4M
tmp_table_size=64M
max_heap_table_size=64M
key_buffer_size=32M
table_open_cache=32
#Below settings added 08/31/20
max_connections=1000
log_warnings=4
# Changes added 9/4/2020
innodb_buffer_pool_instances=4
innodb_buffer_pool_size=4294967296

datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

[mysqld_safe]
# Changes added 9/4/2020
open_files_limit=1024000
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

Awesome, glad you were able to get it sorted out and thanks for sharing the config for others!

Locking thread