NagiosXI - httpd.conf LoadModule

[tylerhoadley]

Hey All,

Over the past few days/weeks, I have been experiencing load spikes on our NagiosXI enterprise solution. I found one post similar to this problem, but had no solution.

iotop, and top couldn't really help identify the problem until I started thinking about RHEL/CentOS default httpd.conf and it fork configuration. Unlike debian based distro's, rhel sets up a heavy memory footprint per httpd process and then has a very high maxclient number. I reduced these setting down to "At max" use 1.5G of the overall 4G memory.

<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 48
MaxClients 48
MaxRequestsPerChild 1000
</IfModule>

using httpd tools like, apachebuddy.pl & apachetuner.sh I was able to lower the load and remove the heavy 20/20/20 load spikes that were occurring.

So now the question... with the default RHEL httpd.conf file, which modules are required to run nagiosXI2014R1.1/2 with Nagvis via apache2? What can I comment out safely to reduce memory footprint per fork as this is an enterprise solution for my place of business?

here is the list (set as default) by rhel.

root@Server ~/# cat /etc/httpd/conf/httpd.conf | grep LoadMod
# have to place corresponding `LoadModule' lines at this location so the
# LoadModule foo_module modules/mod_foo.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
LoadModule env_module modules/mod_env.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule info_module modules/mod_info.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule substitute_module modules/mod_substitute.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule version_module modules/mod_version.so
#LoadModule asis_module modules/mod_asis.so
#LoadModule authn_dbd_module modules/mod_authn_dbd.so
#LoadModule cern_meta_module modules/mod_cern_meta.so
#LoadModule cgid_module modules/mod_cgid.so
#LoadModule dbd_module modules/mod_dbd.so
#LoadModule dumpio_module modules/mod_dumpio.so
#LoadModule filter_module modules/mod_filter.so
#LoadModule ident_module modules/mod_ident.so
#LoadModule log_forensic_module modules/mod_log_forensic.so
#LoadModule unique_id_module modules/mod_unique_id.so

[sreinhardt]

So now the question... with the default RHEL httpd.conf file, which modules are required to run nagiosXI2014R1.1/2 with Nagvis via apache2? What can I comment out safely to reduce memory footprint per fork as this is an enterprise solution for my place of business?

I honestly, do not think we have a full answer to this, at this point. However since you brought it up, I made an internal testing task for it, and we will absolutely post back here once we get to testing it out. While I have some ideas as to what could be removed, I would rather give it a good test before suggesting something that might break XI for you.

[tylerhoadley]

Thanks, I'm very curious to find out. as a web admin with internal and external facing apache servers. Its best to load what is needed and remove what is not. resources, security, application deps... many factors to consider.

on a side note to the load spikes... we had an wan outage yesterday which caused a spike so didn't think to much of it. then today I noticed a small spike again (first thing in morning as others came in)which usually goes up, then again in the late morning (crap, but not nearly as high) ... so I drove right back into it for a solution. in the end, because I can't stop mysql (without change window), I ran

# mysql -u root -p

mysql> use nagios;
mysql> show tables;
mysql> optimize table $XYZ;

on both nagios/ql databases and all its tables then flushed the query cache to reduce data fragments.

between httpd forks and optimizing the tables, I'm getting lower loads then before the upgrade. I will post back if this resolves the spike problems as it still might be too soon to fully trust.

[abrist]

Fair enough. Keep us posted.

[tylerhoadley]

Just an update on my load spikes...

Implemented mod_gearman for nagios 4 has removed the spikes. but only after enabling the embedded perl. with Nagios core 4's removal of embedded perl, and the fact we use many different perl scripts for webinject and snmp checks of network bandwidth (nagvis) and up/down status on switch ports etc. this caused high loads.

Also with mod_gearman I was able to apply _WORKER=queue variable on the service templates these selected perl scripts use to divide the queue and spread the load out better without bottlenecking the main service queue that our nrpe/check_nt checks use solely now. I did some tweaking to the total workers and ttl a worker has so they re-fresh faster and only remain when needed. when using the defaults, the workers caused high loads spikes as well, but when I compare these stats to before the upgrade, I have a lower overall load, and check execution are processing faster although the check latency have gone up slightly which is expected considering its being passed to mod_gearman, executed and then returned.

Below are 2 image, one of the gearman_top with 5 virtual servicegroup queues via _WORKER= variable, and the other of the load before upgrade, post upgrade and then finally post gearman implementation with embedded perl.

Cheers,

[abrist]

I have made Eric[1] aware of this thread as he is the primary dev on this bug. Just an FYI.