Recommended IOPS for nagios/mysql

[paul.jobb]

This past weekend we ran into an issue where the optimize tables function appeared to have hung and locked out our database, it first hung on the table nagios_logentries and then later on nagios_notifications, both times the mysqld daemon was stopped the tables repaired and trunctated. The process has been running longer and longer over time I have noticed(was reindexing both notifications and logenetires hourly lately and taking 4 or 5 minutes to complete). It appears there may have been some latency issues on our storage network this weekend, I have put in an inquiry to have the mysql server moved to a faster service of disk however the storage team has asked me what Nagios requirement is for in terms of IOPS? Also outside of the table optimization that appears to run frequently is there any other type of maintenance that should be performed?

our configuration is as follows
1643 Hosts
12448 active service checks running at 5 min intervals

1 - nagiosxi 2012 server 4 vcpu's 8 gb ram
1 - mysql server 2 vcpu's 2 gb ram(additional cpu added recently, server doesn't appear to be memory bound)
5 - centos mod_gearman worker servers 1 vcpu 2gb(check_wmi_plus primarily used for checks)
2 - windows mod_gearman servers 1 vcpu 2gb (run some vb scripts for ms clusters)

Thanks,
Paul

[mguthrie]

I would suggest optimizing your table trimming intervals for these two tables. Only store this data as long as you absolutely need it. The trimming intervals can be set from the Admin->Performance Settings->Database(tab). With that many service checks those tables will get large pretty quickly.

[paul.jobb]

Thanks,

Does the nagios service need to be restarted after those settings are changed? any other service need to be restarted?

[scottwilkerson]

Nope. These values are read in and processed via cron every 5 minutes

[paul.jobb]

When I change max notifications age to 7 update and go back in after , it seems to reset to 90, Is there a file I can set this in?

[scottwilkerson]

Yes, this was a bug that was fixed and will be included in 2012R1.3 or you can unzip the attached and place in /usr/local/nagiosxi/html/admin

Then you will be able to make the changes.