Page 1 of 1
Database repair and /var/tmp filling up
Posted: Thu May 10, 2018 7:39 am
by DFaught
Hi,
I am a Nagios admin entering this on behalf of our platform support group that maintains the OS and software for Nagios XI.
Attached are files that describe the problem and the system profile.
Thank you for your help.
Regards,
Dave
Re: Database repair and /var/tmp filling up
Posted: Thu May 10, 2018 8:33 am
by rkennedy
What files appear to be filling up the /var/tmp directory? I looked at your profile, and things seem to be OK for disk:
Code: Select all
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 15G 2.4G 13G 16% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 0 16G 0% /sys/fs/cgroup
tmpfs 16G 401M 16G 3% /run
/dev/sda1 1014M 270M 745M 27% /boot
/dev/mapper/rhel-var 7.0G 5.4G 1.7G 77% /var
/dev/mapper/datavg-var_lib_mysql 150G 9.2G 141G 7% /var/lib/mysql
/dev/mapper/rhel-var_tmp 13G 34M 13G 1% /var/tmp
/dev/mapper/rhel-var_log 4.5G 2.6G 2.0G 57% /var/log
/dev/mapper/rhel-var_log_audit 1014M 74M 941M 8% /var/log/audit
/dev/mapper/rhel-home 2.0G 109M 1.9G 6% /home
/dev/mapper/datavg-store 75G 9.9G 66G 14% /store
/dev/mapper/datavg-data_trans 253M 14M 240M 6% /data/trans
/dev/mapper/datavg-usr_local_nagios 25G 20G 5.4G 79% /usr/local/nagios
/dev/mapper/datavg-opt 509M 262M 248M 52% /opt
/dev/mapper/datavg-opt_bit9 1014M 74M 941M 8% /opt/bit9
/dev/mapper/datavg-opt_chef 1014M 228M 787M 23% /opt/chef
/dev/mapper/datavg-opt_tivoli 1014M 762M 253M 76% /opt/tivoli
/dev/mapper/datavg-opt_ftl 125M 8.0M 117M 7% /opt/ftl
/dev/mapper/datavg-opt_ctmagent 1014M 358M 657M 36% /opt/ctmagent
/dev/mapper/datavg-srv 29M 1.8M 27M 6% /srv
/dev/mapper/datavg-srv_bit9 5.0G 603M 4.4G 12% /srv/bit9
/dev/mapper/rhel-tmp 4.0G 42M 4.0G 2% /tmp
tmpfs 3.2G 0 3.2G 0% /run/user/4847
The SQL repair can be needed if things become corrupt - usually not very often though. If it's happening consistently, then you may have a further performance issue going on somewhere. With the amount of services you have, it may be worth segregating out to another instance, and look at Nagios Fusion.
Code: Select all
Total Hosts: 2153
Total Services: 32242
Re: Database repair and /var/tmp filling up
Posted: Thu May 10, 2018 8:40 am
by mcapra
Certain MySQL storage engines (I believe MyISAM is included) store temporary files in the
TMPDIR if there is a pending transaction that is taking a particularly long time, and MySQL 8 natively handles grouping/ordering requests by storing results to
TMPDIR before joining/sorting them. More info on that here:
https://dev.mysql.com/doc/refman/8.0/en ... files.html
In a nut-shell, with an environment this large certain object operations within Nagios XI could be creating rather large temp files.
As this is a particularly large environment, a good first step might be to offload MySQL to a separate machine if you haven't already done that:
https://assets.nagios.com/downloads/nag ... Server.pdf
Re: Database repair and /var/tmp filling up
Posted: Thu May 10, 2018 2:41 pm
by cdienger
@DFaught can you confirm if the directory is having issues with sql temp files? If so then offloading a
@mcpra suggested would be something to investigate for the environment. Additionally, there are other performance improvements that can be made covered in
https://assets.nagios.com/downloads/nag ... ios-XI.pdf