Nagios Backups

[rgoldthwaite]

Hey All,

I'm a relatively new Nagios admin taking over for someone here at my company.

I am noticing some interesting behavior in backup sizes every night. Over the course of a couple months every day the backup file is getting larger and larger. For example:

Last Tuesdays backup was about 1.6GB. This Tuesday's backup was 2.0 GB and I can see sequentially each day they are growing. but we aren't adding new services or hosts to the box.

Anyone know of any DB cleanup or some method that keeps our install/database tidy and in check?

This feels like it could be a database issue but not sure where to start on how to troubleshoot.

NagiosXI 2012R2.5

Thanks!

[slansing]

Hmm, interesting, have you opened the backup tarballs yet to compare file sizes? My guess is it is something that can fluctuate, such as a series of log files, but 400MB a week in growth is pretty astounding.

[rgoldthwaite]

So i've done a bit of digging. looks like the directory /usr/local/nagios/var/spool/xidpe is about 17GB in size. pretty sure this is the directory that's growing. looks like there is a php script that runs via cron that moves the files from that folder to /usr/local/nagios/var/spool/perfdata but i dont think thats happening because if i look at the log file for the cronjob, i see this:

Outbound data DISABLED Thu, 29 Jan 2015 12:59:01 -0500
sh: /bin/mv: Argument list too long
sh: /bin/mv: Argument list too long

I'm pretty sure this is the piece of code it's complaining about: (clip from php script):

//only process files if outbound transfers are enabled
if($enable_outbound) {
$n=0;
$n+=process_data_files();
$t+=$n;
}
else { //simple bulk file move so pnp picks up perfdata
$cmd = 'mv '.$xidpe.'* '.$dest;
exec($cmd);
}

[tmcdonald]

Ahh, yes. The xidpe directory is so full that no operations can be run on it. The solution, unfortunately, is to clear out that directory which will lose some historical performance data. If you are okay with this, you will need to run the following command:

Code: Select all

find /usr/local/nagios/var/spool/xidpe -type f -delete

This may take a while to run, but when it is done the directory will be clear.

[rgoldthwaite]

Should i run this with the nagios service stopped? or can i run this with the system up?

[slansing]

You can run it while the service is running, or stopped, it's still going to grab the stuff that comes in as it is finishing up.