Reduce the size of my Nagios back ups....

[JakeHatMacys]

Right now the last back ups that worked were 8gb's in size and every other one I attempt to get to run ends up failing with a large directory being made instead of a functional back up...

I'm not sure if the /tmp directory is running out of space as I think it uses that to swap with to build the back up??? Regardless i'm going to guess it's host & service history that got us to this point where our back ups are so big. Is there a good utility I can run to say delete a lot of our older data?

If not what would you recommend doing to reduce our back up sizes??? (my version is 2014R2.7 but I plan on upgrading within the next 2 months if possible... but I need a current back up!)

[avandemore]

Another option may be to grow your filesystem, here is a doc on that:

https://assets.nagios.com/downloads/nag ... M-Disk.pdf

What is the output of:

Code: Select all

# du -shcx /usr/local/nagios/*
# du -shcx /var/lib/*
# df -h 
# df -i

[SteveBeauchemin]

Once upon a time - I noticed in my setup that automated backups were incomplete, but a manually run backup was okay.

If that is what you are experiencing, I was told to add one line to a php file, which then did solve my problem.

The file is /usr/local/nagiosxi/html/config.inc.php

And I added to the bottom...

Code: Select all

// prevent backup process from creating incomplete files
$cfg['backup_timeout'] = 3600;

The manually run backup worked though. Maybe try running it from the command line once and see if it works for you manually.
The file is in /usr/local/nagiosxi/scripts and is named backup_xi.sh
For your version of Nagios, I would probably run that as the root user just because.

Code: Select all

cd /usr/local/nagiosxi/scripts
./backup_xi.sh

Steve B

[cdienger]

Thanks Steve! @OP let us know if more questions.

[JakeHatMacys]

Once upon a time - I noticed in my setup that automated backups were incomplete, but a manually run backup was okay.

If that is what you are experiencing, I was told to add one line to a php file, which then did solve my problem.

The file is /usr/local/nagiosxi/html/config.inc.php

And I added to the bottom...

Code: Select all

// prevent backup process from creating incomplete files
$cfg['backup_timeout'] = 3600;

The manually run backup worked though. Maybe try running it from the command line once and see if it works for you manually.
The file is in /usr/local/nagiosxi/scripts and is named backup_xi.sh
For your version of Nagios, I would probably run that as the root user just because.

Code: Select all

cd /usr/local/nagiosxi/scripts
./backup_xi.sh

Steve B

Thanks for the feedback... will try it now. Root? What's that? *cries* they took away our keys a long time ago. I can't do crap without a form and a week long wait... here's hoping our ACL's set up cover it...

[JakeHatMacys]

Another option may be to grow your filesystem, here is a doc on that:

https://assets.nagios.com/downloads/nag ... M-Disk.pdf

What is the output of:

Code: Select all

# du -shcx /usr/local/nagios/*
# du -shcx /var/lib/*
# df -h 
# df -i

I have grown the file system which I don't think is the issue (/tmp alone is 30 gigs) but the problem with your DOC is it wants me to take a back up first which isn't working

[mcapra]

A VM level snapshot would also be appropriate. Or if you have a backup from a previous time period, that would work also. I realize the frustration in your case, but the docs include a backup step because resizing a file system can be dangerous if done irresponsibly.

[dwhitfield]

In addition to what @mcapra said, please provide output for the following commands:

Code: Select all

du -shcx /usr/local/nagios/*
du -shcx /var/lib/*

Those two commands that @avandemore gave you will help us determine what in Nagios is so large.

Database corruption of a sort can cause this.

Please log into MySQL and run the following and provide output:

Code: Select all

SELECT table_schema  "DB Name",
Round(Sum(data_length + index_length) / 1024 / 1024, 1) "DB Size in MB"
FROM   information_schema.tables
GROUP  BY table_schema;

[JakeHatMacys]

In addition to what @mcapra said, please provide output for the following commands:

Code: Select all

du -shcx /usr/local/nagios/*
du -shcx /var/lib/*

Those two commands that @avandemore gave you will help us determine what in Nagios is so large.

Database corruption of a sort can cause this.

Please log into MySQL and run the following and provide output:

Code: Select all

SELECT table_schema  "DB Name",
Round(Sum(data_length + index_length) / 1024 / 1024, 1) "DB Size in MB"
FROM   information_schema.tables
GROUP  BY table_schema;

I'll have to get back to you on MySQL on the selec statements. Was just issued a new laptop and need to install the software to connect to the DB.

Code: Select all

[nagios@******** scripts]$ du -shcx /usr/local/nagios/*
1.6M    /usr/local/nagios/bin
16M     /usr/local/nagios/etc
4.0K    /usr/local/nagios/include
11M     /usr/local/nagios/libexec
5.6M    /usr/local/nagios/sbin
37G     /usr/local/nagios/share
580K    /usr/local/nagios/sshpass-1.05
376K    /usr/local/nagios/sshpass-1.05.tar
536K    /usr/local/nagios/sshpas.tar
40G     /usr/local/nagios/var
77G     total

Code: Select all

[nagios@********* scripts]$ du -shcx /var/lib/*
52K     /var/lib/alternatives
du: cannot read directory `/var/lib/authconfig': Permission denied
4.0K    /var/lib/authconfig
32K     /var/lib/cas
du: cannot read directory `/var/lib/certmonger/requests': Permission denied
du: cannot read directory `/var/lib/certmonger/local': Permission denied
du: cannot read directory `/var/lib/certmonger/cas': Permission denied
16K     /var/lib/certmonger
du: cannot read directory `/var/lib/dav': Permission denied
4.0K    /var/lib/dav
8.0K    /var/lib/dbus
4.0K    /var/lib/DeviceKit-power
4.0K    /var/lib/dhclient
4.0K    /var/lib/fprint
4.0K    /var/lib/games
du: cannot read directory `/var/lib/gdm': Permission denied
4.0K    /var/lib/gdm
4.0K    /var/lib/hiera
8.0K    /var/lib/ipa-client
4.0K    /var/lib/logrotate.status
4.0K    /var/lib/misc
du: cannot read directory `/var/lib/mlocate': Permission denied
4.0K    /var/lib/mlocate
4.0K    /var/lib/mrtg
du: cannot read directory `/var/lib/mysql/mysql': Permission denied
du: cannot read directory `/var/lib/mysql/nagiosql': Permission denied
du: cannot read directory `/var/lib/mysql/test': Permission denied
du: cannot read directory `/var/lib/mysql/nagios': Permission denied
21M     /var/lib/mysql
du: cannot read directory `/var/lib/net-snmp/mib_indexes': Permission denied
12K     /var/lib/net-snmp
du: cannot read directory `/var/lib/nfs/statd': Permission denied
12K     /var/lib/nfs
8.0K    /var/lib/ntp
du: cannot read directory `/var/lib/pgsql': Permission denied
4.0K    /var/lib/pgsql
du: cannot read directory `/var/lib/php/session': Permission denied
500K    /var/lib/php
8.0K    /var/lib/plymouth
du: cannot read directory `/var/lib/polkit-1': Permission denied
4.0K    /var/lib/polkit-1
du: cannot read directory `/var/lib/postfix': Permission denied
4.0K    /var/lib/postfix
12K     /var/lib/prelink
du: cannot read directory `/var/lib/puppet': Permission denied
4.0K    /var/lib/puppet
32K     /var/lib/qpidd
4.0K    /var/lib/random-seed
36K     /var/lib/readahead
8.0K    /var/lib/rhncfg
20K     /var/lib/rhsm
107M    /var/lib/rpm
du: cannot read directory `/var/lib/rsyslog': Permission denied
4.0K    /var/lib/rsyslog
du: cannot read directory `/var/lib/samba/winbindd_privileged': Permission denied
du: cannot read directory `/var/lib/samba/private': Permission denied
16K     /var/lib/samba
du: cannot read directory `/var/lib/sss/pipes/private': Permission denied
du: cannot read directory `/var/lib/sss/keytabs': Permission denied
du: cannot read directory `/var/lib/sss/db': Permission denied
36K     /var/lib/sss
12K     /var/lib/stateless
4.0K    /var/lib/supportinfo
8.0K    /var/lib/udev
4.0K    /var/lib/up2date
12K     /var/lib/uptrack
12K     /var/lib/vmware
8.0K    /var/lib/xkb
du: cannot read directory `/var/lib/yum/history/2013-02-25/17': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/28': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/20': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/16': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/10': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/6': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/31': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/26': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/13': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/7': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/23': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/29': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/30': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/11': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/14': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/25': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/2': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/32': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/1': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/5': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/3': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/15': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/18': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/33': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/8': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/27': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/4': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/24': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/12': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/22': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/21': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/9': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/19': Permission denied
13M     /var/lib/yum
140M    total
[nagios@******** scripts]$

And just to note when I try and repair the DB this is happening:

Code: Select all

[nagios@******* scripts]$ ./repair_databases.sh
mysqld (pid  6427) is running...
mysqld (pid  6427) is running...
DATABASE: nagios
TABLE:
/usr/local/nagiosxi/scripts/repairmysql.sh: line 49: pushd: /var/lib/mysql/nagios: Permission denied
ERROR: Could not change to dir: /var/lib/mysql/nagios
mysqld (pid  6427) is running...
DATABASE: nagiosql
TABLE:
/usr/local/nagiosxi/scripts/repairmysql.sh: line 49: pushd: /var/lib/mysql/nagiosql: Permission denied
ERROR: Could not change to dir: /var/lib/mysql/nagiosql
Stopping ndo2db: head: cannot open `/usr/local/nagios/var/ndo2db.lock' for reading: No such file or directory
done.
Starting ndo2db: done.
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.

^ And that's weird because on other boxes the back ups run fine as the Nagios user with the same permissions but I can't seem to get into those directories on this box... So I'll put in a req on that too...

[dwhitfield]

What are the permissions on /var/lib/mysql/? You say they are the same permissions, but did you check (assuming it is possible for you to check)? I know that that *should* have the same permissions, but maybe that was changed accidentally.

Since you have a custom setup, it might make sense to email customersupport@nagios.com and reference this thread. The downside is no community help, and it is always up to you, but it's going to be difficult with a custom environment without really taking a look at what is actually going on.