Page 1 of 1
Reduce the size of my Nagios back ups....
Posted: Wed Mar 29, 2017 2:42 pm
by JakeHatMacys
Right now the last back ups that worked were 8gb's in size and every other one I attempt to get to run ends up failing with a large directory being made instead of a functional back up...
I'm not sure if the /tmp directory is running out of space as I think it uses that to swap with to build the back up??? Regardless i'm going to guess it's host & service history that got us to this point where our back ups are so big. Is there a good utility I can run to say delete a lot of our older data?
If not what would you recommend doing to reduce our back up sizes??? (my version is 2014R2.7 but I plan on upgrading within the next 2 months if possible... but I need a current back up!)
Re: Reduce the size of my Nagios back ups....
Posted: Wed Mar 29, 2017 3:42 pm
by avandemore
Another option may be to grow your filesystem, here is a doc on that:
https://assets.nagios.com/downloads/nag ... M-Disk.pdf
What is the output of:
Code: Select all
# du -shcx /usr/local/nagios/*
# du -shcx /var/lib/*
# df -h
# df -i
Re: Reduce the size of my Nagios back ups....
Posted: Thu Mar 30, 2017 10:50 am
by SteveBeauchemin
Once upon a time - I noticed in my setup that automated backups were incomplete, but a manually run backup was okay.
If that is what you are experiencing, I was told to add one line to a php file, which then did solve my problem.
The file is /usr/local/nagiosxi/html/config.inc.php
And I added to the bottom...
Code: Select all
// prevent backup process from creating incomplete files
$cfg['backup_timeout'] = 3600;
The manually run backup worked though. Maybe try running it from the command line once and see if it works for you manually.
The file is in /usr/local/nagiosxi/scripts and is named backup_xi.sh
For your version of Nagios, I would probably run that as the root user just because.
Code: Select all
cd /usr/local/nagiosxi/scripts
./backup_xi.sh
Steve B
Re: Reduce the size of my Nagios back ups....
Posted: Thu Mar 30, 2017 3:19 pm
by cdienger
Thanks Steve! @OP let us know if more questions.
Re: Reduce the size of my Nagios back ups....
Posted: Fri Mar 31, 2017 9:26 am
by JakeHatMacys
SteveBeauchemin wrote:Once upon a time - I noticed in my setup that automated backups were incomplete, but a manually run backup was okay.
If that is what you are experiencing, I was told to add one line to a php file, which then did solve my problem.
The file is /usr/local/nagiosxi/html/config.inc.php
And I added to the bottom...
Code: Select all
// prevent backup process from creating incomplete files
$cfg['backup_timeout'] = 3600;
The manually run backup worked though. Maybe try running it from the command line once and see if it works for you manually.
The file is in /usr/local/nagiosxi/scripts and is named backup_xi.sh
For your version of Nagios, I would probably run that as the root user just because.
Code: Select all
cd /usr/local/nagiosxi/scripts
./backup_xi.sh
Steve B
Thanks for the feedback... will try it now. Root? What's that? *cries* they took away our keys a long time ago. I can't do crap without a form and a week long wait... here's hoping our ACL's set up cover it...
Re: Reduce the size of my Nagios back ups....
Posted: Fri Mar 31, 2017 9:27 am
by JakeHatMacys
I have grown the file system which I don't think is the issue (/tmp alone is 30 gigs) but the problem with your DOC is it wants me to take a back up first which isn't working
Re: Reduce the size of my Nagios back ups....
Posted: Fri Mar 31, 2017 11:38 am
by mcapra
A VM level snapshot would also be appropriate. Or if you have a backup from a previous time period, that would work also. I realize the frustration in your case, but the docs include a backup step because resizing a file system can be dangerous if done irresponsibly.
Re: Reduce the size of my Nagios back ups....
Posted: Fri Mar 31, 2017 11:40 am
by dwhitfield
In addition to what
@mcapra said, please provide output for the following commands:
Code: Select all
du -shcx /usr/local/nagios/*
du -shcx /var/lib/*
Those two commands that
@avandemore gave you will help us determine what in Nagios is so large.
Database corruption of a sort can cause this.
Please log into MySQL and run the following and provide output:
Code: Select all
SELECT table_schema "DB Name",
Round(Sum(data_length + index_length) / 1024 / 1024, 1) "DB Size in MB"
FROM information_schema.tables
GROUP BY table_schema;
Re: Reduce the size of my Nagios back ups....
Posted: Fri Mar 31, 2017 3:23 pm
by JakeHatMacys
dwhitfield wrote:In addition to what
@mcapra said, please provide output for the following commands:
Code: Select all
du -shcx /usr/local/nagios/*
du -shcx /var/lib/*
Those two commands that
@avandemore gave you will help us determine what in Nagios is so large.
Database corruption of a sort can cause this.
Please log into MySQL and run the following and provide output:
Code: Select all
SELECT table_schema "DB Name",
Round(Sum(data_length + index_length) / 1024 / 1024, 1) "DB Size in MB"
FROM information_schema.tables
GROUP BY table_schema;
I'll have to get back to you on MySQL on the selec statements. Was just issued a new laptop and need to install the software to connect to the DB.
Code: Select all
[nagios@******** scripts]$ du -shcx /usr/local/nagios/*
1.6M /usr/local/nagios/bin
16M /usr/local/nagios/etc
4.0K /usr/local/nagios/include
11M /usr/local/nagios/libexec
5.6M /usr/local/nagios/sbin
37G /usr/local/nagios/share
580K /usr/local/nagios/sshpass-1.05
376K /usr/local/nagios/sshpass-1.05.tar
536K /usr/local/nagios/sshpas.tar
40G /usr/local/nagios/var
77G total
Code: Select all
[nagios@********* scripts]$ du -shcx /var/lib/*
52K /var/lib/alternatives
du: cannot read directory `/var/lib/authconfig': Permission denied
4.0K /var/lib/authconfig
32K /var/lib/cas
du: cannot read directory `/var/lib/certmonger/requests': Permission denied
du: cannot read directory `/var/lib/certmonger/local': Permission denied
du: cannot read directory `/var/lib/certmonger/cas': Permission denied
16K /var/lib/certmonger
du: cannot read directory `/var/lib/dav': Permission denied
4.0K /var/lib/dav
8.0K /var/lib/dbus
4.0K /var/lib/DeviceKit-power
4.0K /var/lib/dhclient
4.0K /var/lib/fprint
4.0K /var/lib/games
du: cannot read directory `/var/lib/gdm': Permission denied
4.0K /var/lib/gdm
4.0K /var/lib/hiera
8.0K /var/lib/ipa-client
4.0K /var/lib/logrotate.status
4.0K /var/lib/misc
du: cannot read directory `/var/lib/mlocate': Permission denied
4.0K /var/lib/mlocate
4.0K /var/lib/mrtg
du: cannot read directory `/var/lib/mysql/mysql': Permission denied
du: cannot read directory `/var/lib/mysql/nagiosql': Permission denied
du: cannot read directory `/var/lib/mysql/test': Permission denied
du: cannot read directory `/var/lib/mysql/nagios': Permission denied
21M /var/lib/mysql
du: cannot read directory `/var/lib/net-snmp/mib_indexes': Permission denied
12K /var/lib/net-snmp
du: cannot read directory `/var/lib/nfs/statd': Permission denied
12K /var/lib/nfs
8.0K /var/lib/ntp
du: cannot read directory `/var/lib/pgsql': Permission denied
4.0K /var/lib/pgsql
du: cannot read directory `/var/lib/php/session': Permission denied
500K /var/lib/php
8.0K /var/lib/plymouth
du: cannot read directory `/var/lib/polkit-1': Permission denied
4.0K /var/lib/polkit-1
du: cannot read directory `/var/lib/postfix': Permission denied
4.0K /var/lib/postfix
12K /var/lib/prelink
du: cannot read directory `/var/lib/puppet': Permission denied
4.0K /var/lib/puppet
32K /var/lib/qpidd
4.0K /var/lib/random-seed
36K /var/lib/readahead
8.0K /var/lib/rhncfg
20K /var/lib/rhsm
107M /var/lib/rpm
du: cannot read directory `/var/lib/rsyslog': Permission denied
4.0K /var/lib/rsyslog
du: cannot read directory `/var/lib/samba/winbindd_privileged': Permission denied
du: cannot read directory `/var/lib/samba/private': Permission denied
16K /var/lib/samba
du: cannot read directory `/var/lib/sss/pipes/private': Permission denied
du: cannot read directory `/var/lib/sss/keytabs': Permission denied
du: cannot read directory `/var/lib/sss/db': Permission denied
36K /var/lib/sss
12K /var/lib/stateless
4.0K /var/lib/supportinfo
8.0K /var/lib/udev
4.0K /var/lib/up2date
12K /var/lib/uptrack
12K /var/lib/vmware
8.0K /var/lib/xkb
du: cannot read directory `/var/lib/yum/history/2013-02-25/17': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/28': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/20': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/16': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/10': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/6': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/31': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/26': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/13': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/7': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/23': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/29': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/30': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/11': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/14': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/25': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/2': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/32': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/1': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/5': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/3': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/15': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/18': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/33': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/8': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/27': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/4': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/24': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/12': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/22': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/21': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/9': Permission denied
du: cannot read directory `/var/lib/yum/history/2013-02-25/19': Permission denied
13M /var/lib/yum
140M total
[nagios@******** scripts]$
And just to note when I try and repair the DB this is happening:
Code: Select all
[nagios@******* scripts]$ ./repair_databases.sh
mysqld (pid 6427) is running...
mysqld (pid 6427) is running...
DATABASE: nagios
TABLE:
/usr/local/nagiosxi/scripts/repairmysql.sh: line 49: pushd: /var/lib/mysql/nagios: Permission denied
ERROR: Could not change to dir: /var/lib/mysql/nagios
mysqld (pid 6427) is running...
DATABASE: nagiosql
TABLE:
/usr/local/nagiosxi/scripts/repairmysql.sh: line 49: pushd: /var/lib/mysql/nagiosql: Permission denied
ERROR: Could not change to dir: /var/lib/mysql/nagiosql
Stopping ndo2db: head: cannot open `/usr/local/nagios/var/ndo2db.lock' for reading: No such file or directory
done.
Starting ndo2db: done.
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
^ And that's weird because on other boxes the back ups run fine as the Nagios user with the same permissions but I can't seem to get into those directories on this box... So I'll put in a req on that too...
Re: Reduce the size of my Nagios back ups....
Posted: Sun Apr 02, 2017 8:37 pm
by dwhitfield
What are the permissions on
/var/lib/mysql/? You say they are the same permissions, but did you check (assuming it is possible for you to check)? I know that that *should* have the same permissions, but maybe that was changed accidentally.
Since you have a custom setup, it might make sense to email
[email protected] and reference this thread. The downside is no community help, and it is always up to you, but it's going to be difficult with a custom environment without really taking a look at what is actually going on.