Page 1 of 2
Mysql and postgresql backups not updating
Posted: Thu Jun 05, 2014 3:09 am
by plakshmi
We have 4 production servers running Nagios XI 2.9. We have also moved mysql and postgresql backups to /u01/backups directory by changing the backup location in below scripts.
/root/scripts/autopostgresqlbackup
/root/scripts/automysqlbackup
We observed that the mysql and postgresql backups are not updating in daily, monthly and weekly folders since March 2014 on one of our production servers. The other 3 servers are working fine.
Could you please help us on this?
Re: Mysql and postgresql backups not updating
Posted: Thu Jun 05, 2014 2:39 pm
by scottwilkerson
Can you post the output of the following
Code: Select all
cat /etc/cron.d/nagiosxi|head -6
cat /var/log/cron|grep auto
Re: Mysql and postgresql backups not updating
Posted: Fri Jun 06, 2014 3:37 am
by plakshmi
Below is the output.
[q3t5qcc4@usa7061lv981 ~]$ cat /etc/cron.d/nagiosxi|head -6
# /etc/cron.d/nagiosxi: crontab fragment for nagiosxi
# Backup MySQL & PostgreSQL Databases
0 7 * * * root /root/scripts/automysqlbackup
0 8 * * * root /root/scripts/autopostgresqlbackup
____________________________________________________
tmproot@usa7061lv981>cat /var/log/cron|grep auto
Mar 9 07:00:01 usa7061lv981 CROND[21844]: (root) CMD ( /root/scripts/automysqlbackup)
Mar 9 08:00:01 usa7061lv981 CROND[6013]: (root) CMD ( /root/scripts/autopostgresqlbackup)
Re: Mysql and postgresql backups not updating
Posted: Fri Jun 06, 2014 9:28 am
by scottwilkerson
plakshmi wrote:tmproot@usa7061lv981>cat /var/log/cron|grep auto
Mar 9 07:00:01 usa7061lv981 CROND[21844]: (root) CMD ( /root/scripts/automysqlbackup)
Mar 9 08:00:01 usa7061lv981 CROND[6013]: (root) CMD ( /root/scripts/autopostgresqlbackup)
This suggests that crond hasn't run the script since Mar 9
Is crond running
What does the end of the cron log look like?
Re: Mysql and postgresql backups not updating
Posted: Mon Jun 09, 2014 1:09 am
by plakshmi
Yes, crond is running
tmproot@usa7061lv981>service crond status
crond (pid 7798) is running...
Please find output below for tail -20 /var/log/cron
tmproot@usa7061lv981>tail -20 /var/log/cron
Jun 9 02:02:01 usa7061lv981 CROND[6559]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Jun 9 02:02:01 usa7061lv981 CROND[6560]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Jun 9 02:02:01 usa7061lv981 CROND[6561]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Jun 9 02:02:01 usa7061lv981 CROND[6566]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7150]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7151]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7152]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7154]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7155]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7156]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7157]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7158]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7890]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7891]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7892]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7893]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7895]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7896]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7897]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7898]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Re: Mysql and postgresql backups not updating
Posted: Mon Jun 09, 2014 4:53 pm
by slansing
Do you have the backups going to a separate share? Or mounted directory? Is the directory full? Could you give us a listing of it's permissions?
Re: Mysql and postgresql backups not updating
Posted: Thu Jun 12, 2014 6:48 am
by plakshmi
Yes, we have changed the location of backups to /u01 mountpoint. Please find path below for postgresql and mysql
#BACKUPDIR="/store/backups/postgresql"
BACKUPDIR="/u01/backups/postgresql"
#BACKUPDIR="/store/backups/mysql"
BACKUPDIR="/u01/backups/mysql"
/u01 mountpoint is not full
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup01-LogVolU01 60G 14G 43G 24% /u01
drwxrwxr-x+ 8 root root 4.0K May 29 07:16 u01
tmproot@usa7061lv981>pwd
/u01/backups
tmproot@usa7061lv981>ls -lht
drwxr-xr-x 5 root root 4.0K Mar 15 05:11 postgresql
drwxr-xr-x 5 root root 4.0K Mar 9 07:00 mysql
Please let us know if you require any further info
Re: Mysql and postgresql backups not updating
Posted: Thu Jun 12, 2014 3:24 pm
by lmiltchev
The permissions look fine. Did the "auto-backup" cron jobs run yesterday?
Code: Select all
cat /var/log/cron | grep auto | grep "Jun 11"
Run the following commands, and show us the output:
Code: Select all
grep -i failed /var/log/cron
chage -l nagios
If you run the scripts manually, do you get the backups in the "/u01/backups/postgresql" and "/u01/backups/mysql" directories?
Re: Mysql and postgresql backups not updating
Posted: Fri Jun 13, 2014 5:11 am
by plakshmi
The auto-backup cron jobs did not run today as well. We see below errors in /var/log/cron file since March 9th ( the date when last backup was created).
tmproot@usa7061lv981>grep -i failed /var/log/cron
Mar 9 20:00:01 usa7061lv981 crond[11735]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:00:01 usa7061lv981 crond[11736]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:01:01 usa7061lv981 crond[12515]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:01:01 usa7061lv981 crond[12505]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:05:01 usa7061lv981 crond[15793]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:10:01 usa7061lv981 crond[19898]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:10:01 usa7061lv981 crond[19899]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:15:01 usa7061lv981 crond[23957]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:20:01 usa7061lv981 crond[27969]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
______________________________________________________________________________________________________________________
We tried to manually run the backup scripts. Both the mysql and postgresql backups were created in daily folder path but the automysqlbackup script encountered with just a WARNING message below.
###### WARNING ######
Errors reported during AutoMySQLBackup execution.. Backup failed
Error log below..
-- Warning: Skipping the data of table mysql.event. Specify the --events option explicitly.
__________________________________________________________________________________________________________________________
As for nagios account password is non-expiry.
tmproot@usa7061lv981>chage -l nagios
Last password change : Jul 15, 2013
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 1
Maximum number of days between password change : 4294967295
Number of days of warning before password expires : 7
Re: Mysql and postgresql backups not updating
Posted: Fri Jun 13, 2014 1:01 pm
by sreinhardt
Looks like it is actually the root account that is expired. Try: