Mysql and postgresql backups not updating

[plakshmi]

We have 4 production servers running Nagios XI 2.9. We have also moved mysql and postgresql backups to /u01/backups directory by changing the backup location in below scripts.

/root/scripts/autopostgresqlbackup
/root/scripts/automysqlbackup

We observed that the mysql and postgresql backups are not updating in daily, monthly and weekly folders since March 2014 on one of our production servers. The other 3 servers are working fine.

Could you please help us on this?

[scottwilkerson]

Can you post the output of the following

Code: Select all

cat /etc/cron.d/nagiosxi|head -6
cat /var/log/cron|grep auto

[plakshmi]

Below is the output.

[q3t5qcc4@usa7061lv981 ~]$ cat /etc/cron.d/nagiosxi|head -6
# /etc/cron.d/nagiosxi: crontab fragment for nagiosxi

# Backup MySQL & PostgreSQL Databases
0 7 * * * root /root/scripts/automysqlbackup
0 8 * * * root /root/scripts/autopostgresqlbackup
____________________________________________________

tmproot@usa7061lv981>cat /var/log/cron|grep auto
Mar 9 07:00:01 usa7061lv981 CROND[21844]: (root) CMD ( /root/scripts/automysqlbackup)
Mar 9 08:00:01 usa7061lv981 CROND[6013]: (root) CMD ( /root/scripts/autopostgresqlbackup)

[scottwilkerson]

tmproot@usa7061lv981>cat /var/log/cron|grep auto
Mar 9 07:00:01 usa7061lv981 CROND[21844]: (root) CMD ( /root/scripts/automysqlbackup)
Mar 9 08:00:01 usa7061lv981 CROND[6013]: (root) CMD ( /root/scripts/autopostgresqlbackup)

This suggests that crond hasn't run the script since Mar 9
Is crond running

Code: Select all

service crond status

What does the end of the cron log look like?

Code: Select all

tail -20 /var/log/cron

[plakshmi]

Yes, crond is running

tmproot@usa7061lv981>service crond status
crond (pid 7798) is running...

Please find output below for tail -20 /var/log/cron

tmproot@usa7061lv981>tail -20 /var/log/cron
Jun 9 02:02:01 usa7061lv981 CROND[6559]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Jun 9 02:02:01 usa7061lv981 CROND[6560]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Jun 9 02:02:01 usa7061lv981 CROND[6561]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Jun 9 02:02:01 usa7061lv981 CROND[6566]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7150]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7151]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7152]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7154]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7155]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7156]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7157]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Jun 9 02:03:01 usa7061lv981 CROND[7158]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7890]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7891]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7892]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7893]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7895]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7896]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7897]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Jun 9 02:04:01 usa7061lv981 CROND[7898]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)

[slansing]

Do you have the backups going to a separate share? Or mounted directory? Is the directory full? Could you give us a listing of it's permissions?

[plakshmi]

Yes, we have changed the location of backups to /u01 mountpoint. Please find path below for postgresql and mysql

#BACKUPDIR="/store/backups/postgresql"
BACKUPDIR="/u01/backups/postgresql"

#BACKUPDIR="/store/backups/mysql"
BACKUPDIR="/u01/backups/mysql"

/u01 mountpoint is not full

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup01-LogVolU01 60G 14G 43G 24% /u01

drwxrwxr-x+ 8 root root 4.0K May 29 07:16 u01

tmproot@usa7061lv981>pwd
/u01/backups

tmproot@usa7061lv981>ls -lht
drwxr-xr-x 5 root root 4.0K Mar 15 05:11 postgresql
drwxr-xr-x 5 root root 4.0K Mar 9 07:00 mysql


Please let us know if you require any further info

[lmiltchev]

The permissions look fine. Did the "auto-backup" cron jobs run yesterday?

Code: Select all

cat /var/log/cron | grep auto | grep "Jun 11"

Run the following commands, and show us the output:

Code: Select all

grep -i failed /var/log/cron
chage -l nagios

If you run the scripts manually, do you get the backups in the "/u01/backups/postgresql" and "/u01/backups/mysql" directories?

[plakshmi]

The auto-backup cron jobs did not run today as well. We see below errors in /var/log/cron file since March 9th ( the date when last backup was created).

tmproot@usa7061lv981>grep -i failed /var/log/cron
Mar 9 20:00:01 usa7061lv981 crond[11735]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:00:01 usa7061lv981 crond[11736]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:01:01 usa7061lv981 crond[12515]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:01:01 usa7061lv981 crond[12505]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:05:01 usa7061lv981 crond[15793]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:10:01 usa7061lv981 crond[19898]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:10:01 usa7061lv981 crond[19899]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:15:01 usa7061lv981 crond[23957]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Mar 9 20:20:01 usa7061lv981 crond[27969]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
______________________________________________________________________________________________________________________

We tried to manually run the backup scripts. Both the mysql and postgresql backups were created in daily folder path but the automysqlbackup script encountered with just a WARNING message below.

###### WARNING ######
Errors reported during AutoMySQLBackup execution.. Backup failed
Error log below..
-- Warning: Skipping the data of table mysql.event. Specify the --events option explicitly.
__________________________________________________________________________________________________________________________

As for nagios account password is non-expiry.

tmproot@usa7061lv981>chage -l nagios
Last password change : Jul 15, 2013
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 1
Maximum number of days between password change : 4294967295
Number of days of warning before password expires : 7

[sreinhardt]

Looks like it is actually the root account that is expired. Try:

Code: Select all

chage -l root