SSH Backup gets failed

[pavan509]

Hello Team,

I am receiving the email notification stating that "SSH backups are getting failed". Can someone assist on how to fix this issue?

The scheduled SSH backup at 2024-07-07 01:30:03 failed.

View your full scheduled backup logs at /usr/local/nagiosxi/var/components/scheduledbackups.log to find out more details about the error. Your backup may have timed out, you can set a backup_timeout variable in config.inc.php to increase it.

Upon checking the log, I could see the below message.

Code: Select all

\nStarting new backup....\n
tr: write error: Broken pipe
tr: write error
Backing up Nagios Core...
tar: Removing leading `/' from member names
tar: /usr/local/nagios/share/perfdata/qr-con-3148-emea-fr-mop: file changed as we read it
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
tar: /usr/local/nagios/var: file changed as we read it
Backing up Nagios XI...
tar: Removing leading `/' from member names
tar: /usr/local/nagiosxi/tmp/phpmailer.log: file changed as we read it
tar: Removing leading `/' from hard link targets
Backing up MRTG...
tar: Removing leading `/' from member names
Backing up the SNMP directories
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up NRDP...
tar: Removing leading `/' from member names
Backing up Nagvis...
tar: Removing leading `/' from member names
Backing up nagios user home dir...
tar: Removing leading `/' from member names
tar: /home/nagios/output.txt: file changed as we read it
Backing up MySQL databases...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.
Backing up cronjobs for Apache...
Backing up logrotate config files...
Backing up Apache config files...
Compressing backup...

gzip: stdout: No space left on device
tar: nagiosxi.1718838004.tar.gz: Wrote only 6144 of 10240 bytes
tar: Child returned status 1
tar: Error is not recoverable: exiting now

===============
BACKUP COMPLETE
===============
Backup stored in /store/backups/nagiosxi/nagiosxi.1718838004.tar.gz

I tried to do the backup manually but it also resulted into failure.

Code: Select all

Backing up MySQL databases...
+ mkdir -p /store/backups/nagiosxi/xi-backup-1720418421-5b2unzg1KMjy82d3u96B/mysql
+ [[ localhost == *\:* ]]
+ ndoutils_dbport=3306
+ ndoutils_dbserver=localhost
+ mysqldump --column-statistics=0 -h localhost --port=3306 -u ndoutils --password=n@gweb --add-drop-database -B nagios
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: Got error: 145: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired when using LOCK TABLES
+ res=2
+ '[' 2 '!=' 0 ']'
+ echo 'Error backing up MySQL database '\''nagios'\'' - check the password in this script!'
Error backing up MySQL database 'nagios' - check the password in this script!
+ rm -r /store/backups/nagiosxi/xi-backup-1720418421-5b2unzg1KMjy82d3u96B
+ exit 2

[kg2857]

gzip: stdout: No space left on device

You're using all the space on the filesystem.

[cnorell]

pavan509,

As kg2857 pointed out, you're out of disk space. We have an article on some possible solutions to lack of disk space. The article is for a virtual machine, but the same concepts apply to hardware servers.

In short, you have a few options:

• Expand the disk

• Remove/move existing backups

• Remove/move existing log files

• Remove/move archived log files, located at:
  

  Code: Select all

  /usr/local/nagiosxi/var/archives/

  Hopefully something here gets you on the right track. Let us know if you want any more information on any specific approach.
  
  Best Regards,
  
  Cory Norell

[pavan509]

I don't see any major space issues below is the output

Code: Select all

 df -h
Filesystem                          Size  Used Avail Use% Mounted on
devtmpfs                            4.0M     0  4.0M   0% /dev
tmpfs                               7.6G     0  7.6G   0% /dev/shm
tmpfs                               3.1G  314M  2.7G  11% /run
/dev/nvme0n1p4                       45G   37G  8.3G  82% /
/dev/nvme0n1p3                      495M  411M   85M  83% /boot
/dev/nvme0n1p2                      200M   20K  200M   1% /efi
/dev/mapper/vg_store-lv_store        20G  845M   20G   5% /store
/dev/mapper/vg_nagios-lv_nagiosapp  800G  320G  480G  41% /nagiosapp
/dev/mapper/vg_nagios-lv_nagiosdb   100G   59G   42G  59% /nagiosdb
tmpfs                               1.6G  4.0K  1.6G   1% /run/user/3060
tmpfs                               1.6G  4.0K  1.6G   1% /run/user/1044
tmpfs                               1.6G  4.0K  1.6G   1% /run/user/1043

Code: Select all

du -sh /usr/local/nagios/var/
1.1G    /usr/local/nagios/var/

Code: Select all

du -sh /usr/local/nagiosxi/var/
303M    /usr/local/nagiosxi/var/

[cnorell]

Hmm, interesting. Sometimes "No space left on device" can really be caused by permissions issues. When trying to manually run the backup, are you doing it as root?

Tangentially, did your backup directory's (defaults to /store/backups/nagiosxi/) permissions change? The directory should have the following permissions:

Code: Select all

drwxr-xr-x. 2 nagios nagios 31 Jun  3 15:54 nagiosxi

Best Regards,

Cory Norell