Nagios Support Forum

I ran into a problem yesterday when I went to take a backup before updating our Nagios XI install with the backup failing with a permissions error. This was particularly surprising because it is configured to take a scheduled backup every day, and to email on failures, and we had not gotten any emails for failures (until the one we got when the manual backup I ran failed). I tried running the backup script from command line and got the same error.

This was the backup output:

\nStarting new backup....\n
Backing up Nagios Core...
tar: Removing leading `/' from member names
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
Backing up Nagios XI...
tar: Removing leading `/' from member names
Backing up MRTG...
tar: Removing leading `/' from member names
Backing up the SNMP directories
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up NRDP...
tar: Removing leading `/' from member names
Backing up Nagvis...
tar: Removing leading `/' from member names
Backing up nagios user home dir...
tar: Removing leading `/' from member names
Backing up MySQL databases...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: Error: 'Access denied; you need (at least one of) the PROCESS privilege(s) for this operation' when trying to dump tablespaces
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: Error: 'Access denied; you need (at least one of) the PROCESS privilege(s) for this operation' when trying to dump tablespaces
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: Error: 'Access denied; you need (at least one of) the PROCESS privilege(s) for this operation' when trying to dump tablespaces
Backing up cronjobs for Apache...
Backing up logrotate config files...
Backing up Apache config files...
Compressing backup...

===============
BACKUP COMPLETE
===============
Backup stored in /store/backups/nagiosxi/1617216057.tar.gz

After a bit of digging, I granted the process permission to each of the three users the backup script appeared to be using to connect to the MySQL database, and then retried the backup script. It no longer encountered the mysqldump permissions error, however it still emailed that the backup failed, even though the closest thing to an error in the output was the warning messages about using passwords with the command line interface. What's even stranger, is that when the scheduled backup ran later that day, it did not send a failure email, even though the output from it was identical.

I'm a bit lost on why it does not seem to be sending the email on failure for the scheduled backup, why it is still sending the email for failure when run manually when it does not seem to be failing anymore, and why it started to or was getting the permissions error to begin with. Also on a side note, is the output from the backup script archived anywhere when the script runs or is it really just overwriting the same file every time? It would be nice to be able to go through an actual log to see when it started getting this error or if it was always getting it.

Any insight or guidance is appreciated.

Hi,
I have just tried the "/usr/local/nagiosxi/scripts/backup_xi.sh" backup script as "root" and it completed just fine.

What was the "user" you used (logged in) to run this script?

Also, can you please show me the complete command you used? ..... did you use "sudo"?

I'm not sure what OS you are on, but could you please check the database's log and see if there is any crashed table that need repair.

Log location: If your database crashed, please run this command: Also, check the below logs after or during the running of the backup script with the "tail -f" command:
If you still having issue running the backup script, please upload the "profile.zip".


Regards,
Vinh

vtrac wrote:What was the "user" you used (logged in) to run this script?

Also, can you please show me the complete command you used? ..... did you use "sudo"?

The script was run as root (run using sudo).
Full command:
sudo /usr/local/nagiosxi/scripts/backup_xi.sh

It's worth noting that I first got the error running the script from the web UI (admin --> system backups --> local backup archives --> create backup). After that I tried running the script via command line to see if it was any different, but got the same error there as well.

The host is running CentOS 8. I do not see any errors being written to the database logs (/var/log/mysql/mysqld.log). I restarted both the Nagios processes and mysqld anyway, and now I am only getting the failure email when running the backup via the web UI.

I ran the backup via the web UI while tailing the files you mentioned. I did not see any errors in the nagios.log. I have attached the output from the cmdsubsys.log that I saw for the backup. It stated that the process ended with a return code of 0...

I have also attached the profile.zip.

Hi,
From what I see, your backup completed just fine.

The below were just warnings, not errors. Also, is your database local?

I do not see the database log in the "profile.zip".

What is the output of the below command on your XI machine?
Regards,
Vinh

Hi Vinh,

I agree that based on the output it seems to be completing fine and doesn't seem to be getting any errors anymore, so i'm trying to understand why it is still sending the failure email when run via the web UI.

Yes, the database is local.

$ ps -ef | grep mysql
mysql 3278033 1 3 Apr06 ? 01:00:11 /usr/libexec/mysqld --basedir=/usr
$

Hi,
Did you make changes to mysql config files?

Please upload all mysql config files so I can take a look.

After a bit of digging, I granted the process permission to each of the three users the backup script appeared to be using to connect to the MySQL database, and then retried the backup script

Based on the statement above, what are the three users you have granted access to.

Why did those user's permission changed? .... please investigate why someone changed these user permissions.
From what I know, it was working before.


When I ran the "ps -ef | grep mysql", I got the below outputs.
Checked on couple of my machines (VM), all looked the same: Also, based on the cmdsubsys.log, it completed fine with a status of "0": Can you please post the email with the error? ... please post the latest one.


Regards,
Vinh

Hi,

I did not make any changes to mysql config files.

The 3 users that I granted "process" permissions to were ndoutils, nagiosql, and nagiosxi.

This is the text in the failure email. I did try setting a backup_timeout variable as it mentions, but it did not make any difference.

The scheduled local backup at 2021-04-06 10:41:22 failed.

Last output from the system:


View your full scheduled backup logs at /usr/local/nagiosxi/var/components/scheduledbackups.log to find out more details about the error. Your backup may have timed out, you can set a backup_timeout variable in config.inc.php to increase it.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------

Hi,
Could you please upload these logs (below): Let see if I understand the issue here (please correct me, if I'm wrong):
- You are able to run the "backup_xi.sh" script manually.
- Your scheduled backup failed.

Also, could you please upload the "/etc/sudoer" file.

I'm guessing that these user (apache or nagios) night not have correct sudo privileges to run the backup script ... just a guess ...


Regards,
Vinh

I tested out restoring from one of the "failed" backups and it restored fine, so I went ahead and tried updating Nagios XI to the latest release to see if that would fix whatever the problem is, but no luck. I have decided to open a formal support ticket to try to get this issue resolved, thanks for all the help!

Update: support ticket opened
https://support.nagios.com/tickets/scp/ ... p?id=13829

locking thread!!