Migrated Nagios XI from Physical Server to VM

[vignesha]

Hello Team,

Recently we have migrated our Nagios XI servers from Physical server to VM's and we have successfully performed this activity without any issue but we have noticed an issue where while taking fresh backup of PROD server and once completed .tar file is having only read only access and then we manually giving permission to send the file to DR server for restoring PROD server backup. After restoring a backup successfully we have noticed an issue where Graph are not working so we troubleshoot and checked the issue where /etc/mrtg/conf.d/ file all having only read only access so we manually given permission post that i have run a command manually from /etc/cron.d/mrtg file and checked all working fine and Graph are started working.

We would require your help to resolve this issue permanently.

[benjaminsmith]

Hi @vignesha,

Appreciate your detailed description. Wondering about the cron daemon, is that running on the system? You can verify this with the following command:

Code: Select all

systemctl status crond.service

If that's running then, let's make sure the permissions are set correctly?

Code: Select all

chown apache:nagios /etc/mrtg -R
chmod 775 /etc/mrtg -R
chown apache:nagios /var/lib/mrtg -R
chmod 775 /var/lib/mrtg -R

Let me know if the issue is resolved after checking those.

Benjamin

Reference
Nagios XI - Debugging Bandwidth Performance Graphs

[vignesha]

Hello Benjamin,

I have followed the same process and resolved the issue temperately during that time but for our other Nagios XI server will take fresh backup of PRODUCTION and will restore without and issue on DR.

To explain in details, i have taken a fresh backup on same PROD server and restored on DR. Please find below attached log for reference.

Log:
PROD
[root@xxxxxxxxxxx ~]# /usr/local/nagiosxi/scripts/backup_xi.sh
Backing up Nagios Core...
tar: Removing leading `/' from member names
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
tar: /usr/local/nagios/var/ndo.sock: socket ignored
Backing up Nagios XI...
tar: Removing leading `/' from member names
tar: /usr/local/nagiosxi/var/sysstat.log: file changed as we read it
Backing up MRTG...
tar: Removing leading `/' from member names
Backing up the SNMP directories
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up NRDP...
tar: Removing leading `/' from member names
Backing up Nagvis...
tar: Removing leading `/' from member names
Backing up nagios user home dir...
tar: Removing leading `/' from member names
Backing up MySQL databases...
Backing up cronjobs for Apache...
Backing up logrotate config files...
Backing up Apache config files...
Compressing backup...

===============
BACKUP COMPLETE
===============
Backup stored in /store/backups/nagiosxi/1598425876.tar.gz
You have new mail in /var/spool/mail/root
[root@xxxxxxxxxxx ~]# ls -l /store/backups/nagiosxi/1598425876.tar.gz
-rw------- 1 nagios nagios 365654233 Aug 26 07:58 /store/backups/nagiosxi/1598425876.tar.gz

[root@xxxxxxxxxxx ~]# chmod 777 /store/backups/nagiosxi/1598425876.tar.gz
-rwxrwxr-x 1 nagios nagios 365792810 Aug 26 08:13 1598425876.tar.gz
------------------------------------------------------
DR:
[root@xxxxxxxxxxx ~]# /usr/local/nagiosxi/scripts/restore_xi.sh /tmp/1598425876.tar.gz
TS=1598426521
Extracting backup to /store/backups/nagiosxi/1598426521-restore...
In /store/backups/nagiosxi/1598426521-restore/1598425876...
Backup files look okay. Preparing to restore...
Shutting down services...
Restoring directories to /...
Restoring Nagios Core...
Restoring Nagios XI...
Restoring NRDP backups...
Restoring MRTG...
Restoring SNMP configuration files...
Restoring SNMP MIBs...
Restoring Nagvis backups...
Restoring nagios home dir...
Restoring MySQL databases...
Restoring Nagios XI MySQL database...
Restarting database servers...
Restoring Apache cronjobs...
Restoring logrotate config files...
Restoring Apache config files...

===============
RESTORE COMPLETE
===============
You have new mail in /var/spool/mail/root
[root@xxxxxxxxxxx ~]# ls -l /etc/mrtg/
total 12
drwxrwxr-x. 2 apache nagios 4096 Jul 28 15:27 archive
drwx------. 2 apache nagios 4096 Aug 26 08:13 conf.d
-rwx------. 1 apache nagios 789 Aug 26 08:13 mrtg.cfg
You have mail in /var/spool/mail/root

[root@xxxxxxxxxxx ~]# ls -l /var/lib/mrtg/
total 68224
-rwxrwxr-x 1 apache nagios 105312 Aug 26 08:11 xx.xxx.xx.xxx_11.rrd
-rwxrwxr-x 1 apache nagios 105312 Aug 26 08:11 xx.xxx.xx.xxx_12.rrd
-rwxrwxr-x 1 apache nagios 105312 Aug 26 08:11 xx.xxx.xx.xxx_13.rrd

[root@xxxxxxxxxxx ~]# chown apache:nagios /etc/mrtg -R
[root@xxxxxxxxxxx~]# chmod 775 /etc/mrtg -R
[root@xxxxxxxxxxx~]# chown apache:nagios /var/lib/mrtg -R
[root@xxxxxxxxxxx ~]# chmod 775 /var/lib/mrtg -R
[root@xxxxxxxxxxx ~]# ls -l /etc/mrtg
total 12
drwxrwxr-x. 2 apache nagios 4096 Jul 28 15:27 archive
drwxrwxr-x. 2 apache nagios 4096 Aug 26 08:13 conf.d
-rwxrwxr-x. 1 apache nagios 789 Aug 26 08:13 mrtg.cfg

[benjaminsmith]

Hi Vignesha,

Just want to isolate if this is a distribution or script issue. Which distributions are you using on the production and the DR systems?

I would also like to confirm if the permissions are getting carried over from the backup, can you check the permissions on those folders from the other server?

Thank you.

[vignesha]

Hello Benjamin,

We haven't noticed any permission issue in PROD server while taking backup. PFB output for reference

[root@xxxxxxxxxx ~]# ls -l /usr/local/
total 64
drwxr-xr-x. 2 root root 4096 Jul 27 21:12 bin
drwxr-xr-x. 2 root root 4096 Dec 14 2017 etc
drwxr-xr-x. 2 root root 4096 Dec 14 2017 games
drwxr-xr-x. 3 root root 4096 Jul 27 20:50 include
drwxr-xr-x. 2 root root 4096 Jul 27 20:50 lib
drwxr-xr-x. 3 root root 4096 Jul 27 20:47 lib64
drwxr-xr-x. 2 root root 4096 Dec 14 2017 libexec
drwxr-xr-x 9 root root 4096 Jan 9 2019 nagios
drwxr-xr-x. 6 root root 4096 Jul 27 21:06 nagiosmobile
drwxr-xr-x 10 root nagios 4096 Nov 2 2015 nagiosxi
drwxr-xr-x 5 apache nagios 4096 Jan 4 2019 nagvis
drwxr-xr-x 7 nagios nagios 4096 Jan 4 2019 nrdp
drwxr-xr-x. 4 root root 4096 Jul 27 21:12 nsis
drwxr-xr-x. 2 root root 4096 Dec 14 2017 sbin
drwxr-xr-x. 6 root root 4096 Jul 27 20:47 share
drwxr-xr-x. 2 root root 4096 Dec 14 2017 src
[root@xxxxxxxxxx ~]# ls -l /usr/local/nagiosxi/
total 112
drwxr-xr-x 2 root nagios 4096 Aug 20 16:56 cron
drwxr-xr-x 4 root nagios 4096 Jan 18 2020 etc
drwxr-xr-x 19 root nagios 4096 May 31 2018 html
drwxr-xr-x 3 root nagios 4096 Nov 2 2015 nom
drwxr-xr-x 4 root nagios 4096 Aug 17 18:00 scripts
drwsrwsr-x 2 root nagios 86016 Sep 1 17:23 tmp
drwxr-xr-x 2 root nagios 4096 Jan 18 2020 tools
drwxrwxr-x 7 nagios nagios 4096 Sep 1 17:47 var
[root@xxxxxxxxxx ~]# ls -l /usr/local/nagiosxi/scripts/
total 752
drwxr-xr-x 3 nagios nagios 4096 Jan 4 2019 automation
-r-xr-x--- 1 root nagios 7766 Jan 18 2020 backup_xi.sh
-r-xr-x--- 1 nagios nagios 8165 Jan 18 2020 ccm_delete_object.php
-r-xr-x--- 1 nagios nagios 1041 Jan 18 2020 ccm_export.php
-r-xr-x--- 1 nagios nagios 1630 Jan 18 2020 ccm_import.php
-r-xr-x--- 1 nagios nagios 3180 Jan 18 2020 ccm_snapshot.sh
-r-xr-x--- 1 root nagios 1800 Jan 18 2020 change_timezone.sh
drwxr-xr-x 2 root nagios 4096 Jan 18 2020 components
-rwxr-xr-x 1 nagios nagios 2839 Jan 18 2020 contact_notification_handler.php
-rwxr-xr-x 1 nagios nagios 660 May 31 2018 export_nagiosql.sh
-rwxr-xr-x 1 nagios nagios 266 Jan 18 2020 handle_nagioscore_event.php
-rwxr-xr-x 1 nagios nagios 3903 Jan 18 2020 handle_nagioscore.inc.php
-rwxr-xr-x 1 nagios nagios 267 Jan 18 2020 handle_nagioscore_notification.php
-rwxr-xr-x 1 nagios nagios 425 May 31 2018 import_nagiosql.sh
-r-xr-x--- 1 root nagios 1244 Jan 18 2020 import_xiconfig.php
-rwxr-xr-x 1 nagios nagios 705 Jan 18 2020 initialize_mibs.php
-rwxr-xr-x 1 nagios nagios 93 May 31 2018 kill_rrdtool.sh
-r-xr-x--- 1 root nagios 3615 Jan 18 2020 manage_services.sh
-r-xr-x--- 1 root nagios 3815 Jan 18 2020 manage_ssl_config.sh
-rwxr-xr-x 1 nagios nagios 161 Dec 7 2018 nagiosql.cookies
-rwxr-xr-x 1 nagios nagios 283685 Jan 18 2020 nagiosql_defaults.sql
-rwxr-xr-x 1 nagios nagios 48753 Nov 19 2018 nagiosql.delete.contact
-rwxr-xr-x 1 nagios nagios 895 May 31 2018 nagiosql_delete_contact.php
-rwxr-xr-x 1 nagios nagios 1937 May 31 2018 nagiosql_delete_host.php
-rwxr-xr-x 1 nagios nagios 497 May 31 2018 nagiosql_delete_object.sh
-rwxr-xr-x 1 nagios nagios 2766 May 31 2018 nagiosql_delete_service.php
-rwxr-xr-x 1 nagios nagios 50370 Nov 19 2018 nagiosql.delete.timeperiod
-rwxr-xr-x 1 nagios nagios 896 May 31 2018 nagiosql_delete_timeperiod.php
-rwxr-xr-x 1 nagios nagios 1272 May 31 2018 nagiosql_exportall.php
-rwxr-xr-x 1 nagios nagios 21480 Dec 7 2018 nagiosql.export.monitoring
-rwxr-xr-x 1 nagios nagios 1296 May 31 2018 nagiosql_importall.php
-rwxr-xr-x 1 nagios nagios 30779 Nov 9 2018 nagiosql.import.monitoring
-rwxr-xr-x 1 nagios nagios 39294 Dec 7 2018 nagiosql.login
-rwxr-xr-x 1 nagios nagios 0 Mar 29 2018 nagiosql.login--2018-03-29
-rwxr-xr-x 1 nagios nagios 1681 May 31 2018 nagiosql_login.php
-rwxr-xr-x 1 nagios nagios 2247 May 31 2018 nagiosql_snapshot.sh
-rwxr-xr-x 1 nagios nagios 200 May 31 2018 nagiosql_trim_backups.sh
-rwxr-xr-x 1 nagios nagios 366 May 31 2018 nagiosxi_dbtype.php
-rwxr-xr-x 1 nagios nagios 506 Jan 18 2020 nom_create_nagioscore_checkpoint_cond.sh
-rwxr-xr-x 1 nagios nagios 986 Jan 18 2020 nom_create_nagioscore_checkpoint.sh
-rwxr-xr-x 1 nagios nagios 785 Jan 18 2020 nom_create_nagioscore_errorpoint.sh
-rwxr-xr-x 1 nagios nagios 968 Jan 18 2020 nom_restore_nagioscore_checkpoint.sh
-rwxr-xr-x 1 nagios nagios 947 Jan 18 2020 nom_restore_nagioscore_checkpoint_specific.sh
-rwxr-xr-x 1 nagios nagios 4627 Jan 18 2020 nom_trim_checkpoints.sh
-rwxr-xr-x 1 nagios nagios 2893 May 31 2018 nom_trim_nagioscore_checkpoints.sh
-rwxr-xr-x 1 nagios nagios 7779 Jan 18 2020 nxti_import.php
-rwxr-xr-x 1 nagios nagios 1351 Jan 18 2020 nxti.php
-rwxr-xr-x 1 nagios nagios 12504 Jan 18 2020 parse_core_eventlog.php
-rwxr-xr-x 1 nagios nagios 6716 Jan 18 2020 patch_ndoutils.php
-rwxr-xr-x 1 nagios nagios 971 Jan 18 2020 reconfigure_nagios.sh
-rwxr-xr-x 1 nagios nagios 4224 Jan 18 2020 remove_historical_data.sh
-r-xr-x--- 1 root nagios 1705 Jan 18 2020 repair_databases.sh
-rwxr-xr-x 1 nagios nagios 3895 Jan 18 2020 repairmysql.sh
-r-xr-x--- 1 root nagios 4835 Jan 18 2020 reset_config_perms.sh
-rwxr-xr-x 1 nagios nagios 1859 Jan 18 2020 reset_nagiosadmin_password.php
-rwxr-xr-x 1 nagios nagios 1881 Jan 18 2020 restart_nagios_with_export.sh
-rwxr-xr-x 1 nagios nagios 4027 Jan 18 2020 restore_defaults.sh
-rwxr-xr-x 1 nagios nagios 13105 Jan 18 2020 restore_xi.sh
-rwxr-xr-x 1 nagios nagios 1908 Jan 18 2020 send_to_auditlog.php
-rwxr-xr-x 1 nagios nagios 2702 May 31 2018 uninstall_xi.sh
-rwxr-xr-x 1 nagios nagios 1345 Jan 18 2020 unlock_user_account.php
-rwxr-xr-x 1 nagios nagios 722 Jan 18 2020 update_check.php
-r-xr-x--- 1 root nagios 2914 Jan 18 2020 upgrade_to_latest.sh
[root@xxxxxxxxxx ~]#

[benjaminsmith]

Hi,

What about the MRTG folders on the production system, can you verify the permission settings on those.

Code: Select all

ls -l /etc/mrtg/
ls -l /var/lib/mrtg/

Thanks for the profile, I did notice a few things that need attention.

1. Can you check to see if you have SELinux enabled?

Code: Select all

getenforce

Sep 1 17:23:22 vwukm2sunix021 sudo: pam_unix(sudo:account): password for user nagios will expire in 9 days

2. It looks like the nagios user account is about to expire if so it can cause a number of issues, run the following command to verify this.

Code: Select all

chage -l nagios

Reset it with the following command.

Code: Select all

chage -I -1 -m 0 -M 99999 -E -1 nagios

[vignesha]

Hi Benjamin,

I have PM all the details, please check and let me know.

Thanks,
Vigneshwar

[benjaminsmith]

Hi @Vigneshwar

Thank you for your patience, I'm reviewing the logs right now with our team and I'll follow up with you shortly.

[vignesha]

Hi Benjamin,

Could you please provide update on this.

Thanks,
Vigneshwar

[benjaminsmith]

Hi Vigneshwar,

The file permission for /var/lib/mrtg/ on the production instance look ok. I set up a couple of 5.8 test servers and did not have an issue taking a backup and restoring to another server. I also changed the permissions of /var/lib/mrtg, took a backup, and then restored this on the test instance and the modified permissions were transferred.

I believe there are modified permissions on this production instance the differ from the other XI servers in your environment. After testing, this does not look to be a bug in the backup or restore script. Can you let me know the exact steps you are taking to correct the backup server after restoring so we can double-check those once more on the production server?

Benjamin