Plugins from /usr/local/nagios/libexec path not restored

[atc0005]

Hi,

I've recently transitioned into a role where I'm assisting with migrating a production Nagios XI instance on RHEL 6 (XI 5.6.14) to a new box on RHEL 8 (5.8.7).

As I understand it, we need to install the same version of Nagios XI on the destination system we're migrating to that we're currently using to run it on the source/original system. Unfortunately, as I understand it Nagios XI 5.6.14 won't work on RHEL 8 and a direct upgrade to the latest XI version on the RHEL 6 instance won't work either.

I have a RHEL 7 box at my disposal that I'm using as an intermediate box to restore a 5.6.14 backup to so that I can perform an in-place Nagios XI upgrade to 5.8.7 before I create another backup and then restore to the intended replacement production RHEL 8 system. It's a bit of a hop, but it seems like it will work.

I restored the backup (created on the RHEL 6 box) on the RHEL 7 box today, but initial testing revealed that not all of our plugins (located in the /usr/local/nagios/libexec path on the RHEL 6 box) were restored on the RHEL 7 system.

This text was shown during the restore attempt:

Code: Select all

WARNING: you are trying to restore a el6 backup on a el7 system
         Compiled plugins and other binaries as well as httpd configurations
         will NOT be restored.

         You will need to re-download the Nagios XI tarball, and re-install
         the subcomponents for this system. More info here:
         https://assets.nagios.com/downloads/nagiosxi/docs/Backing-Up-And-Restoring-Nagios-XI.pdf

Are you sure you want to continue? [y/N] Y
TS=1638378587
Extracting backup to /store/backups/nagiosxi/1638378587-restore...
In /store/backups/nagiosxi/1638378587-restore/1638309772...
Backup files look okay.  Preparing to restore...
Shutting down services...
Restoring directories to /...
Restoring Nagios Core...
Restoring Nagios XI...
Restoring NRDP backups...
Restoring MRTG...
Restoring SNMP configuration files...
Restoring SNMP MIBs...
Restoring Nagvis backups...
Restoring nagios home dir...
Restoring MySQL databases...
Restoring Nagios XI MySQL database...
Restarting database servers...
Restoring Apache cronjobs...
Restoring logrotate config files...
Skipping Apache config files restoration

===============
RESTORE COMPLETE
===============

This is the relevant/important text:

Compiled plugins and other binaries as well as httpd configurations
will NOT be restored.

Reading the Backing-Up-And-Restoring-Nagios-XI.pdf document, I found mention of the https://assets.nagios.com/downloads/nag ... _repair.sh script and I used that in order to restore missing plugins/compiled binaries.

It did not however restore our text/script-based (Perl, Bash, etc.) plugins which are not compiled.

I tracked this down to this section of the restore_xi.sh script:

Code: Select all

# Nagios Core
echo "Restoring Nagios Core..."
if [ "$arch" == "$backuparch" ] && [ $SPECIAL_BACKUP -eq 0 ]; then
    rm -rf /usr/local/nagios
    cd $rootdir && tar xzf $backupdir/nagios.tar.gz
else
    rm -rf /usr/local/nagios/etc /usr/local/nagios/share /usr/local/nagios/var
    cd $rootdir && tar --exclude="usr/local/nagios/bin" --exclude="usr/local/nagios/sbin" --exclude="usr/local/nagios/libexec" -xzf $backupdir/nagios.tar.gz
    cd $rootdir && tar --wildcards 'usr/local/nagios/libexec/*.*' -xzf $backupdir/nagios.tar.gz
fi

Specifically, this line:

Code: Select all

    cd $rootdir && tar --wildcards 'usr/local/nagios/libexec/*.*' -xzf $backupdir/nagios.tar.gz

If I modify the wildcard to just one asterisk then all plugins are restored.

I have to admit, I don't fully understand the intent behind the *.* wildcard, but I can reproduce the results with a ls call, so the wildcard pattern is having the same effect for ls as it is for the tar extraction command.

Regarding plugins that are not copied over, one example is a plugin named k_auPanopto (no file extension).

After reviewing the restore_xi.sh script I think that would be fine just manually restoring non-binary plugins from the /usr/local/nagios/libexec/ path on the source system to the intermediate (RHEL 7) and final (RHEL 8) boxes, but I wanted to reach out and make sure there wasn't other content (aside from Apache configs) that is not being restored by this workflow:

1) sudo /usr/local/nagiosxi/scripts/restore_xi.sh /store/backups/nagiosxi/1638309772.tar.gz
2) cd /tmp/
3) wget https://assets.nagios.com/downloads/nag ... _repair.sh
4) chmod +x restore_repair.sh
5) ./restore_repair.sh

I'm of the impression that either there is a bug in one of:

- the output from restore_xi.sh script (it implies that only compiled plugins are skipped)
- the restore_xi.sh script extraction syntax (the tar wildcard) which skips restoration of numerous non-binary plugins

Thanks in advance for your help!

[ssax]

That's expected. Your backup file is from EL6 and so the restore is basically telling you that they will need to be recompiled to fix that (aka, run the restore_repair script like you did to recompile the nagios binaries) because they won't work properly on EL7. But any 3rd party plugins not included with XI that you manually compiled/installed from the CLI would need to be recompiled on the new system as well.

The general process is this:

Is your EL6 system running 32-bit or 64-bit?

Code: Select all

uname -a

- Spin up new server running EL7/EL8 latest (EL7 in this case)

- Install the exact same version of XI that the old server has on the new server, no need to configure it, just install it and go through the initial setup (do not try to restore over different versions of XI you will cause issues)

https://assets.nagios.com/downloads/nag ... -Linux.pdf

- If you have a RAMDisk installed, are using RRDCacheD, or are running Mod_Gearman on the old system, set them up before restoring:

https://assets.nagios.com/downloads/nag ... giosXI.pdf
https://assets.nagios.com/downloads/nag ... ios-XI.pdf
https://assets.nagios.com/downloads/nag ... ios_XI.pdf

If you are monitoring vmware/oracle you'll need to reinstall the proper supporting packages on the new server again following these guides:

https://assets.nagios.com/downloads/nag ... ios-XI.pdf
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

- Backup the old/transfer to new/restore to the new

- Follow the After The Restore section from the backup/restore guide completely

- Since you're changing OS versions, run the restore_repair script from the backup/restore guide

https://assets.nagios.com/downloads/nag ... ios-XI.pdf


Optional:

- Shutdown (or re-IP) the old, set the old IP on the new system so you don't have to update agent configs/FW rules

https://support.nagios.com/kb/article/c ... s-549.html

Then for your specific case you would upgrade to the latest version of XI, then do the same migration process to the EL8 system when you're ready. Please note that there are 3rd party plugins that only support EL7 that would need to be updated by their respective authors to work on EL8, I would stay on EL7 for better compatibility but you can go to EL8 if you'd like.

That's pretty much it.

Please be aware of what a single XI license entitles you to:

https://support.nagios.com/kb/article.php?id=145

If your current system is 32-bit you need to follow this as well to migrate the performance data, if it's 64-bit already, no need to follow this:

https://support.nagios.com/kb/article.php?id=166

[atc0005]

Thanks for the follow-up.

That's expected. Your backup file is from EL6 and so the restore is basically telling you that they will need to be recompiled to fix that (aka, run the restore_repair script like you did to recompile the nagios binaries) because they won't work properly on EL7. But any 3rd party plugins not included with XI that you manually compiled/installed from the CLI would need to be recompiled on the new system as well.

I believe I understand the requirement regarding compiled plugins, but what about script-based (text file) plugins which do not require compilation for execution? Seeing some script-based plugins migrate to the new box but not other script-based (text file) plugins is confusing.

The general process is this:

Is your EL6 system running 32-bit or 64-bit?

Thankfully all three boxes are 64-bit.

- Install the exact same version of XI that the old server has on the new server, no need to configure it, just install it and go through the initial setup (do not try to restore over different versions of XI you will cause issues)

That is what I'm doing exactly. The EL7 box has a fresh installation of 5.6.14 (to match the version on the EL6 box) and the EL8 box has a fresh 5.8.7 installation.

- If you have a RAMDisk installed, are using RRDCacheD, or are running Mod_Gearman on the old system, set them up before restoring:

I don't believe we're using this. I checked recently and did not find it enabled.

If you are monitoring vmware/oracle you'll need to reinstall the proper supporting packages on the new server again following these guides:

https://assets.nagios.com/downloads/nag ... ios-XI.pdf
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

Thanks for the links. We're definitely monitoring VMware environments, not sure yet about Oracle-specific service checks (it's on the list to be confirm).

- Backup the old/transfer to new/restore to the new

- Follow the After The Restore section from the backup/restore guide completely

- Since you're changing OS versions, run the restore_repair script from the backup/restore guide

This is the workflow I'm following.

Optional:

- Shutdown (or re-IP) the old, set the old IP on the new system so you don't have to update agent configs/FW rules

This is our plan. We're also using this migration event to go ahead and have sysadmins add an additional IP Address to their firewalls to permit connections from a future dev XI instance while retaining the existing production IP Address that we're assign to the replacement box.

Then for your specific case you would upgrade to the latest version of XI, then do the same migration process to the EL8 system when you're ready. Please note that there are 3rd party plugins that only support EL7 that would need to be updated by their respective authors to work on EL8, I would stay on EL7 for better compatibility but you can go to EL8 if you'd like.

Thanks for the warning. Are either of the Oracle or VMware plugins not yet updated for EL8?

The plan is to go with a replacement EL8 host in order to have the longest OS patch/support window, but if there are significant issues please let us know and we'll pivot accordingly.

Please be aware of what a single XI license entitles you to:

https://support.nagios.com/kb/article.php?id=145

Please confirm, but I believe I understand our license to cover a prod instance, a backup instance (which we do not have immediate plans to setup) and a dev instance. Our plan is to decommission the current production instance once we finish migrating (plan is by the end of the year).

Once we finish the migration work we'll be left with a powered on production instance (EL8) and a test instance (also EL8, to be setup in the future).

Thanks for your help.

[benjaminsmith]

Hi,

Thanks for the warning. Are either of the Oracle or VMware plugins not yet updated for EL8?

You can use those plugins on EL8. Just be sure to follow the docs carefully as there are some differences between systems.

https://assets.nagios.com/downloads/nag ... ios-XI.pdf
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

Please confirm, but I believe I understand our license to cover a prod instance, a backup instance (which we do not have immediate plans to setup) and a dev instance. Our plan is to decommission the current production instance once we finish migrating (plan is by the end of the year).

That's correct, each license is allowed 3 activations, production, test and backup.

See: https://support.nagios.com/kb/article.php?id=145

Let us know if you have any other questions.

[atc0005]

Thanks for the warning. Are either of the Oracle or VMware plugins not yet updated for EL8?

You can use those plugins on EL8. Just be sure to follow the docs carefully as there are some differences between systems.

https://assets.nagios.com/downloads/nag ... ios-XI.pdf
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

Will do, thanks.

Please confirm, but I believe I understand our license to cover a prod instance, a backup instance (which we do not have immediate plans to setup) and a dev instance. Our plan is to decommission the current production instance once we finish migrating (plan is by the end of the year).

That's correct, each license is allowed 3 activations, production, test and backup.

See: https://support.nagios.com/kb/article.php?id=145

Let us know if you have any other questions.

Thanks for confirming.

One point I'm not clear on is 3rd-party text-based or script plugins on the original EL6 system. When I restored the backup on the intermediate EL7 system some of the 3rd-party text-based plugins (e.g., Perl, Bash, etc.) were present, but many were not.

I understand what was said about compiled plugins not carrying over to the new system (or needing to be recompiled via the restore_repair.sh script), but I'm not clear whether a partial copy of 3rd-party text-based plugins is intended, or whether they should all be left out of the backup or all included in the backup for restoration on the target system.

Thanks in advance for helping me understand the intent. As things stand, it seems like there is a bug in the logic used by the script (invalid wildcard pattern perhaps).

[benjaminsmith]

Hi,

One point I'm not clear on is 3rd-party text-based or script plugins on the original EL6 system. When I restored the backup on the intermediate EL7 system some of the 3rd-party text-based plugins (e.g., Perl, Bash, etc.) were present, but many were not.

The backup script is going to tar up the entire Nagios Core directory, so it if was in the default plugin directory, those files should have been transferred.

It's possible those scripts could have been in another location on the server and either the path was set in the command definition or a link to the file was created.

I'd have to look at the configurations in the system profile and possibly search the EL6 to determine the root cause. Is this system still available?

--Benjamin

[atc0005]

Hi,

One point I'm not clear on is 3rd-party text-based or script plugins on the original EL6 system. When I restored the backup on the intermediate EL7 system some of the 3rd-party text-based plugins (e.g., Perl, Bash, etc.) were present, but many were not.

The backup script is going to tar up the entire Nagios Core directory, so it if was in the default plugin directory, those files should have been transferred.

It's possible those scripts could have been in another location on the server and either the path was set in the command definition or a link to the file was created.

I'd have to look at the configurations in the system profile and possibly search the EL6 to determine the root cause. Is this system still available?

--Benjamin

The system is still available.

The tarballs created by the backup process contain the contents, but when files are extracted, not all files are restored.

For example, this command (using the same wildcard pattern as the restore_xi.sh script) gives back an incomplete list of the plugins that I expect to see.:

Code: Select all

tar --wildcards 'usr/local/nagios/libexec/*.*' --list -zf nagios.tar.gz

But if I modify the command like so:

Code: Select all

tar --wildcards 'usr/local/nagios/libexec/*' --list -zf nagios.tar.gz

All of the expected files are listed.

The same is true if I opt to actually extract the files (not shown) vs just listing them (as shown above).

It's possible those scripts could have been in another location on the server and either the path was set in the command definition or a link to the file was created.

I first noticed that one plugin wasn't transferred as I reviewed the failing service checks on the EL7 intermediate host. It was only when I looked closer that I noticed that many plugins (script/text-based) were not restored to the EL7 intermediate host.

Looking now, this is the command definition for the k_auPanopto command:

Code: Select all

Command Name: k_auPanopto
Command Line: $USER1$/k_auPanopto

When I look at the https://FQDN_HERE/nagiosxi/admin/monitoringplugins.php listing I see the plugin. On the EL6 (current active prod box) the plugin is functioning as intended.

[benjaminsmith]

Hi,

Okay, thanks for posting those commands, that helps clarify the issue.

So, it would not restore the compiled plugins as those will already be on the new system. And in this case, you would not want to transfer older plugins from the Nagios plugin project to the new server.

I can check back with development on this one but for the moment, I would check the statuses on the home page to find all the services with missing plugin assignments and transfer those over to the new system.

Ben

[atc0005]

Hi,

Okay, thanks for posting those commands, that helps clarify the issue.

So, it would not restore the compiled plugins as those will already be on the new system. And in this case, you would not want to transfer older plugins from the Nagios plugin project to the new server.

I can check back with development on this one but for the moment, I would check the statuses on the home page to find all the services with missing plugin assignments and transfer those over to the new system.

Ben

Hi Ben,

Apologies if my statements have been unclear, but what I've noticed not transferring is the text-based or script-based (e.g., Perl, Bash, Python, etc.) plugins. I expect that it likely has something to do with the plugin names (e.g., k_auPanopto is such a text-based or script-based / non-compiled plugin).

As suggested, I'll be restoring plugins reported as missing on the new in-progress production system.

My biggest concern was that the plugins failing to restore was a sign of a larger issue, but based on the responses I've received thus far that doesn't appear to be the case.

[benjaminsmith]

Hi,

The issue presented here with the plugin migrations was posted in our QA channel and bug report filed. The development team will take a closer look into the issue to determine if changes are required to the backup/restore scripts.

Let us know if you need further assistance with this issue.

--Benjamin