Leaving Centos and migrating Nagios to what?

[davide.bonicelli]

Hi, we've a very old (and big) nagiosxi (2014R2.6) on an old and unsupported Centos 6.
So now it's the time to migrate and upgrade both..due to Centos 8 notorius EOL what OS do you suggest for this operation?
Is Ubuntu ok or for now is better to switch to Centos 7?

[benjaminsmith]

Hi @davide.bonicelli,

You can go either way here, we fully support Ubuntu as well. However, Cent 7 will continue to have support until 20204 and is the most common OS for Nagios XI.

There are some additional steps when restoring to a different OS family. This is covered in the following guide.'

Backing Up And Restoring Your Nagios XI System

I would recommend leveraging the test server, your XI license allows for 3 activations, to help smooth the migration since you coming from such an old system. You need to run the restore from the same versions. The requires either upgrading the old system or installing the old version on the new server and then upgrading.

Upgrade Instructions
https://assets.nagios.com/downloads/nag ... ctions.pdf

Previous versions are available for download here.
https://assets.nagios.com/downloads/nag ... rsions.php

Let us know if you need assistance with anything.

--Benjamin

[davide.bonicelli]

Hi, so i restored the backup from the old nagios server with centos 6 x86 to the new with centos 7 x64, same version.
All green, except when i try the restore script as in the guide i've this error

Code: Select all

./restore_repair.sh
--2021-02-16 00:04:22--  https://assets.nagios.com/downloads/nagiosxi/5/xi-2014R2.6.tar.gz
Resolving assets.nagios.com (assets.nagios.com)... 72.14.181.71, 2600:3c00::f03c:91ff:fedf:b821
Connecting to assets.nagios.com (assets.nagios.com)|72.14.181.71|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://assets.nagios.com/redirects/redirect.php?url=https://assets.nagios.com/downloads/nagiosxi/5/xi-2014R2.6.tar.gz [following]
--2021-02-16 00:04:23--  https://assets.nagios.com/redirects/redirect.php?url=https://assets.nagios.com/downloads/nagiosxi/5/xi-2014R2.6.tar.gz
Reusing existing connection to assets.nagios.com:443.
HTTP request sent, awaiting response... 404 Not Found
2021-02-16 00:04:23 ERROR 404: Not Found.

tar: xi-2014R2.6.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
./restore_repair.sh: line 13: cd: nagiosxi: No such file or directory
./restore_repair.sh: line 14: ./init.sh: No such file or directory
./restore_repair.sh: line 19: cd: /tmp/nagiosxi/subcomponents/extraplugins: No such file or directory
./restore_repair.sh: line 23: ./upgrade: No such file or directory
./restore_repair.sh: line 19: cd: /tmp/nagiosxi/subcomponents/mrtg: No such file or directory
./restore_repair.sh: line 23: ./upgrade: No such file or directory
./restore_repair.sh: line 19: cd: /tmp/nagiosxi/subcomponents/nagioscore: No such file or directory
./restore_repair.sh: line 23: ./upgrade: No such file or directory
./restore_repair.sh: line 19: cd: /tmp/nagiosxi/subcomponents/nagiosplugins: No such file or directory
./restore_repair.sh: line 23: ./upgrade: No such file or directory
./restore_repair.sh: line 19: cd: /tmp/nagiosxi/subcomponents/ndoutils: No such file or directory
./restore_repair.sh: line 23: ./upgrade: No such file or directory
./restore_repair.sh: line 19: cd: /tmp/nagiosxi/subcomponents/nrpe: No such file or directory
./restore_repair.sh: line 23: ./upgrade: No such file or directory
./restore_repair.sh: line 19: cd: /tmp/nagiosxi/subcomponents/nsca: No such file or directory
./restore_repair.sh: line 23: ./upgrade: No such file or directory
./restore_repair.sh: line 19: cd: /tmp/nagiosxi/subcomponents/pnp: No such file or directory
./restore_repair.sh: line 23: ./upgrade: No such file or directory
./restore_repair.sh: line 19: cd: /tmp/nagiosxi/subcomponents/webinject: No such file or directory
./restore_repair.sh: line 21: ./install: No such file or directory

[benjaminsmith]

Hi davide.bonicelli,

Sot that is the restore_repair.sh script that is used for recompling packages when changing OS families. Were you able to successfully run the XI restore script?

Code: Select all

/usr/local/nagiosxi/scripts/restore_xi.sh </full/path/to/backupfile.tar.gz>

Since you are going from centos 6 x86 to centos 7 x64 it shouldn't be necessary to run that script (restore_repair.sh).

Regards,
Benjamin

[davide.bonicelli]

yep, my bad!
it works without restore_repair.sh
but i have some question: now i see that

Code: Select all

/var/log/mysqld.log

didn't exist anymore..is it normal?
because i've a script that check it for any sql errors..
and i see a lot (and i mean a lot!) of this error in /var/log/messages

Code: Select all

wmic: ldapdb_canonuser_plug_init() failed in sasl_canonuser_add_plugin(): invalid parameter supplied

do you know what it means?
and at last..the new server now runs with the old nagios with a different ip..to upgrade i've to activate the license first with the new ip, correct?
thanks!

[benjaminsmith]

Hi @davide.bonicelli,

Glad that worked out. For the database, the location differs in Debian/Ubunut systems. Login to terminal, you should find the log file in the following directory/path.

Code: Select all

/var/log/mysql
/var/log/mysql/error.log

For the second question, I haven't seen that before, do you have Active Directory setup with this Nagios XI instance?

To update the license, just go to Admin > System Config > License Information, enter the new license key, click on "Update License", then activate the new license. You can use the same IP address or a new one. Each Nagios XI license is approved for up to three installations: one primary monitoring/production, one backup/failover, and one test environment. However, only one system can be used for production monitoring.

References:
Activating Your License

[davide.bonicelli]

Finally updated nagios to latest version!
This is a great satisfaction!
But i have migrated to a Centos 7, so i don't have /var/log/mysql or /var/log/mysql/error.log
And unfurnately i still have a lot of

Code: Select all

wmic: ldapdb_canonuser_plug_init() failed in sasl_canonuser_add_plugin(): invalid parameter supplied

but i don't have any Active Directory integration configurated..

[davide.bonicelli]

another big problem..today i've a database error on the login page

Code: Select all

Database Error
A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB:

/usr/local/nagiosxi/scripts/repair_databases.sh

the system has high resource usage (and has 10 cpu, 8gb ram and 4Gb Swap!) and become unresponsive.
the repair database script end with errors like this

Code: Select all

- recovering (with sort) MyISAM-table 'tbl_variabledefinition'
Data records: 7350
- Fixing index 1
/usr/bin/pkttyagent: error while loading shared libraries: libbz2.so.1: failed to map segment from shared object: Cannot allocate memory
Failed to start mariadb.service: Connection timed out
See system logs and 'systemctl status mariadb.service' for details.
~

===============
REPAIR COMPLETE
===============
Failed to restart nagios.service: Connection timed out
See system logs and 'systemctl status nagios.service' for details.

=======================
nagios database repair FAILED, please check output above!
nagiosql database repair succeeded

if i stop before the nagios service the repair goes to the end but the problem back very soon
in the nagios.log i see errors like this

Code: Select all

[1613756114] wproc:   early_timeout=0; exited_ok=1; wait_status=256; error_code=0;
[1613756114] wproc:   stdout line 01: UNABLE TO CONNECT TO DB - EXITING!
[1613756114] wproc: GLOBAL SERVICE EVENTHANDLER job 1596 from worker Core Worker 1851 is a non-check helper but exited with return code 1
[1613756114] wproc:   early_timeout=0; exited_ok=1; wait_status=256; error_code=0;
[1613756114] wproc:   stdout line 01: UNABLE TO CONNECT TO DB - EXITING!
[1613756114] wproc: GLOBAL SERVICE EVENTHANDLER job 1591 from worker Core Worker 1851 is a non-check helper but exited with return code 1
[1613756114] wproc:   early_timeout=0; exited_ok=1; wait_status=256; error_code=0;
[1613756114] wproc:   stdout line 01: UNABLE TO CONNECT TO DB - EXITING!
[1613756114] wproc: NOTIFY job 1592 from worker Core Worker 1855 is a non-check helper but exited with return code 1
[1613756114] wproc:   host=ztodi.siris.it; service=Uptime; contact=Server_Pro
[1613756114] wproc:   early_timeout=0; exited_ok=1; wait_status=256; error_code=0;
[1613756114] wproc:   stdout line 01: UNABLE TO CONNECT TO DB - EXITING!
[1613756115] HOST ALERT: 10.136.161.0;DOWN;HARD;10;check_icmp: Failed to resolve 10136161254: Name or service not known
[1613756115] HOST ALERT: 10.136.146.0;DOWN;HARD;10;check_icmp: Failed to resolve 10136146254: Name or service not known
[1613756115] HOST ALERT: 88.40.167.160-191;DOWN;SOFT;7;CRITICAL - 88: rta nan, lost 100%
[1613756115] HOST ALERT: 10.136.126.0;DOWN;HARD;10;check_icmp: Failed to resolve 10136126254: Name or service not known
[1613756115] HOST ALERT: 10.136.153.0;DOWN;HARD;10;check_icmp: Failed to resolve 10136153254: Name or service not known
[1613756115] HOST ALERT: 172.23.100.176;DOWN;HARD;10;CRITICAL - 172.23.100.190: Host unreachable @ 195.120.150.130. rta nan, lost 100%
[1613756115] HOST ALERT: 10.136.145.0;DOWN;HARD;10;check_icmp: Failed to resolve 10136145254: Name or service not known
[1613756125] Warning: Service performance data file processing command '/bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/1613756115.perfdata.service' timed out after 5 seconds

the core interface it seems to work more or less..
what can i do next?

[benjaminsmith]

Hi @davide.bonicelli,

Please try running the following command as root from the CLI to manually repair the databases.

Code: Select all

mysqlcheck -f -r -u root -pnagiosxi --all-databases

Run a top command and post the output so we can see which processes are using up the memory.

I'd like to get a fresh system profile, and please check to make sure the database log is included (if not, can you send it separately).

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button

Please upload the database configuration file as well.

Code: Select all

/etc/my.cnf

Are you continuing to use the older system for production monitoring? If not, it would best to get a ticket opened right away for faster resolution (reference this thread).

https://support.nagios.com/tickets/

Benjamin

[ssax]

Locking thread, ticket received, we will continue support through the ticket.

Thank you