Migration bethween different versions (5.5.9 to 5.6.8)

[benjaminsmith]

Hello @junkertf,

There are two separate issues here, performance data and restoring to the new server. Initially, it looked like the restore was successful, but now some of the performance graphs were not updating. Is this correct?

1. According to the following message, the backup archive is broken. Is this backup from the production system or the new instance? If it's from the production server, did you upgrade to 5.6..8 before running the backup?

tar: usr/local/nagiosxi/var/xi-sys.cfg: Not found in archive
tar: Exiting with failure status due to previous errors
tar: usr/local/nagiosxi/var/xi-sys.cfg: Not found in archive

2. The database error message suggest that the DB is either corrupted or not running. Is the mariadb service running? Post the output of the following command.

Code: Select all

systemctl status mariadb

if it's not running try to restarting systemctl restart mariadb and logging in again.

3. Post the system messages log. Thanks.

Code: Select all

tail -n 50 /var/log/messages

[junkertf]

Hello,

Surely i had upgraded it, actually what you see as version in the profile?
I made the profile from that version. And also the backup crated at the last night as from the same version.

yes mariadb not running and could not start it in standalone way, it is running as cluster primitive (as in the drbd-nagios-cluster described)

i noticed today, that if i try to connect to the instance like
mysql -u root -p'pass' -h localhost
then the connection not working, but if i try to connect it
mysql -u root -p'pass' -h 127.0.0.1
then it works.

Also made a null-backup from our new prod server, with null config, but that backup also cannot be restored, because the script gives error message, that the password not match. I tried to change the password, so way that i tested it as above wroted, without success.

Note, that i dont licenced our new host, maybe is it also a problem?

Thank you, best regards,

Ferenc

[junkertf]

Hello,

PM sent for you benjaminsmith.

Thank you, best regards,

Ferenc

[benjaminsmith]

Hello Ferenc,

yes mariadb not running and could not start it in standalone way, it is running as cluster primitive (as in the drbd-nagios-cluster described)

1. Looking over the profile, mariadb is running. Looking over the documentation, Pacemaker is used to manage the cluster.
https://www.linbit.com/downloads/tech-g ... _RHEL7.pdf
Can you upload the /etc/my.cnf file to the thread?

3. Localhost not resolving.
Please post the /etc/hosts file to the thread.

4. Backup/Restore Failure
If you've changed the password, you'll need to edit the restore script. Directions can be found on page 11 of the backup/restore guide.
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

5. License
Your license is showing 2 activations, so this should not be a problem. Thanks.

[junkertf]

Hello,

Lets see first the output you're asked:

grep -vE "^$|^#" /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
innodb-buffer-pool-size=4G
innodb-buffer-pool-instances=4
bind-address = 127.0.0.1
innodb_file_per_table=1
query_cache_size=16M
query_cache_limit=4M
tmp_table_size=64M
max_heap_table_size=64M
key_buffer_size=32M
table_open_cache=32
max_connections=512
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
!includedir /etc/my.cnf.d

cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
...

Additinal information, that i had make some test today and mysql login works for all the 3 db instance:



I had tried to modify a bit the file
nagiosxi/html/includes/db.inc.php line line 321
throw new Exception(_("Connection failed: " . mysqli_connect_error()." A database co.......
and i become the following:

Connection failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (13) A database connection error has been detected....

So i tried it from cli:

mysql -undoutils -pdefault
Welcome to the MariaDB monitor. Commands end with ; or \g.
...
MariaDB [(none)]> use nagios
...
Database changed
MariaDB [nagios]> Ctrl-C -- exit!


mysql -unagiosql -pn@gweb
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 3
...
MariaDB [(none)]> use nagiosql
...
Database changed
MariaDB [nagiosql]> Bye


mysql -unagiosxi -pn@gweb
Welcome to the MariaDB monitor. Commands end with ; or \g.
...
MariaDB [(none)]> use nagiosxi
...
Database changed
MariaDB [nagiosxi]> exit

also lets see the xi-sys.cfg file:
grep __db /app/nagiosxi/var/xi-sys.cfg
cfg__dbtype=''
cfg__dbserver='localhost'
cfg__db_conn_persistent=1
cfg__db_info__nagiosxi__dbtype='mysql'
cfg__db_info__nagiosxi__dbserver=''
cfg__db_info__nagiosxi__user='nagiosxi'
cfg__db_info__nagiosxi__pwd='n@gweb'
cfg__db_info__nagiosxi__db='nagiosxi'
cfg__db_info__nagiosxi__charset='utf8'
cfg__db_info__nagiosxi__dbmaint__max_auditlog_age=180
cfg__db_info__nagiosxi__dbmaint__max_commands_age=480
cfg__db_info__nagiosxi__dbmaint__max_events_age=480
cfg__db_info__nagiosxi__dbmaint__optimize_interval=60
cfg__db_info__nagiosxi__dbmaint__repair_interval=0
cfg__db_info__ndoutils__dbtype='mysql'
cfg__db_info__ndoutils__dbserver='localhost'
cfg__db_info__ndoutils__user='ndoutils'
cfg__db_info__ndoutils__pwd='n@gweb'
cfg__db_info__ndoutils__db='nagios'
cfg__db_info__ndoutils__charset='latin1'
cfg__db_info__ndoutils__dbmaint__max_externalcommands_age=7
cfg__db_info__ndoutils__dbmaint__max_logentries_age=90
cfg__db_info__ndoutils__dbmaint__max_statehistory_age=730
cfg__db_info__ndoutils__dbmaint__max_notifications_age=90
cfg__db_info__ndoutils__dbmaint__max_timedevents_age=5
cfg__db_info__ndoutils__dbmaint__max_systemcommands_age=5
cfg__db_info__ndoutils__dbmaint__max_servicechecks_age=5
cfg__db_info__ndoutils__dbmaint__max_hostchecks_age=5
cfg__db_info__ndoutils__dbmaint__max_eventhandlers_age=5
cfg__db_info__ndoutils__dbmaint__optimize_interval=60
cfg__db_info__ndoutils__dbmaint__repair_interval=0
cfg__db_info__nagiosql__dbtype='mysql'
cfg__db_info__nagiosql__dbserver='localhost'
cfg__db_info__nagiosql__user='nagiosql'
cfg__db_info__nagiosql__pwd='n@gweb'
cfg__db_info__nagiosql__db='nagiosql'
cfg__db_info__nagiosql__charset='utf8'
cfg__db_info__nagiosql__dbmaint__max_logbook_age=480
cfg__db_info__nagiosql__dbmaint__optimize_interval=60
cfg__db_info__nagiosql__dbmaint__repair_interval=0
cfg__db_prefix__ndoutils='nagios_'
cfg__db_prefix__nagiosxi='xi_'
cfg__db_prefix__nagiosql='tbl_'
cfg__db_version=113

Best regards,
Ferenc

[junkertf]

Hello,
The Connection error message i mentioned aboce is coming for all three DB instance (nagiosxi,ndoutils,nagiosql)

best regards,

Ferenc

[junkertf]

Hello,

Also lets see the version check from our PROD node:

[root@myhuslhqbpinagp scripts]# grep -A 8 ^xiver /app/nagiosxi/var/xi-sys.cfg
xiver='5.6.8'
distro='RedHatEnterpriseServer'
version='7.7'
ver='7'
architecture='x86_64'
dist='el7'
arch='x86_64'
php_extension_dir='/usr/lib64/php/modules'
mysqld='mariadb'

and our new node:
[root@myhuslhqbpngxkp ~]# grep -A 8 ^xiver /app/nagiosxi/var/xi-sys.cfg
xiver='5.6.8'
distro='RedHatEnterpriseServer'
version='7.7'
ver='7'
architecture='x86_64'
dist='el7'
arch='x86_64'
php_extension_dir='/usr/lib64/php/modules'
mysqld='mariadb'


Best regards,

Ferenc

[benjaminsmith]

Hi Ference,

Looking at the output of the xi-sys.cfg file, this is not in the default path (/usr/local/nagiosxi/var/xi-sys.cfg ). This is probably why it's not in the backup archive.

/app/nagiosxi/var/xi-sys.cfg

What is the output of the following command?

Code: Select all

crm_mon

You mentioned when you first set up the clustered database it was working, are you able to revert to a working snapshot. Also, have you reached out to the Linbit support team regarding this issue?

[junkertf]

Hello,

i tried to roll back the cluster original config with xi_restore.sh script without success.

I could check it with the nagiosxi database xi_users table content (it contain more than 50 rows with our production database and only 1 row if i restore our orginal state)



crm resource stop g_nagios

Node myhuslhqbpngxkp.hu.money.ge.com (2): online
p-drbd-app_fs (ocf::linbit:drbd): Master
Node myhuslhqbpngxvp.hu.money.ge.com (1): online
p-drbd-app_fs (ocf::linbit:drbd): Slave

Inactive resources:

Resource Group: g_nagios
p_httpd (ocf:apache): Stopped (disabled)
p_fs_drbd (ocf:Filesystem): Stopped (disabled)
p_mysql (ocf:mysql): Stopped (disabled)
p_crond (systemd:crond): Stopped (disabled)
p_ndo2db (systemd:ndo2db): Stopped (disabled)
p_npcd (systemd:npcd): Stopped (disabled)
p_nagios (systemd:nagios): Stopped (disabled)
p_virtip (ocf:IPaddr2): Stopped (disabled)
p_virtsrcip (ocf:IPsrcaddr): Stopped (disabled)

crm resource start g_nagios

Last updated: Thu Jan 23 08:19:22 2020 Last change: Thu Jan 23 08:19:06 2020 by root via cibadmin on

2 nodes and 11 resources configured

Node xxx (2): online
p-drbd-app_fs (ocf::linbit:drbd): Master
p_crond (systemd:crond): Started
p_ndo2db (systemd:ndo2db): Started
p_npcd (systemd:npcd): Started
p_nagios (systemd:nagios): Started
p_virtsrcip (ocf:IPsrcaddr): Started
p_fs_drbd (ocf:Filesystem): Started
p_virtip (ocf:IPaddr2): Started
p_httpd (ocf:apache): Started
p_mysql (ocf:mysql): Started
Node xxy (1): online
p-drbd-app_fs (ocf::linbit:drbd): Slave

As i see after that crm restart, the mysql connection working well from cli:

[root@host ~]# mysql -unagiosql -pn@gweb
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 1
Server version: 5.5.64-MariaDB MariaDB Server
MariaDB [(none)]> use nagiosql
Database changed
MariaDB [nagiosql]> Bye

[root@host ~]# mysql -undoutils -pn@gweb
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.64-MariaDB MariaDB Server
MariaDB [(none)]> use nagios
Database changed
MariaDB [nagios]> Ctrl-C -- exit!
Aborted

[root@host ~]# mysql -unagiosxi -pn@gweb
Your MariaDB connection id is 3
Server version: 5.5.64-MariaDB MariaDB Server
MariaDB [(none)]> use nagiosxi
Database changed
MariaDB [nagiosxi]> Ctrl-C -- exit!

From the side of the browser i become the following error, whe i push the "Access NagiosXI" blue button ath the start:
Database Error
Connection failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (13)DB:localhost, DBUSER: nagiosxi, DBPASS: DBNAME:nagiosxi A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.
Run the following from the CLI as root to attempt to repair the DB....

Actually not contacted LinBit, because all the service running now... I think the question is more reliable the side of nagios rather than linbit, but because the question i open a ticket at the side of the LinBit also...

Best regards,

Ferenc

[junkertf]

Hello,

A second reply for today, i had found that /var/lib/mysql is a symlink so i had made some changes in the my.cnf (mysql side), without success:

ls -ld /var/lib/mysql
lrwxrwxrwx 1 root root 10 Dec 11 08:02 /var/lib/mysql -> /app/mysql
[root@myhuslhqbpngxkp ~]# cat /etc/my.cnf | grep -vE "^$|^#"
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=1
innodb-buffer-pool-size=4G
innodb-buffer-pool-instances=4
bind-address = 127.0.0.1
innodb_file_per_table=1
query_cache_size=16M
query_cache_limit=4M
tmp_table_size=64M
max_heap_table_size=64M
key_buffer_size=32M
table_open_cache=32
max_connections=512
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
!includedir /etc/my.cnf.d

thank you, best regards,

Ferenc