Migration Problems (32 to 64bit)

[dkuntz-jp]

Due to extreme slowness in our initial NagiosXI system (started with a 32bit only VMware version in 2010), even with extensive resources (8 CPU cores, 16gig RAM), we are attempting to move this to a 64bit install (RHEL 6.4 x86_64

We have ensured that the original system is running the latest release 2012r2.0, and ran the backup_xi script. We then installed NagiosXI using the source tarball on the new server (also 2012r2.0), and ran the restore_xi script.
Once that was done we followed the extra steps to ensure that the 64bit binaries were properly there (remove the installed.subcomponents file in /tmp/nagiosxi, run A-subcomponents, then build-perms-bin, as outlined in the backup/restore documentation).

We then ran the script provided in the FAQ to migrate the perf data from 32bit to 64bit using rrdtool dump and rrdtool restore commands (outlined here: http://support.nagios.com/wiki/index.ph ... Install.3F)

At this time, we can log into our new NagiosXI install, and as we use AD integration, our AD credentials work properly. Where we run into a problem is if we try to apply any changes.
Most of the time, it will return an error, and says to look at the snapshot file for errors, however, there is no recent snapshot file to review. What also happens is that it will just sit there trying to verify, without ending.

We also have tested the "Write Config Files". What happens with this is that we just get a blank frame. No completion notice, no failures, nothing. Running the Verify Config from there returns Duplicate Warnings (existing install does the same), but no errors.

I have run "reconfigure_nagios.sh &> reconfig.txt" and will include the output here.
I am also including the output from "tail -f /usr/local/nagiosxi/var/cmdsubsys.log &> cmd.txt", however, cmdsubsys.log (and thus cmd.txt) is not returning much of anything, as the last few runs, Apply Configuration is just sitting there.

We *HAD* a paid support/maintenance contract, however, our Operations dept. failed to renew. This is in process now, and we should be back on the paid support shortly.

[lmiltchev]

Run the following command and show the output:

Code: Select all

ll /usr/local/nagiosxi/scripts

[dkuntz-jp]

Code: Select all

[root@colnagiosxi-new perfdata]# ll /usr/local/nagiosxi/scripts
total 436
-rwxr-x---. 1 apache nagios   2899 May 23 08:41 backup_xi.sh
-rw-r--r--. 1 root   root     2620 May 23 09:51 cmd.txt
-rwxr-x---. 1 apache nagios    649 May 23 08:41 export_nagiosql.sh
-rwxr-x---. 1 apache nagios   1017 Nov  8  2011 fixperms.sh
-rwxr-x---. 1 apache nagios    821 May 23 08:41 handle_nagioscore_event.php
-rwxr-x---. 1 apache nagios    829 May 23 08:41 handle_nagioscore_notification.php
-rwxr-x---. 1 apache nagios    429 May 23 08:41 import_nagiosql.sh
-rwxr-x---. 1 apache nagios    149 May 23 08:41 kill_rrdtool.sh
-rwxr-x---. 1 apache nagios 272372 May 23 08:41 nagiosql_defaults.sql
-rwxr-x---. 1 apache nagios    970 May 23 08:41 nagiosql_delete_contact.php
-rwxr-x---. 1 apache nagios   2010 May 23 08:41 nagiosql_delete_host.php
-rwxr-x---. 1 apache nagios    505 May 23 08:41 nagiosql_delete_object.sh
-rwxr-x---. 1 apache nagios   3118 May 23 08:41 nagiosql_delete_service.php
-rwxr-x---. 1 apache nagios    993 May 23 08:41 nagiosql_delete_timeperiod.php
-rwxr-x---. 1 apache nagios   1637 May 23 08:41 nagiosql_exportall.php
-rwxr-x---. 1 apache nagios   1323 May 23 08:41 nagiosql_importall.php
-rwxr-x---. 1 apache nagios   1847 May 23 08:41 nagiosql_login.php
-rwxr-x---. 1 apache nagios   2246 May 23 08:41 nagiosql_snapshot.sh
-rwxr-x---. 1 apache nagios    258 May 23 08:41 nagiosql_trim_backups.sh
-rwxr-x---. 1 apache nagios    515 May 23 08:41 nom_create_nagioscore_checkpoint_cond.sh
-rwxr-x---. 1 apache nagios    826 May 23 08:41 nom_create_nagioscore_checkpoint.sh
-rwxr-x---. 1 apache nagios    632 May 23 08:41 nom_create_nagioscore_errorpoint.sh
-rwxr-x---. 1 apache nagios    784 May 23 08:41 nom_restore_nagioscore_checkpoint.sh
-rwxr-x---. 1 apache nagios    839 May 23 08:41 nom_restore_nagioscore_checkpoint_specific.sh
-rwxr-x---. 1 apache nagios   2972 May 23 08:41 nom_trim_nagioscore_checkpoints.sh
-rwxr-x---. 1 apache nagios   4592 May 23 08:41 parse_core_eventlog.php
-rwxr-x---. 1 apache nagios   6074 May 23 08:41 patch_ndoutils.php
-rw-r--r--. 1 apache nagios   2634 May 23 08:48 reconfig.txt
-rwxr-x---. 1 apache nagios    597 May 23 08:41 reconfigure_nagios.sh
-rwxr-x---. 1 apache nagios   1037 May 23 08:41 repairmysql.sh
-rwxr-xr-x. 1 apache nagios   6846 May 23 08:41 reset_config_perms
-rwxr-x---. 1 apache nagios    280 May 23 08:41 reset_config_perms.c
-rwxr-xr-x. 1 apache nagios   1187 May 23 08:41 reset_config_perms.sh
-rwxr-x---. 1 apache nagios   1155 May 23 08:41 reset_nagiosadmin_password.php
-rwxr-x---. 1 apache nagios    964 May 23 08:41 restart_nagios_with_export.sh
-rwxr-x---. 1 apache nagios    765 May 23 08:41 restore_defaults.sh
-rwxr-x---. 1 apache nagios   5342 May 23 08:41 restore_xi.sh
-rwxr-x---. 1 apache nagios   1847 May 23 08:41 send_to_auditlog.php
-rw-r-----. 1 apache nagios   2379 May 23 08:41 uninstall_xi.sh

[sreinhardt]

You are definitely going to want to run "chown -R nagios.nagios /usr/local/nagiosxi/scripts/"

While we're at it please send back the following:

Code: Select all

ll /usr/local/nagiosxi/var
ll /usr/local/nagios/var
ll /usr/local/nagios/var/rw

[dkuntz-jp]

You are definitely going to want to run "chown -R nagios.nagios /usr/local/nagiosxi/scripts/"

While we're at it please send back the following:

Code: Select all

ll /usr/local/nagiosxi/var
ll /usr/local/nagios/var
ll /usr/local/nagios/var/rw

Code: Select all

[root@colnagiosxi-new etc]# ll /usr/local/nagiosxi/var
total 624
-rw-r--r--. 1 apache nagios    303 May 23 08:45 cleaner.log
-rw-r--r--. 1 apache nagios    303 May 19 04:03 cleaner.log.1
-rw-r--r--. 1 apache nagios    303 May 12 04:02 cleaner.log.2
-rw-r--r--. 1 apache nagios    302 May  5 04:02 cleaner.log.3
-rw-r--r--. 1 apache nagios    302 Apr 28 04:02 cleaner.log.4
-rw-r--r--. 1 apache nagios   2863 May 23 08:46 cmdsubsys.log
-rw-r--r--. 1 apache nagios     22 May 19 04:04 cmdsubsys.log.1
-rw-r--r--. 1 apache nagios     22 May 12 04:03 cmdsubsys.log.2
-rw-r--r--. 1 apache nagios     22 May  5 04:03 cmdsubsys.log.3
-rw-r--r--. 1 apache nagios     22 Apr 28 04:03 cmdsubsys.log.4
drwsrwsr-x. 2 apache nagios   4096 Jan 30 13:05 components
-rw-r--r--. 1 apache nagios      8 May 22 19:56 corelog.data
-rw-r--r--. 1 apache nagios  19297 May 22 19:56 corelog.diff
-rwxrwxr-x. 1 nagios nagios   3014 May 17 09:48 corelog.newobjects
-rw-r--r--. 1 apache nagios   2677 May 23 08:45 dbmaint.log
-rw-r--r--. 1 apache nagios   2677 May 19 04:00 dbmaint.log.1
-rw-r--r--. 1 apache nagios   2677 May 12 04:00 dbmaint.log.2
-rw-r--r--. 1 apache nagios   2677 May  5 04:00 dbmaint.log.3
-rw-r--r--. 1 apache nagios   2677 Apr 28 04:00 dbmaint.log.4
-rw-r--r--. 1 apache nagios     21 May 23 08:45 deadpool.log
-rw-r--r--. 1 apache nagios     21 May 19 04:00 deadpool.log.1
-rw-r--r--. 1 apache nagios     21 May 12 04:00 deadpool.log.2
-rw-r--r--. 1 apache nagios     21 May  5 04:00 deadpool.log.3
-rw-r--r--. 1 apache nagios     21 Apr 28 04:00 deadpool.log.4
-rw-r--r--. 1 apache nagios  23831 May 23 08:46 eventman.log
-rw-r--r--. 1 apache nagios   4099 May 19 04:04 eventman.log.1
-rw-r--r--. 1 apache nagios     20 May 12 04:03 eventman.log.2
-rw-r--r--. 1 apache nagios     20 May  5 04:03 eventman.log.3
-rw-r--r--. 1 apache nagios   2154 Apr 28 04:03 eventman.log.4
-rw-r--r--. 1 apache nagios     18 May 23 08:45 feedproc.log
-rw-r--r--. 1 apache nagios     25 May 19 04:04 feedproc.log.1
-rw-r--r--. 1 apache nagios     25 May 12 04:03 feedproc.log.2
-rw-r--r--. 1 apache nagios     25 May  5 04:03 feedproc.log.3
-rw-r--r--. 1 apache nagios     25 Apr 28 04:03 feedproc.log.4
-rw-r--r--. 1 apache nagios      0 May 19 04:03 load_url.log
-rw-r--r--. 1 apache nagios    824 May 18 01:32 load_url.log.1
-rw-r--r--. 1 apache nagios    826 May 13 11:25 load_url.log.2
-rw-r--r--. 1 apache nagios    802 Mar 20 14:03 load_url.log.3
-rw-r--r--. 1 apache nagios    827 Mar 18 18:16 load_url.log.4
-rw-r--r--. 1 apache nagios      0 May 23 08:45 nom.log
-rw-r--r--. 1 apache nagios    567 May 23 08:46 perfdataproc.log
-rw-r--r--. 1 apache nagios    324 May 19 04:04 perfdataproc.log.1
-rw-r--r--. 1 apache nagios    243 May 12 04:03 perfdataproc.log.2
-rw-r--r--. 1 apache nagios    243 May  5 04:03 perfdataproc.log.3
-rw-r--r--. 1 apache nagios    243 Apr 28 04:03 perfdataproc.log.4
-rw-r--r--. 1 apache nagios  31616 May 22 17:01 recurringdowntime.log
-rw-r--r--. 1 apache nagios 242202 May 19 04:01 recurringdowntime.log.1
-rw-r--r--. 1 apache nagios  31155 May 12 04:01 recurringdowntime.log.2
-rw-r--r--. 1 apache nagios  31144 May  5 04:01 recurringdowntime.log.3
-rw-r--r--. 1 apache nagios  31145 Apr 28 04:01 recurringdowntime.log.4
-rw-r--r--. 1 apache nagios      0 May 23 08:45 reportengine.log
-rw-r--r--. 1 apache nagios     28 Apr 15  2012 reportengine.log.1
-rw-r--r--. 1 apache nagios     28 Apr  8  2012 reportengine.log.2
-rw-r--r--. 1 apache nagios     28 Apr  1  2012 reportengine.log.3
-rw-r--r--. 1 apache nagios     28 Mar 25  2012 reportengine.log.4
drwxr-xr-x. 2 apache nagios   4096 May 24 08:49 subsys
-rw-r--r--. 1 apache nagios   8153 May 23 08:46 sysstat.log
-rw-r--r--. 1 apache nagios    272 May 19 04:04 sysstat.log.1
-rw-r--r--. 1 apache nagios    272 May 12 04:03 sysstat.log.2
-rw-r--r--. 1 apache nagios    275 May  5 04:03 sysstat.log.3
-rw-r--r--. 1 apache nagios    275 Apr 28 04:03 sysstat.log.4
-rw-r--r--. 1 apache nagios    790 May 23 08:41 xi-sys.cfg
-rwxr-xr-x. 1 apache nagios    202 May 23 08:41 xiversion

Code: Select all

[root@colnagiosxi-new etc]# ll /usr/local/nagios/var
total 475768
drwxrwxr-x. 2 apache nagios     36864 May 24 00:00 archives
-rwxrwxr-x. 1 apache nagios  10291698 May 13 12:24 graphapi.log
-rw-rw-r--. 1 apache nagios  17753092 May 24 10:47 host-perfdata
-rw-r--r--. 1 nagios nagios         6 May 24 08:49 nagios.lock
-rw-rw-r--. 1 nagios nagios   1516672 May 24 10:47 nagios.log
-rwxrwxr-x. 1 apache nagios    698522 Oct 24  2011 nagios.log.old
-rw-r--r--. 1 apache nagios         5 May 22 19:48 ndo2db.lock
srwxr-xr-x. 1 apache nagios         0 May 22 19:48 ndo.sock
-rw-r--r--. 1 apache nagios   1788503 May 22 17:01 npcd.log
-rw-r--r--. 1 apache nagios  10485888 May 22 17:01 npcd.log.old
-rw-rw-r--. 1 apache nagios   4780183 May 24 08:49 objects.cache
-rw-rw-r--. 1 apache nagios   1222397 Oct 31  2011 objects.precache
-rw-rw-rw-. 1 apache nagios   1264434 May 22 16:17 perfdata.log
-rw-rw-rw-. 1 apache nagios      2743 Oct 12  2011 perfdata.log.old
-rw-------. 1 nagios nagios   7508093 May 24 09:49 retention.dat
drwxrwsr-x. 2 apache nagios      4096 May 24 08:49 rw
-rw-rw-r--. 1 apache nagios 422304510 May 24 10:47 service-perfdata
drwxrwxr-x. 5 apache nagios      4096 Jan 26  2011 spool
drwxrwxr-x. 2 apache nagios      4096 May 22 17:01 stats
-rw-rw-r--. 1 nagios nagios   7469606 May 24 10:47 status.dat

Code: Select all

[root@colnagiosxi-new etc]# ll /usr/local/nagios/var/rw
total 0
prw-rw----. 1 nagios nagios 0 May 24 08:49 nagios.cmd

[lmiltchev]

Let's try running the following commands first:

Code: Select all

chown -R nagios.nagios /usr/local/nagiosxi/scripts/
chown root.nagios /usr/local/nagiosxi/scripts/reset_config_perms
chown root.nagios /usr/local/nagiosxi/scripts/reset_config_perms.sh

Then run:

Code: Select all

cd /usr/local/nagiosxi/scripts/
./reset_config_perms.sh
./reconfigure_nagios.sh

Do "reconfigure" completes successfully now or fails?

[dkuntz-jp]

The reconfigure_nagios.sh seemed to complete, however, when clicking "Apply Configuration", it's still sitting and processing for an extended period of time (well over 5 minutes at this time)

Code: Select all

Command submitted for processing...
Waiting for configuration verification............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

[slansing]

Can you post the output of the following after you start the apply configuration:

Code: Select all

tail -20 tail /var/log/mysqld.log

What did you migrate over? You should have only needed to provision a new server/vm, then run through the import config documentation, and move additional data over "if the data is supported for migration." Nothing revolving around these directories or scripts permissions should have ever been changed during this process..

[dkuntz-jp]

Code: Select all

130523  9:09:16  InnoDB: Completed initialization of buffer pool
130523  9:09:16  InnoDB: Started; log sequence number 0 44233
130523  9:09:16 [Note] Event Scheduler: Loaded 0 events
130523  9:09:16 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.69'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
130523  9:56:13 [Note] /usr/libexec/mysqld: Normal shutdown

130523  9:56:13 [Note] Event Scheduler: Purging the queue. 0 events
130523  9:56:15  InnoDB: Starting shutdown...
130523  9:56:19  InnoDB: Shutdown completed; log sequence number 0 44233
130523  9:56:19 [Note] /usr/libexec/mysqld: Shutdown complete

130523 09:56:19 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130523 14:57:44 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130523 14:57:44  InnoDB: Initializing buffer pool, size = 8.0M
130523 14:57:44  InnoDB: Completed initialization of buffer pool
130523 14:57:44  InnoDB: Started; log sequence number 0 44233
130523 14:57:44 [Note] Event Scheduler: Loaded 0 events
130523 14:57:44 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.69'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

What I did was run the "/usr/local/nagiosxi/scripts/backup_xi.sh" on the original server. I then copied the /store/backups/* directory over to the new server, then ran "/usr/local/nagiosxi/scripts/restore_xi.sh /store/backups/nagiosxi/1369231512.tar.gz"

Once that was complete, I followed the directions on the Backup/Restore PDF, which said to remove /tmp/nagiosxi/installed.subcomponents, and then run "/tmp/nagiosxi/A-subcomponents" followed by "/tmp/nagiosxi/build-perms-bin".
After that, I ran the rrdtool script to export the 32bit RRDs to xml, moved them over to the new machine, and ran rrdtool to import the XMLs back in as rrd.

The only difference between the two systems is one CentOS 5.9 32bit, and the new one is RedHat Enterprise 6.4 x86_64.

[scottwilkerson]

Is crond running on the new system?

Can you post the output of the following

Code: Select all

ps -ef |grep cmdsubsys