Off-load Database option at install?

[TBT]

UPDATE: As we wait for a reply to my previous post. On the current system we've been discussing, I modified the RAMdisk setup using the defaults provided in the documentation. I then ran the checks we did last time, it appears the RAMdisk error has gone away and the Host/Services are there, but Nagios isn't running. Restarting the service with "systemctl restart nagios" has no affect.

Code: Select all

~# systemctl status nagios
● nagios.service - Nagios Core 4.4.6
   Loaded: loaded (/lib/systemd/system/nagios.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2021-10-07 13:13:53 EDT; 4min 11s ago
     Docs: https://www.nagios.org/documentation
  Process: 4022 ExecStartPre=/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg (code=exited, status=0/SUCCESS)
  Process: 4023 ExecStart=/usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg (code=exited, status=0/SUCCESS)
  Process: 4043 ExecStopPost=/usr/bin/rm -f /usr/local/nagios/var/rw/nagios.cmd (code=exited, status=0/SUCCESS)
 Main PID: 4024 (code=exited, status=1/FAILURE)

Oct 07 13:13:51 nagios-dev nagios[4024]: wproc: Registry request: name=Core Worker 4041;pid=4041
Oct 07 13:13:51 nagios-dev nagios[4024]: wproc: Registry request: name=Core Worker 4040;pid=4040
Oct 07 13:13:51 nagios-dev nagios[4024]: wproc: Registry request: name=Core Worker 4042;pid=4042
Oct 07 13:13:51 nagios-dev nagios[4024]: Event broker module '/usr/local/nagios/bin/ndo.so' initialized successfully.
Oct 07 13:13:51 nagios-dev nagios[4024]: Error: Could not load module '/usr/local/lib/mk-livestatus/livestatus.o' -> file not found
Oct 07 13:13:51 nagios-dev nagios[4024]: Error: Failed to load module '/usr/local/lib/mk-livestatus/livestatus.o'.
Oct 07 13:13:51 nagios-dev nagios[4024]: Error: Module loading failed. Aborting.
Oct 07 13:13:51 nagios-dev nagios[4024]: Event broker module '/usr/local/nagios/bin/ndo.so' deinitialized successfully.
Oct 07 13:13:51 nagios-dev systemd[1]: nagios.service: Main process exited, code=exited, status=1/FAILURE
Oct 07 13:13:53 nagios-dev systemd[1]: nagios.service: Failed with result 'exit-code'.

Code: Select all

# /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.4.6
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2020-04-28
License: GPL

Website: https://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
        Checked 211 services.
        Checked 10 hosts.
        Checked 9 host groups.
        Checked 9 service groups.
        Checked 6 contacts.
        Checked 2 contact groups.
        Checked 150 commands.
        Checked 13 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 10 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 13 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check

[benjaminsmith]

Hi,

Okay, I went through the last post, and since you have 8 systems to migrate, either path is going to take a little work due to the error you're getting.

I would recommend to remove or modify the ramdisk settings on the new systems, so you do not have to modify your productions servers. There's a chance you get to take down or disrupt your monitoring while editing these files. Unfortunately, we do not have an uninstall ramdisk script. Once everything is migrated and working properly, you can switch over the systems.

Regarding the latest issue, that is related to the livestatus broker module not being available.

Oct 07 13:13:51 nagios-dev nagios[4024]: Error: Could not load module '/usr/local/lib/mk-livestatus/livestatus.o' -> file not found
Oct 07 13:13:51 nagios-dev nagios[4024]: Error: Failed to load module '/usr/local/lib/mk-livestatus/livestatus.o'.
Oct 07 13:13:51 nagios-dev nagios[4024]: Error: Module loading failed. Aborting.

Open up the nagios.cfg file and comment out the following line, and the restart Nagios Core

Code: Select all

# MKLivestatus
#broker_module=/usr/local/lib/mk-livestatus/livestatus.o /usr/local/nagios/var/rw/live

Reference
Remove Ramdisk

[TBT]

Perhaps I am misunderstanding your reply. According to the documentation, prerequisite of a restore/repair is that the new system must have RAMdisk installed if the old system has it in use. Did you mean remove or modify the ramdisk settings on the OLD systems?

I am leaning towards removing the RAMdisk from OLD servers (current production) so it's no longer is a prerequisite. Thus our migration steps will change slightly. Please confirm the logic below.

New system is Debian 10, old system is CentOS 6.

1. Old system: Remove RAMdisk as per steps outlined in linked reference.
2. New system: Install the same XI version (5.7.5) as the old system.
3. New system: Setup RRDcache (in use on the old system).
4. New system: Offload the Database.
5. New system: Backup offloaded DB modified files.
6. Old system: Created a backup using backup_xi.sh and copy over to new system.
7. New system: Run restore using restore_xi.sh
8. New system: Copy offloaded DB modified files back in place.
9. New system: Login to XI, ensure Program URL, External URL and License key are correct.
10. New system: Run repair using restore_repair.sh
11. New system: Modify /etc/postgresql/11/main/postgresql.conf as per your suggestion.
12. New system: Setup RAMDisk using Automatic Install script.

------------------------------------------

According to that RAMdisk removal post, the default directories for XI are:

/usr/local/nagios/var/spool/checkresults
/usr/local/nagios/var/spool/perfdata
/usr/local/nagios/var/spool/xidpe

I noticed files dating back to 2012 in those dirs (I assume when we implemented RAMdisk), most likely we should clean these out prior to RAMdisk removal?

------------------------------------------

Additionally, I've modified nagios.cfg should the event_broker_options line be commented out too?

Code: Select all

# MKLivestatus
# broker_module=/usr/local/lib/mk-livestatus/livestatus.o /usr/local/nagios/var/rw/live
event_broker_options=-1

Nagios is now running, I see the Hosts/Services have carried over. But all users are missing and performance grapher is still offline (red).

Please note that Monday is a holiday, I will reply Tuesday.

[benjaminsmith]

Hi,

Perhaps I am misunderstanding your reply. According to the documentation, prerequisite of a restore/repair is that the new system must have RAMdisk installed if the old system has it in use. Did you mean remove or modify the ramdisk settings on the OLD systems?

My recommendation would be to not modify the production (OLD) systems if those are currently being used for monitoring. The issue here is the new server does not have the mount point for the ramdisk, so the configurations are pointing to directories that do not exist on the new server.

Benjamin

[TBT]

Hi,

Perhaps I am misunderstanding your reply. According to the documentation, prerequisite of a restore/repair is that the new system must have RAMdisk installed if the old system has it in use. Did you mean remove or modify the ramdisk settings on the OLD systems?

My recommendation would be to not modify the production (OLD) systems if those are currently being used for monitoring. The issue here is the new server does not have the mount point for the ramdisk, so the configurations are pointing to directories that do not exist on the new server.

Benjamin

Alright, so we'll follow our initial process and modify the RAMdisk configuration on the new system once everything is in place.

How about my other question and outstanding issues...

1. Should we comment out event_broker_options=-1 too?

2. Nagios is now running, I see the Hosts/Services have carried over. But all users are missing and performance grapher is still offline (red).

[benjaminsmith]

Hi,

Should we comment out event_broker_options=-1 too?

No, the backend database loads as a broker module, but comment out or remove the livestatus broker modules as that was not loading.

Please try to res-start the performance grapher.

Code: Select all

systemctl restart npcd

Can you send us a fresh profile now that Core is running?

--Benajmin

Reference
Nagios XI - Performance Graph Problems

[TBT]

Hi,

Should we comment out event_broker_options=-1 too?

No, the backend database loads as a broker module, but comment out or remove the livestatus broker modules as that was not loading.

Please try to res-start the performance grapher.

Code: Select all

systemctl restart npcd

Can you send us a fresh profile now that Core is running?

--Benajmin

Reference
Nagios XI - Performance Graph Problems

Okay, I'll leave the event_broker_options=-1 line as is.

The performance grapher is now online with a restart, odd since we rebooted after all the previous changes.

Profile sent via PM.

UPDATE: An observation, the OLD server /usr/local/nagiosxi/html/config.inc.php the nagiosxi DB is set to "dbserver" => 'localhost', where as the on the NEW server it's pointing to the offloaded DB server address.

[benjaminsmith]

Hi,

Thanks for the profile, so the nagios process is now running. I still see errors with the ramdisk setup, for example, in the npcd (nagios performance data processor) log:

[10-08-2021 14:14:57] NPCD: Error while get file list from spooldir (/var/nagiosramdisk/spool/perfdata/) - No such file or directory

Can you check the permission on that directory?

Code: Select all

ls -ld /var/nagiosramdisk/spool/perfdata/

--Benjamin

[TBT]

Hi,

Thanks for the profile, so the nagios process is now running. I still see errors with the ramdisk setup, for example, in the npcd (nagios performance data processor) log:

[10-08-2021 14:14:57] NPCD: Error while get file list from spooldir (/var/nagiosramdisk/spool/perfdata/) - No such file or directory

Can you check the permission on that directory?

Code: Select all

ls -ld /var/nagiosramdisk/spool/perfdata/

--Benjamin

Code: Select all

drwxrwxr-x 2 nagios nagios 40 Oct 12 17:15 /var/nagiosramdisk/spool/perfdata/

How would RAMdisk affect XI users (only nagiosadmin exists) though?

I just logged in and noticed the performance grapher is offline again too.

[benjaminsmith]

Hi,

The ramdisk would not affect XI users. Something is still not set up right with the ramdisk, can you post the output to the following commands:

Code: Select all

df -h
ls -lR /var/nagiosramdisk/
cat /lib/systemd/system/ramdisk.service
systemctl status  ramdisk.service

Additionally, the manual setup instructions are on page 3 of the doc, I would go through each step again to make sure nothing was missed.

Nagios XI Utilizing a RAM Disk in Nagios XI

Benjamin