Page 1 of 2

Unable to upgrade Nagios Fuson

Posted: Tue Feb 27, 2018 12:13 pm
by dlukinski
Hello Nagios Support

We were unable to upgrade Nagios Fusion as follows:

Please help!

Code: Select all

[root@fikc-nagfsprod01 ~]# sh ./upgrade.sh
check if /tmp/nagiosfusion exists
Downloading latest Nagios Fusion release
--2018-02-27 16:59:59--  https://assets.nagios.com/downloads/nagiosfusion/fusion-latest.tar.gz
Resolving assets.nagios.com... 72.14.181.71, 2600:3c00::f03c:91ff:fedf:b821
Connecting to assets.nagios.com|72.14.181.71|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10872845 (10M) [application/x-gzip]
Saving to: “/tmp/fusion-latest.tar.gz”

100%[===========================================================================>] 10,872,845  4.75M/s   in 2.2s

2018-02-27 17:00:07 (4.75 MB/s) - “/tmp/fusion-latest.tar.gz” saved [10872845/10872845]

Running upgrade
/tmp/nagiosfusion/setup /tmp/nagiosfusion
/tmp/nagiosfusion
Stopping cron, killing existing cron jobs
/tmp/nagiosfusion/setup /tmp/nagiosfusion
Running './06-sourceguardian.sh'...
Archive:  sg/ixed4.lin.x86-64.zip
  inflating: /usr/lib64/php/modules/ixed.5.3.lin
Sourceguardian extension found for PHP version 5.3
Sourceguardian extension already in php.ini
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
SourceGuardian installed OK
RESULT=0
/tmp/nagiosfusion
/tmp/nagiosfusion/setup/upgrades /tmp/nagiosfusion
Stored root mysql password is good to go...
WARNING: Do NOT interupt the upgrade script during this process
============================================
Backing up databases
============================================
Skipping db: information_schema
Backing up fusion...
mysqldump: Got errno 28 on write
SOMETHING WENT WRONG BACKING UP fusion
Skipping db: mysql
Skipping db: test
An error occured while trying to backup your databases
while the upgrade script was attempting to recover your
ibdata1 file

The manual process for this includes:
 * Backing up each of your databases into a sql file
   - EXCEPT FOR 'information_schema'
   - EXCEPT FOR 'performance_schema'
   - EXCEPT FOR 'mysql'
 * Dropping the databases
   - EXCEPT FOR 'information_schema'
   - EXCEPT FOR 'performance_schema'
   - EXCEPT FOR 'mysql'
 * Stopping the mysql/mariadb service
 * Deleting the /var/lib/mysql/ibdata1 file
 * Starting the mysql/mariadb service
 * Creating the databases that were previously dropped
 * Restoring the data into the databases

This script attempted to automate those steps, and
and failed during the backup step - so it will not
continue
[root@fikc-nagfsprod01 ~]#

Re: Unable to upgrade Nagios Fuson

Posted: Tue Feb 27, 2018 3:19 pm
by tmcdonald
At first glance this appears to be insufficient disk space during the backup portion: https://stackoverflow.com/questions/376 ... 8-on-write

Can you check if this is the case? If so, we can either try to free up some disk space, or else comment out the section of the script that performs the backup (risky, but maybe necessary).

Re: Unable to upgrade Nagios Fuson

Posted: Tue Feb 27, 2018 3:53 pm
by dlukinski
tmcdonald wrote:At first glance this appears to be insufficient disk space during the backup portion: https://stackoverflow.com/questions/376 ... 8-on-write

Can you check if this is the case? If so, we can either try to free up some disk space, or else comment out the section of the script that performs the backup (risky, but maybe necessary).
Strangely it is using a lot of space, is there any way to cleanup?
- 2 x XI servers and 8 x CORE connected

Code: Select all

[root@fikc-nagfsprod01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       57G   31G   23G  58% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             477M  122M  330M  27% /boot
[root@fikc-nagfsprod01 ~]#

Re: Unable to upgrade Nagios Fuson

Posted: Wed Feb 28, 2018 12:59 pm
by cdienger
Is this an upgrade from 4.1 to the latest? If you're looking to keep the data then it'd probably be best to mount larger drive and write the backup there. The part of the code responsible fro the backup/error is found in 4100.sh:

Code: Select all

if ! mysqldump -u root -p"$mysqlpass" $db >databasebackups/$db; then
            echo "SOMETHING WENT WRONG BACKING UP $db"
            errors="True"
If possible I would recommend mounting a new drive and modifying the script to point at the new location:

Code: Select all

if ! mysqldump -u root -p"$mysqlpass" $db >/newlymounteddrivetosaveolddbto/$db; then
            echo "SOMETHING WENT WRONG BACKING UP $db"
            errors="True"

Re: Unable to upgrade Nagios Fuson

Posted: Wed Feb 28, 2018 1:59 pm
by dlukinski
cdienger wrote:Is this an upgrade from 4.1 to the latest? If you're looking to keep the data then it'd probably be best to mount larger drive and write the backup there. The part of the code responsible fro the backup/error is found in 4100.sh:

Code: Select all

if ! mysqldump -u root -p"$mysqlpass" $db >databasebackups/$db; then
            echo "SOMETHING WENT WRONG BACKING UP $db"
            errors="True"
If possible I would recommend mounting a new drive and modifying the script to point at the new location:

Code: Select all

if ! mysqldump -u root -p"$mysqlpass" $db >/newlymounteddrivetosaveolddbto/$db; then
            echo "SOMETHING WENT WRONG BACKING UP $db"
            errors="True"
If we do not need to keep the data, only keep connections?
How to alter the upgrade script?

Re: Unable to upgrade Nagios Fuson

Posted: Wed Feb 28, 2018 6:04 pm
by tgriep
You can remove the polling data from the server which should free up the space on the server.
Run this as root and see if the drive space is increased.

Code: Select all

cd /usr/local/nagiosfusion/scripts
./truncate_polled.php

Re: Unable to upgrade Nagios Fuson

Posted: Thu Mar 01, 2018 11:41 am
by dlukinski
tgriep wrote:You can remove the polling data from the server which should free up the space on the server.
Run this as root and see if the drive space is increased.

Code: Select all

cd /usr/local/nagiosfusion/scripts
./truncate_polled.php

Space actually decreased

[root@fikc-nagfsprod01 ~]# cd /usr/local/nagiosfusion/scripts
[root@fikc-nagfsprod01 scripts]# ./truncate_polled.php
polled tables truncated!
[root@fikc-nagfsprod01 scripts]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
57G 33G 22G 61% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 477M 122M 330M 27% /boot
[root@fikc-nagfsprod01 scripts]#


This is the 2nd time we run inot Fusion space problems: were no help from Support the first time and we had to re-spin FUSION from your template granting it 60 GB instead of 40 GB as before.

Maybe we could cleanup /usr/local/nagiosfusion/var/log? - It appears that none ofr them would be delete from the Fusion console..
- see those GZ files ever since VM got re-spinned..

We need help please!

- We do not understand why data takes so much space and believe product developers to be responsible !

Re: Unable to upgrade Nagios Fuson

Posted: Thu Mar 01, 2018 4:43 pm
by tgriep
What the Fusion server does it to download all of the status information from all of the fused servers and store it in the local MYSQL database so it can use it to display the data in the interface.

If you have a lot of large servers with a lot of checks, that is a lot of data to store.

The default setting is to store 5 days of data. This can be adjusted by logging in to the Fusion Interface and going to the Admin > System Settings menu. Click on the Data & Polling tab and decrease the Log Data Retention: to 1.

That will decrease the data stored in the database.

The old log files in the /usr/local/nagiosfusion/var/log filder can be deleted if needed but they should be automatically rotated everyday.

If you want the server to keep a smaller number of log files, you can edit this file

Code: Select all

/etc/logrotate.d/nagiosfusion
and decrease the rotate setting and that will save more space on the hard drive.

Re: Unable to upgrade Nagios Fuson

Posted: Thu Mar 01, 2018 8:19 pm
by dlukinski
tgriep wrote:What the Fusion server does it to download all of the status information from all of the fused servers and store it in the local MYSQL database so it can use it to display the data in the interface.

If you have a lot of large servers with a lot of checks, that is a lot of data to store.

The default setting is to store 5 days of data. This can be adjusted by logging in to the Fusion Interface and going to the Admin > System Settings menu. Click on the Data & Polling tab and decrease the Log Data Retention: to 1.

That will decrease the data stored in the database.

The old log files in the /usr/local/nagiosfusion/var/log filder can be deleted if needed but they should be automatically rotated everyday.

If you want the server to keep a smaller number of log files, you can edit this file

Code: Select all

/etc/logrotate.d/nagiosfusion
and decrease the rotate setting and that will save more space on the hard drive.
This still does not answer the question of logs: our config was set to 7-2-2 (rotate) >> changed 3-2-2
But AuthSub logs are out there since mid-January: None of them deleted automatically

When just started polling, 1-2 GB were used. Not 30+ GB
- no changes from the servers connected, so why does DB keeps growing?

Code: Select all

[root@fikc-nagfsprod01 lib]# cd mysql
[root@fikc-nagfsprod01 mysql]# du -s * | sort -nr | head -n10
31582240        ibdata1
5120    ib_logfile1
5120    ib_logfile0
988     mysql
212     fusion
4       test
0       mysql.sock
[root@fikc-nagfsprod01 mysql]#
Is this a PRODUCT PROBLEM?
VM got spinned from NAGIOS template... twice after it ran out os space the first time w/o any meaningful support offered!

Re: Unable to upgrade Nagios Fuson

Posted: Fri Mar 02, 2018 9:43 am
by tgriep
There was a bug fixed in version 4.1.0 of Fusion that fixed the issue of the MYSQL database that would grow to a very large size.
Here is the entry from the changelog.
Added fix for large mysql ibdata files -BH
So after you upgrade the server, that should be fixed.

The large log files, I would have to see the log rotate file from the server so can you post the following file.

Code: Select all

/etc/logrotate.d/nagiosfusion
And the output of this command run as root

Code: Select all

ls -l /usr/local/nagiosfusion/var/log/
What version of Fusion is the server currently running?