Unable to upgrade Log Server 2.1.10 "Illegal instruction"

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Post Reply
JesseLambBDK
Posts: 7
Joined: Tue Nov 15, 2022 9:42 am

Unable to upgrade Log Server 2.1.10 "Illegal instruction"

Post by JesseLambBDK »

When attempting to upgrade Log Server 2.1.10 to 2.1.13 I get the following error:

-2022-11-14 16:02:43-- https://assets.nagios.com/downloads/nag ... upgrade.sh
Resolving assets.nagios.com (assets.nagios.com)... 45.79.49.120, 2600:3c00::f03c:92ff:fef7:45ce
Connecting to assets.nagios.com (assets.nagios.com)|45.79.49.120|:443... connected.
Illegal instruction (core dumped)

This occurs when trying to do the automatic upgrade and when trying to upgrade from the tmp folder.

[root@logserver2 tmp]# wget https://assets.nagios.com/downloads/nag ... est.tar.gz
--2022-11-15 09:37:13-- https://assets.nagios.com/downloads/nag ... est.tar.gz
Resolving assets.nagios.com (assets.nagios.com)... 45.79.49.120, 2600:3c00::f03c:92ff:fef7:45ce
Connecting to assets.nagios.com (assets.nagios.com)|45.79.49.120|:443... connected.
Illegal instruction (core dumped)
[root@logserver2 tmp]#

The log server is running on CentOS Stream 8.

Any and all thoughts would be appreciated.
JesseLambBDK
Posts: 7
Joined: Tue Nov 15, 2022 9:42 am

Re: Unable to upgrade Log Server 2.1.10 "Illegal instruction

Post by JesseLambBDK »

Still looking for some thoughts on why it's not working. I tried today to download the 2.1.13 update directly and got the same error.

wget https://assets.nagios.com/downloads/nag ... .13.tar.gz
--2022-11-16 11:09:32-- https://assets.nagios.com/downloads/nag ... .13.tar.gz
Resolving assets.nagios.com (assets.nagios.com)... 45.79.49.120, 2600:3c00::f03c:92ff:fef7:45ce
Connecting to assets.nagios.com (assets.nagios.com)|45.79.49.120|:443... connected.
Illegal instruction (core dumped)

I also checked wget is installed and up to date.

It also works just fine on our other log server

wget https://assets.nagios.com/downloads/nag ... est.tar.gz
--2022-11-16 11:14:40-- https://assets.nagios.com/downloads/nag ... est.tar.gz
Resolving assets.nagios.com (assets.nagios.com)... 45.79.49.120, 2600:3c00::f03c:92ff:fef7:45ce
Connecting to assets.nagios.com (assets.nagios.com)|45.79.49.120|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 227541085 (217M) [application/x-gzip]
Saving to: ‘nagioslogserver-latest.tar.gz’

nagioslogserver-latest.tar.gz 100%[===============================================================>] 217.00M 37.1MB/s in 5.9s

2022-11-16 11:14:46 (36.9 MB/s) - ‘nagioslogserver-latest.tar.gz’ saved [227541085/227541085]
gormank
Posts: 1114
Joined: Tue Dec 02, 2014 12:00 pm

Re: Unable to upgrade Log Server 2.1.10 "Illegal instruction

Post by gormank »

Why not scp the .gz file from the working NLS to the other and run the upgrade script?
JesseLambBDK
Posts: 7
Joined: Tue Nov 15, 2022 9:42 am

Re: Unable to upgrade Log Server 2.1.10 "Illegal instruction

Post by JesseLambBDK »

gormank wrote:Why not scp the .gz file from the working NLS to the other and run the upgrade script?
Mostly because I would like to know why it's not working as it should, but it looks like that might be the best option.
JesseLambBDK
Posts: 7
Joined: Tue Nov 15, 2022 9:42 am

Re: Unable to upgrade Log Server 2.1.10 "Illegal instruction

Post by JesseLambBDK »

gormank wrote:Why not scp the .gz file from the working NLS to the other and run the upgrade script?

Well that's a no go too. This is what happens.

[root@logserver2 nagioslogserver]# ./upgrade
Your system $PATH does not include /sbin and /usr/sbin. This is usually the result of installing GNOME rather than creating a clean system.
Adding /sbin and /usr/sbin to $PATH.
Nagios Log Server Upgrade Log
==========================
DATE: Thu Nov 17 10:58:47 EST 2022

DISTRO INFO:
CentOS
8
x86_64

Old Version: 2110
New Version: 2113

Last metadata expiration check: 0:41:32 ago on Thu 17 Nov 2022 10:17:16 AM EST.
Package php-devel-7.2.24-1.module_el8.2.0+313+b04d0a66.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
Archive: sourceguardian/ixed4.lin.x86-64.zip
inflating: /usr/lib64/php/modules/ixed.7.2.lin
Sourceguardian extension found for PHP version 7.2
Sourceguardian extension already in php.ini
Upgrading Elasticsearch...
Stopping elasticsearch (via systemctl): [ OK ]
Starting elasticsearch (via systemctl): [ OK ]
Elasticsearch upgraded OK
Upgrading Logstash...
Applying Nagios patches to Logstash...
Updating Logstash plugins...
Updating Logstash plugins...
Validating logstash-input-tcp-4.2.4-java.gem
Installing logstash-input-tcp
Installation successful
Validating logstash-filter-geoip-4.0.4-java.gem
Installing logstash-filter-geoip
Installation successful
Logstash upgraded OK
INSTALL: NCPA is being upgraded...
--2022-11-17 10:59:36-- https://repo.nagios.com/nagios/8/ncpa-2 ... x86_64.rpm
Resolving repo.nagios.com (repo.nagios.com)... 45.79.49.120, 2600:3c00::f03c:92ff:fef7:45ce
Connecting to repo.nagios.com (repo.nagios.com)|45.79.49.120|:443... connected.
./upgrade: line 39: 613624 Illegal instruction (core dumped) wget "$assets$pkgname"
gormank
Posts: 1114
Joined: Tue Dec 02, 2014 12:00 pm

Re: Unable to upgrade Log Server 2.1.10 "Illegal instruction

Post by gormank »

./upgrade: line 39: 613624 Illegal instruction (core dumped) wget "$assets$pkgname"

Maybe edit the script and add
echo "\n\n$assets$pkgname\n\n"
On line 38
JesseLambBDK
Posts: 7
Joined: Tue Nov 15, 2022 9:42 am

Re: Unable to upgrade Log Server 2.1.10 "Illegal instruction

Post by JesseLambBDK »

gormank wrote:./upgrade: line 39: 613624 Illegal instruction (core dumped) wget "$assets$pkgname"

Maybe edit the script and add
echo "\n\n$assets$pkgname\n\n"
On line 38
Here is the upgrade script. Line 39 where it says the failure is the old and new version which did display during the upgrade attempt. So I am not sure where the correct spot to put the echo command would be. I will also include the upgrade log in another message. It ends after NCPA is being updated.
-------------------------------
#!/bin/bash -e

cd $(dirname $(readlink -e "$0"))
. ./libinstall.sh

# Explicitly set umask
umask 0022

# Restart the upgrade
if [ -t 1 ]; then
$0 | tee -a "upgrade.log"
exit 0
fi

path='nagioslogserver'
proddir="/var/www/html/nagioslogserver"
backenddir="/usr/local/nagioslogserver"

do_upgrade_check

#source the old version number
. "$proddir/lsversion"
oldversion=$VERSION

#source the new number
. "$path/basedir/html/lsversion"
newversion=$VERSION

# Initialize upgrade.log
echo "Nagios Log Server Upgrade Log
==========================
DATE: $(date)

DISTRO INFO:
$distro
$version
$architecture

Old Version: $oldversion
New Version: $newversion
"

ELASTICSEARCH_UPDATED=0
LOGSTASH_UPDATED=0
INTERACTIVE="True"

# Make sure php-dev/php-devel is added
if [ "$distro" == "Ubuntu" ] || [ "$distro" == "Debian" ]; then
if [ "$dist" == "ubuntu16" ] || [ "$dist" == "ubuntu18" ] || [ "$dist" == "ubuntu20" ] || [ "$dist" == "debian9" ] || [ "$dist" == "debian10" ]; then
apt-get -y install php-dev
else
apt-get -y install php5-dev
fi
else
yum install -y php-devel
fi

# Install sourceguardian and resetart Apache
install_sourceguardian
if [ `command -v systemctl` ]; then
systemctl restart $httpd
else
service $httpd restart
fi

# Verify server has needed prereqs
if [ "$oldversion" -ge 200 ]; then

/usr/bin/php $proddir/www/index.php install/verify_prereqs

fi


#### VERSION SPECIFIC UPDATES ##########


if [ "$oldversion" -le 1 ]; then

# Update elasticsearch to latest
if [ $ELASTICSEARCH_UPDATED -eq 0 ]; then
(
cd ./subcomponents/elasticsearch
./upgrade
)
ELASTICSEARCH_UPDATED=1
fi

fi


if [ "$oldversion" -le 2 ]; then

pip install setuptools --upgrade

fi


if [ "$oldversion" -le 100 ]; then

cp -r nagioslogserver/basedir/html/application/config/config.local.php "$proddir/application/config/config.local.php"

fi


if [ "$oldversion" -le 103 ]; then

sed -i 's/cluser_id/cluster_id/g' "$proddir/application/config/config.local.php"

fi


if [ "$oldversion" -le 104 ]; then
# Update MAX_LOCKED_MEMORY to unlimited so we can use the mlockall feature in ES
sed -i 's/^#MAX_LOCKED_MEMORY=$/MAX_LOCKED_MEMORY=unlimited/g' /etc/sysconfig/elasticsearch
# Force a 1g heap if not already changed
mem='$(expr $(free -m|awk '\''/^Mem:/{print $2}'\'') / 2 )m'
sed -i 's~^#ES_HEAP_SIZE=2g$~# Nagios Log Server Default to 0.5 physical Memory\nES_HEAP_SIZE='"$mem"'~g' /etc/sysconfig/elasticsearch

fi


if [ "$oldversion" -le 105 ]; then
yum install php-ldap -y

# Update Elasticsearch
if [ $ELASTICSEARCH_UPDATED -eq 0 ]; then
(
cd subcomponents/elasticsearch
./upgrade
)
ELASTICSEARCH_UPDATED=1
fi

# Update Logstash
if [ $LOGSTASH_UPDATED -eq 0 ]; then
(
cd subcomponents/logstash
./upgrade
)
LOGSTASH_UPDATED=1
fi

fi


# Version 1.4.0
if [ "$oldversion" -le 140 ]; then

yum install openssl pyOpenSSL -y

# Update elasticsearch curator to latest
pipextra=""
if [ "$dist" == "el6" ]; then
pipextra="urllib3==1.22"
fi
pip install --upgrade elasticsearch-curator==3.4.0 $pipextra

# Re-install apache config - first check if the readwrite strings have been previously added
apacheregex=$(sed -n '/RewriteEngine/{:start /\[L,QSA\]/!{N;b start};/.*/p}' "$httpdconfdir/nagioslogserver.conf")

if [ -z "$apacheregex" ]; then
echo >> "$httpdconfdir/nagioslogserver.conf" # newline
grep "^Rewrite" nagioslogserver/httpd.nagioslogserver.conf >> "$httpdconfdir/nagioslogserver.conf" # append readwrite conditions
fi
fi


# Version 1.4.2
# - Minor patches for older versions
if [ "$oldversion" -lt 142 ]; then

# Generate a random key value for encryption_key
key=$(openssl rand -hex 32)
sed -i "/\$config\['encryption_key'\]/c\$config['encryption_key'] = '$key';" $proddir/application/config/config.local.php

fi


# Version 2.0.0
# - Updates to LS and ES versions
if [ "$oldversion" -lt 200 ]; then

if [ "$INTERACTIVE" = "True" ]; then
# CentOS, RedHat, Ubuntu, Debian, openSUSE, or SUSE Enterprise
fmt -s -w $(tput cols) <<-EOF

===========================
Nagios Log Server 2 Upgrade
===========================

Upgrading to version 2 requires active maintenance on your license. Please double check before upgrading.

Once upgraded, the Log Server interface will not be usable until you have activated. You can activate do this directly in the interface after upgrade.

You will need the following information to activate:

- Log Server license key
- Account ID or Token

EOF
read -p "Do you want to continue? [Y/n] " res

case "$res" in
Y | y | "")
echo "Proceeding with upgrade..."
;;
*)
echo "Upgrade cancelled"
exit 0
esac
fi

# Update Elasticsearch -> 1.7.6
if [ $ELASTICSEARCH_UPDATED -eq 0 ]; then
(
cd subcomponents/elasticsearch
./upgrade "$distro" "$dist"
)
ELASTICSEARCH_UPDATED=1
fi

# Update Logstash -> 2.4.1
if [ $LOGSTASH_UPDATED -eq 0 ]; then
(
cd subcomponents/logstash
./upgrade "$distro" "$dist"
)
LOGSTASH_UPDATED=1
fi

fi


# Version 2.0.3
# - Update LS tcp input to not have tcp connection leak and install Java 8
if [ "$oldversion" -lt 2003 ]; then

# Install Java 8
yum install -y java-1.8.0-openjdk

# Update tcp input plugin
(
cd subcomponents/logstash/mods/plugins
$backenddir/logstash/bin/logstash-plugin install --version 4.2.4 logstash-input-tcp.4.2.4-java.gem || true
)

# Restart logstash
if [ `command -v systemctl` ]; then
systemctl restart logstash
else
service logstash restart
fi

fi


# Version 2.0.8
# - Update the LS filter geoip to use the latest geoip2 dbs
if [ "$oldversion" -lt 2008 ]; then

# Set the version and update the filter for geoip2
(
cd subcomponents/logstash/mods/plugins
$backenddir/logstash/bin/logstash-plugin install --version 4.0.4 logstash-filter-geoip.4.0.4-java.gem || true
)

# Restart logstash
if [ `command -v systemctl` ]; then
systemctl restart logstash
else
service logstash restart
fi

# re-copy init script for Ubuntu and Debian required to allow more than 1GB memory
if [ "$distro" == "Debian" ] || [ "$distro" == "Ubuntu" ]; then
mv /etc/init.d/elasticsearch /etc/init.d/elasticsearch.old
cp ./subcomponents/elasticsearch/mods/etc/init.d/elasticsearch.deb.init /etc/init.d/elasticsearch
chmod +x /etc/init.d/elasticsearch
if [ `command -v systemctl` ]; then
systemctl daemon-reload
fi
fi
fi


# Version 2.1.0
# Need to install some options for AD/LDAP certificate management
if [ "$oldversion" -lt 2100 ]; then

# Make sure wkhtmltox dependencies are installed
if [ "$distro" == "Ubuntu" ] || [ "$distro" == "Debian" ]; then
apt-get -y install xfonts-base xfonts-75dpi

# Set up languages for Debian and Ubuntu
(
cd nagioslogserver/basedir/html/application/language
for lang in *; do
if [ -d "$lang" ]; then
if [ "$dist" == "debian8" ] || [ "$dist" == "debian9" ]; then
sed -i "/#[[:blank:]]*$lang/s/#//" /etc/locale.gen
else
locale-gen "$lang" &> /dev/null
locale-gen "$lang.UTF-8" &> /dev/null
fi
fi
done
echo "Configuring locale... this might take a minute..."
if [ `command -v locale-update` ]; then
locale-update &> /dev/null
else
locale-gen &> /dev/null
fi
)

fi

# This is the same as configure_ldap() in the fullinstall script
# php-ldap already installed in prereqs
if [ "$distro" == "Ubuntu" ] || [ "$distro" == "Debian" ]; then
ldap_config="/etc/ldap/ldap.conf"
ldap_dir="/etc/ldap"
cacerts_dir="/etc/ldap/cacerts"
else
ldap_config="/etc/openldap/ldap.conf"
ldap_dir="/etc/openldap"
cacerts_dir="/etc/openldap/cacerts"
fi

# Set the permissions of the openldap configuration files/folders
mkdir -p $cacerts_dir
mkdir -p $ldap_dir/certs
chown $apacheuser.$nagiosgroup $ldap_dir $ldap_dir/certs $cacerts_dir $ldap_config
chmod 664 $ldap_config
chmod 775 $ldap_dir $ldap_dir/certs $cacerts_dir

echo "\$config['ldap_dir'] = '$ldap_dir';" >> $proddir/application/config/config.local.php
echo "\$config['ldap_cacerts_dir'] = '$cacerts_dir';" >> $proddir/application/config/config.local.php

# Edit line in ldap config
sed -i 's/TLS_CACERTDIR/#TLS_CACERTDIR/g' $ldap_config
echo "TLS_CACERTDIR $cacerts_dir" >> $ldap_config

touch $backenddir/var/auditlog.log

# Install NCPA
(
cd subcomponents/ncpa
./install "$distro" "$dist" "$ver"
)

# Install wkhtmltox (html to pdf/image exporter)
(
cd subcomponents/wkhtmltox
./install "$distro" "$dist"
)

# Update permissions on logstash
(
cd subcomponents/logstash
./upgrade "$distro" "$dist"
)
fi


# Version 2.1.2
# Need to install JP fonts for wkhtmltopdf
if [ "$oldversion" -lt 2102 ]; then

if [ "$distro" == "Ubuntu" ] || [ "$distro" == "Debian" ]; then
apt-get install -y fonts-ipafont-mincho fonts-ipafont-gothic
else
yum install -y ipa-gothic-fonts ipa-mincho-fonts ipa-pgothic-fonts ipa-pmincho-fonts
fi

fi

# Version 2.1.11
# Reinstall elasticsearch and logstash to remove log4j dependencies
if [ "$oldversion" -lt 2111 ]; then

# Recompile core for added XSS patches
if [ $ELASTICSEARCH_UPDATED -eq 0 ]; then
(
cd subcomponents/elasticsearch
./upgrade "$distro" "$dist"
)
ELASTICSEARCH_UPDATED=1
fi

if [ $LOGSTASH_UPDATED -eq 0 ]; then
(
cd subcomponents/logstash
./upgrade "$distro" "$dist"
)
LOGSTASH_UPDATED=1
fi
fi

# Version 2.1.12
# Reinstall logstash again due to an incorrect path in 2.1.11
if [ "$oldversion" -lt 2112 ]; then

if [ $LOGSTASH_UPDATED -eq 0 ]; then
(
cd subcomponents/logstash
./upgrade "$distro" "$dist"
)
LOGSTASH_UPDATED=1
fi

(
cd subcomponents/ncpa
./upgrade "$distro" "$dist" "$ver"
)
fi

# Version 2.1.13
# Reinstall elasticsearch and logstash to add reload4j in places where log4j was.
if [ "$oldversion" -lt 2113 ]; then

if [ $ELASTICSEARCH_UPDATED -eq 0 ]; then
(
cd subcomponents/elasticsearch
./upgrade "$distro" "$dist"
)
ELASTICSEARCH_UPDATED=1
fi

if [ $LOGSTASH_UPDATED -eq 0 ]; then
(
cd subcomponents/logstash
./upgrade "$distro" "$dist"
)
LOGSTASH_UPDATED=1
fi
fi

##### ALL VERSION UPDATES ##############


# ---------------------------------
# Update sudoers if it needs it
# ---------------------------------
sudoers

# update cron
sed -i "s/@APACHE@/$apacheuser/g" nagioslogserver/nagioslogserver.cron.d
install -m 644 nagioslogserver/nagioslogserver.cron.d /etc/cron.d/nagioslogserver

# Copy source directory to the system
/bin/cp -f "$proddir/application/config/config.local.php" /tmp
/bin/rm -rf "$proddir/system"
/bin/rm -rf "$proddir/application/controllers"
/bin/rm -rf "$proddir/application/models"
/bin/rm -rf "$proddir/application/views"
/bin/rm -rf "$proddir/application/libraries"
/bin/cp -rf nagioslogserver/basedir/html/* "$proddir"

/bin/mv -f /tmp/config.local.php "$proddir/application/config/"

# Copy new data files to log server
/bin/cp -rf nagioslogserver/basedir/etc/* "$backenddir/etc"

# Copy backend directory scripts and replace permissions
/bin/cp -rf nagioslogserver/basedir/scripts/* "$backenddir/scripts"
chown -R nagios.nagios "$backenddir"
chmod -R 0775 "$backenddir"
chmod 0554 $backenddir/scripts/*

# Update log rotation file
cp -r ./nagioslogserver/logrotate.nagioslogserver /etc/logrotate.d/nagioslogserver

# Update Kibana
(
cd subcomponents/kibana
./upgrade
)

# Set up permissions
for dir in application/cache application/config application/logs application/dashboards www/media; do
mkdir -p "$proddir/$dir"
chgrp -R $apachegroup "$proddir/$dir"
chmod -R g+rwx "$proddir/$dir"
done

# Run the upgrade script to import new system dashboards
/usr/bin/php $proddir/www/index.php install/upgrade/$oldversion

# Special Logstash changes
if [ $LOGSTASH_UPDATED -eq 1 ]; then

echo "Re-applying logstash configuration..."
/usr/bin/php $proddir/www/index.php configure/write_configs_for_node

# Restart logstash
if [ `command -v systemctl` ]; then
systemctl restart logstash
else
service logstash restart
fi

fi

# Version 2.1.0
# This goes after source files are updated but before elasticsearch is restarted.
if [ "$oldversion" -lt 2100 ]; then

# Add Audit Log to global config
# Depends on new sources
auditlog_path=$(pwd)/nagioslogserver/basedir/etc/configs/xi_auditlog_config
auditlog_path=$(echo $auditlog_path | sed 's#/#%2F#g')
php "$proddir/www/index.php" cli CLI_AddToConfig add config_filters 'XI Audit Log' "$auditlog_path"

fi

# Version 2.1.3
# - Update existing users to have a username_lower property
if [ "$oldversion" -lt 2103 ]; then
/usr/bin/php $proddir/www/index.php install/update_usernames
fi

# Version 2.1.4
# - Real-time Alerts need to have their logstash configurations regenerated
if [ "$oldversion" -lt 2104 ]; then
/usr/bin/php $proddir/www/index.php install/update_realtimealert_configs
/usr/bin/php $proddir/www/index.php configure/apply_to_instances
fi

# Apply elasticsearch template
mkdir -p $backenddir/elasticsearch/config/templates
cp -rf subcomponents/elasticsearch/mods/elasticsearch/config/templates/* $backenddir/elasticsearch/config/templates
if [ `command -v systemctl` ]; then
systemctl restart elasticsearch
else
service elasticsearch restart
fi


#### After Upgrade Final Changes ####


# Version 2.0.0
# - Reset the backend commands
if [ "$oldversion" -lt 200 ]; then

/usr/bin/php $proddir/www/index.php install/reset_commands

fi

#### Complete Upgrade ####


echo
echo "Nagios Log Server Upgrade Complete!"
echo

if [ -n "$path" ]; then
get_ip

echo "You can access the Nagios Log Server web interface by visiting:"
echo " http://$ip/$path/"
echo
fi
JesseLambBDK
Posts: 7
Joined: Tue Nov 15, 2022 9:42 am

Re: Unable to upgrade Log Server 2.1.10 "Illegal instruction

Post by JesseLambBDK »

Here is the upgrade log:

Nagios Log Server Upgrade Log
==========================
DATE: Thu Nov 17 10:58:47 EST 2022

DISTRO INFO:
CentOS
8
x86_64

Old Version: 2110
New Version: 2113

Last metadata expiration check: 0:41:32 ago on Thu 17 Nov 2022 10:17:16 AM EST.
Package php-devel-7.2.24-1.module_el8.2.0+313+b04d0a66.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
Archive: sourceguardian/ixed4.lin.x86-64.zip
inflating: /usr/lib64/php/modules/ixed.7.2.lin
Sourceguardian extension found for PHP version 7.2
Sourceguardian extension already in php.ini
Upgrading Elasticsearch...
Stopping elasticsearch (via systemctl): [60G[[1;32m OK [0;39m]
Starting elasticsearch (via systemctl): [60G[[1;32m OK [0;39m]
Elasticsearch upgraded OK
Upgrading Logstash...
Applying Nagios patches to Logstash...
Updating Logstash plugins...
Updating Logstash plugins...
Validating logstash-input-tcp-4.2.4-java.gem
Installing logstash-input-tcp
Installation successful
Validating logstash-filter-geoip-4.0.4-java.gem
Installing logstash-filter-geoip
Installation successful
Logstash upgraded OK
INSTALL: NCPA is being upgraded...


This is where it failed and the log stopped.
Post Reply