NLS upgrade issue - elasticsearch dead but subsys locked

[esmie]

Hi Nagios Support,

I'm currently working on the upgrade of one of our Nagios Log Servers, and it seems that elasticsearch won't start after the upgrade.
I wasn't able to get the first log, but I noticed that the JAVA was upgraded.

OS: Red Hat Enterprise Linux Server release 6.9
Task: Upgrade from NLS 2.0.2 to 2.0.4
Status of Elasticsearch after the upgrade: elasticsearch dead but subsys locked

I have rebooted the server and try to re-run the upgrade process but seems to be the same:

Loaded plugins: product-id, search-disabled-repos, security, subscription-manager
Setting up Update Process
No Match for argument: nagios-repo
No package nagios-repo available.
No Packages marked for Update
Loaded plugins: product-id, search-disabled-repos, security, subscription-manager
Setting up Install Process
Package php-devel-5.3.3-49.el6.x86_64 already installed and latest version
Nothing to do
Archive: sourceguardian/ixed4.lin.x86-64.zip
inflating: /usr/lib64/php/modules/ixed.5.3.lin
Sourceguardian extension found for PHP version 5.3
Sourceguardian extension already in php.ini
Stopping httpd: [ OK ]
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using hostname for ServerName
[ OK ]
Upgrading Kibana...
Kibana upgraded OK
Stopping elasticsearch: [FAILED]
Starting elasticsearch: /usr/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
[ OK ]

Nagios Log Server Upgrade Complete!

You can access the Nagios Log Server web interface by visiting:
http://hostnamexxx/nagioslogserver/

[cdienger]

Try reinstalling it with:

yum remove java-1.8.0-openjdk-headless
yum install java-1.8.0-openjdk-headless

If it still doesn't work after this, please provide the output of:

/usr/bin/java -version
yum -y install mlocate
updatedb
yum list | grep java-1.8.0

[esmie]

I have recovered from backup the last working state of NLS (v2.0.2) and restored the snapshot yesterday. I will have to reschedule the upgrade again within this week.

I will try the option you mentioned, hopefully, we won't be encountering the same issue. But if it does, I'll try the steps you mentioned and provide you feedback.

Thank you.

[cdienger]

Thanks for the update.

[esmie]

Hi,

We had encountered the same issue .. it looks like that the shared library object "libjli.so" for java is missing.

What we did to resolve the issue:
Check java version:
java -version
Check shared object libraries
ldd /usr/bin/java
find files for libjli.so
find /usr/ -name libjli.so
Change directory to the lib directory shown on result of "ldd /usr/bin/java/"
cd /lib64
List the directory content and verify if libjli.so is not available
ll
ls libjli.so
Once verified that it's not existing, create a symlink on the correct java version of libjli.so (result on the find libjli.so) on /lib64 directory
ln /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-8.b10.el6_9.x86_64/jre/lib/amd64/jli/libjli.so libjli.so
Verify that shared object is now showing correctly
ldd /usr/bin/java
Restarted elasticsearch
service elasticsearch stop
service elasticsearch start
Verified that NLS is working as expected.

Thanks,

[cdienger]

Thank you for the update and steps you took to resolve it.