Elasticsearch service failure on Nagios Logserver

[HIINNS]

I have been trying to stand up the latest version of Nagios Log server for several months. I had an open case on this in the past, but was told that because the server was using SELINUX that this was probably what was causing not getting the application working. Well, SELINUX is now disabled on this server ->

[myuid@myserver usr]$ getenforce
Disabled

I am still unable to get the app working. The problem seems to be that the elasticsearch service refuses to come up. See below.

[nyuid@myserver system]$ sudo systemctl status elasticsearch.service
Last login: Wed Nov 10 05:40:01 EST 2021 on cron
● elasticsearch.service - Elasticsearch
Loaded: loaded (/etc/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2021-11-10 05:24:39 EST; 19min ago
Docs: https://www.elastic.co
Process: 21544 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=233/RUNTIME_DIRECTORY)
Main PID: 21544 (code=exited, status=233/RUNTIME_DIRECTORY)

Nov 10 05:24:39 myserver systemd[1]: Ignoring invalid environment assignment 'if [ "$GET_ES_CONFIG_RETURN" !=0]; then': /etc/sysconfig/elasticsearch
Nov 10 05:24:39 myserver systemd[1]: Starting Elasticsearch...
Nov 10 05:24:39 myserver systemd[21544]: Failed at step RUNTIME_DIRECTORY spawning /usr/share/elasticsearch/bin/systemd-entrypoint: File exists
Nov 10 05:24:39 myserver systemd[1]: elasticsearch.service: main process exited, code=exited, status=233/RUNTIME_DIRECTORY
Nov 10 05:24:39 myserver systemd[1]: Failed to start Elasticsearch.
Nov 10 05:24:39 myserver systemd[1]: Unit elasticsearch.service entered failed state.
Nov 10 05:24:39 myserver systemd[1]: elasticsearch.service failed.

Any ideas??? Thank you in advance for your help.

[pbroste]

Hello @HIINNS

Thanks for reaching out so we can tackle this issue and get it resolved for you.

Want to see the /var/log/elasticsearch logs so we can see what is going on:

Code: Select all

tar -czvf /tmp/elasticsearchlogs.tar.gz /var/log/elasticsearch/ /var/log/logstash/

Please [PM] me the compressed logs found in /tmp/.

Thanks,
Perry

[HIINNS]

Requested file in PM.

[pbroste]

Hello @HIINNS

Could you please re-send the PM, it did not make the trip.

Thanks,
Perry

[HIINNS]

I have place the tar.gz file into PM several times. Is it still not making it. I can only surmise that my installation is preventing it from being sent this way.

[pbroste]

Hello @HIINNS

Thanks for following up and getting the logs over to me, after reviewing on the logs we see that there really isn't anything that jumps out but want to have you check and verify the following:

• Java:
[list]

• Code: Select all

  which java

• Code: Select all

  java -version

• Code: Select all

  ps -aux | grep -Ei 'java'

[*]firewall; allow ports 9200, and 9300[/*]

• Code: Select all

  iptables -L

• Also check with any firewall application that you may be running as well

[*]Make sure that System date/time/timezone are all synced up[/*]

• Code: Select all

  date
  ls -l /etc/localtime
  php -r 'echo date("D M j G:i:s T Y")."\n";'
  grep "date.timezone =" /etc/php.ini
  grep date.timezone /etc/php.ini
  php -r "echo date_default_timezone_get();" #timezone
  php -r " echo date('Y-m-d H:i:s');"        #date
  php -r " echo gmdate('Y-m-d H:i:s');"      #time

[*]Verify that there is no 'elasticsearch.conf' in here:[/*]

• Code: Select all

  less /usr/lib/tmpfiles.d/elasticsearch.conf

[*]System information:[/*]

• Code: Select all

  sudo cat /etc/os-release
   sudo uname -a
  sudo hostnamectl
  cat /etc/os-release
  uname -a
   hostnamectl
  php -v
  yum repolist

[/list]

Please follow up with the info,
Perry

[HIINNS]

which java
/usr/bin/java

java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

ps -aux | grep -Ei 'java'
nagios 16768 0.3 52.3 16131124 12843288 ? SLl Nov09 37:12 /bin/java -Xms11973m -Xmx11973m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Des.cluster.name=c0d6d20a-0a3c-4d14-9cda-1f2f4fcb6b55 -Des.node.name=bec4d3fc-0bab-49f5-88cf-fb1094c85cfd -Des.discovery.zen.ping.unicast.hosts=localhost -Des.path.repo=/ -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/local/nagioslogserver/elasticsearch -cp :/usr/local/nagioslogserver/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/local/nagioslogserver/elasticsearch/lib/*:/usr/local/nagioslogserver/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/local/nagioslogserver/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/usr/local/nagioslogserver/elasticsearch/data -Des.default.path.work=/usr/local/nagioslogserver/tmp/elasticsearch -Des.default.path.conf=/usr/local/nagioslogserver/elasticsearch/config org.elasticsearch.bootstrap.Elasticsearch

iptables -L
iptables v1.4.21: can't initialize iptables table `filter': Permission denied (you must be root)
sudo netstat -tulpn | grep 9200
tcp 0 0 127.0.0.1:9200 0.0.0.0:* LISTEN
sudo netstat -tulpn | grep 9300
tcp 0 0 0.0.0.0:9300 0.0.0.0:* LISTEN

date
Tue Nov 16 12:32:08 EST 2021

ls -l /etc/localtime
lrwxrwxrwx. 1 root root 36 Apr 7 2021 /etc/localtime -> /usr/share/zoneinfo/America/New_York

php -r 'echo date("D M j G:i:s T Y")."\n";'
Tue Nov 16 12:32:08 EST 2021
grep "date.timezone =" /etc/php.ini
date.timezone = America/New_York

grep date.timezone /etc/php.ini
; http://php.net/date.timezone
date.timezone = America/New_York

php -r "echo date_default_timezone_get();" #timezone
America/New_York[
$ php -r " echo date('Y-m-d H:i:s');" #date
2021-11-16 12:32:08
php -r " echo gmdate('Y-m-d H:i:s');" #time

less /usr/lib/tmpfiles.d/elasticsearch.conf
d /var/run/elasticsearch 0755 elasticsearch elasticsearch - -

sudo cat /etc/os-release
Last login: Tue Nov 16 12:52:35 EST 2021 on pts/1
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.9 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.9"

sudo uname -a
Last login: Tue Nov 16 12:56:28 EST 2021 on pts/1
Linux myservername 3.10.0-1160.31.1.el7.x86_64 #1 SMP Wed May 26 20:18:08 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

sudo hostnamectl
Last login: Tue Nov 16 12:58:16 EST 2021 on pts/1
Static hostname: myservername
Icon name: computer-vm
Chassis: vm
Machine ID: d8f91b7043bc44fea4e0a8a3a0c89653
Boot ID: 157c73c2617d4e39a8b6bb9e012e05c9
Virtualization: vmware
Operating System: Red Hat Enterprise Linux Server 7.9 (Maipo)
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.9:GA:server
Kernel: Linux 3.10.0-1160.31.1.el7.x86_64
Architecture: x86-64

cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.9 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.9"
uname -a
Linux myservername 3.10.0-1160.31.1.el7.x86_64 #1 SMP Wed May 26 20:18:08 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

hostnamectl
Static hostname: myservername
Icon name: computer-vm
Chassis: vm
Machine ID: d8f91b7043bc44fea4e0a8a3a0c89653
Boot ID: 157c73c2617d4e39a8b6bb9e012e05c9
Virtualization: vmware
Operating System: Red Hat Enterprise Linux Server 7.9 (Maipo)
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.9:GA:server
Kernel: Linux 3.10.0-1160.31.1.el7.x86_64
Architecture: x86-64

php -v
PHP 5.4.16 (cli) (built: Oct 29 2019 09:56:22)
Copyright (c) 1997-2013 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies

yum repolist
Plugin "product-id" can't be imported
Plugin "search-disabled-repos" can't be imported
Plugin "subscription-manager" can't be imported
Loaded plugins: langpacks
Skipping unreadable repository '/etc/yum.repos.d/redhat.repo'
repolist: 0

[HIINNS]

I have been doing some investigation as well. From executing the sudo systemctl status elasticsearch.status command I was able to determine that the /usr/share/elasticsearch directory was not created and the ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet command couldn't be run. Further looking into this revealed that the /usr/share/elasticsearch directory is set up using docker. I have been unable to find out where this directory is created in your code.

I realize I may be on the wrong track, but could this be what is causing my issue as I am unable to start the elasticsearch service.

[pbroste]

Hello @HIINNS

Thanks for following up, you stated that the "/usr/share/elasticsearch directory is set up using docker". Sounds like there was an existing elasticsearch Docker container installed prior to installing Nagios Log Server. The Nagios Log Server install is not officially supported in a Docker Container.

Please try to manually start the elasticsearch by: /.../..../elasticsearch start

Usage: /the/directory/where/Elasticsearch [ console | start | stop | restart | condrestart | status | install | remove | dump ]

Thanks,
Perry

[HIINNS]

sudo systemctl start elasticsearch.service
Last login: Thu Nov 18 05:15:01 EST 2021 on cron
Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

sudo systemctl -l status elasticsearch.service
Last login: Thu Nov 18 05:18:24 EST 2021 on pts/0
● elasticsearch.service - Elasticsearch
Loaded: loaded (/etc/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2021-11-18 05:15:13 EST; 4min 2s ago
Docs: https://www.elastic.co
Process: 1444 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=200/CHDIR)
Main PID: 1444 (code=exited, status=200/CHDIR)

Nov 18 05:15:13 myserver systemd[1]: Ignoring invalid environment assignment 'if [ "x$1"== "xstart" -o "x$1" == "xrestart" -o "x$1" == "xreload" -o "x$1" == "xforce-reload" ];then': /etc/sysconfig/elasticsearch
Nov 18 05:15:13 myserver systemd[1]: Ignoring invalid environment assignment 'if [ "$GET_ES_CONFIG_RETURN" !=0]; then': /etc/sysconfig/elasticsearch
Nov 18 05:15:13 myserver systemd[1]: Starting Elasticsearch...
Nov 18 05:15:13 myserver systemd[1444]: Failed at step CHDIR spawning /usr/share/elasticsearch/bin/systemd-entrypoint: No such file or directory
Nov 18 05:15:13 myserver systemd[1]: elasticsearch.service: main process exited, code=exited, status=200/CHDIR
Nov 18 05:15:13 myserver systemd[1]: Failed to start Elasticsearch.
Nov 18 05:15:13 myserver systemd[1]: Unit elasticsearch.service entered failed state.
Nov 18 05:15:13 myserver systemd[1]: elasticsearch.service failed

sudo journalctl -xe
Last login: Thu Nov 18 05:20:01 EST 2021 on cron
Nov 18 05:20:02 myserver audispd[1602]: node=myserver type=USER_ACCT msg=audit(
Nov 18 05:20:02 myserver audispd[1602]: node=myserver type=CRED_DISP msg=audit(
Nov 18 05:20:02 myserver audispd[1602]: node=myserver type=USER_END msg=audit(1
Nov 18 05:20:02 myserver audispd[1602]: node=myserver type=CRED_DISP msg=audit(
Nov 18 05:20:02 myserver audispd[1602]: node=myserver type=USER_END msg=audit(1
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=SYSCALL msg=audit(16
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=EXECVE msg=audit(163
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=CWD msg=audit(163723
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=PATH msg=audit(16372
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=PATH msg=audit(16372
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=PROCTITLE msg=audit(
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=USER_ACCT msg=audit(
Nov 18 05:20:48 myserver sudo[3149]: myuid : TTY=pts/0 ; PWD=/home/myuid
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=USER_CMD msg=audit(1
Nov 18 05:20:48 myserver adclient[2204]: INFO AUDIT_TRAIL|Centrify Suite|PAM|1
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=CRED_REFR msg=audit(
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=SYSCALL msg=audit(16
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=CWD msg=audit(163723
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=PATH msg=audit(16372
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=PROCTITLE msg=audit(
Nov 18 05:20:48 myserver sudo[3149]: pam_unix(sudo:session): session opened for
Nov 18 05:20:48 myserver audispd[1602]: node=myserver type=USER_START msg=audit