Elastic search will not start - Java may be the cause.
Elastic search will not start - Java may be the cause.
Hi Everyone,
I have been trying to fix some vulnerabilities found by our scanner on the nagios log server. It seems like I cannot start elasticsearch right now.
On the web page i'm getting the waiting for database startup message. When I look at the server itself i get this message from elastic search status. (see below).
Has anyone run into this issue? I believe it has to do with Java.
[root@PGE-NAGIOSLOG ~]# service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
Active: active (exited) since Mon 2022-02-07 01:43:00 EST; 9h ago
Docs: man:systemd-sysv-generator(8)
Process: 1010 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
Feb 07 01:42:56 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
Feb 07 01:43:00 PGE-NAGIOSLOG.powergrid.lcl runuser[1245]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
Feb 07 01:43:00 PGE-NAGIOSLOG.powergrid.lcl runuser[1245]: pam_unix(runuser:session): session closed for user nagios
Feb 07 01:43:00 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[1010]: Starting elasticsearch: [ OK ]
Feb 07 01:43:00 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Started LSB: This service manages the elasticsearch daemon.
Feb 07 01:42:04 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[1010]: Unrecognized VM option 'UseParNewGC'
Feb 07 01:42:04 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[1010]: Error: Could not create the Java Virtual Machine.
Feb 07 01:42:04 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[1010]: Error: A fatal exception has occurred. Program will exit.
[root@PGE-NAGIOSLOG ~]#
I have been trying to fix some vulnerabilities found by our scanner on the nagios log server. It seems like I cannot start elasticsearch right now.
On the web page i'm getting the waiting for database startup message. When I look at the server itself i get this message from elastic search status. (see below).
Has anyone run into this issue? I believe it has to do with Java.
[root@PGE-NAGIOSLOG ~]# service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
Active: active (exited) since Mon 2022-02-07 01:43:00 EST; 9h ago
Docs: man:systemd-sysv-generator(8)
Process: 1010 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
Feb 07 01:42:56 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
Feb 07 01:43:00 PGE-NAGIOSLOG.powergrid.lcl runuser[1245]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
Feb 07 01:43:00 PGE-NAGIOSLOG.powergrid.lcl runuser[1245]: pam_unix(runuser:session): session closed for user nagios
Feb 07 01:43:00 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[1010]: Starting elasticsearch: [ OK ]
Feb 07 01:43:00 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Started LSB: This service manages the elasticsearch daemon.
Feb 07 01:42:04 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[1010]: Unrecognized VM option 'UseParNewGC'
Feb 07 01:42:04 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[1010]: Error: Could not create the Java Virtual Machine.
Feb 07 01:42:04 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[1010]: Error: A fatal exception has occurred. Program will exit.
[root@PGE-NAGIOSLOG ~]#
Re: Elastic search will not start - Java may be the cause.
Hello @isaacl86
Thanks for reaching out, want to get the following:
Along with that let's also get the log server system profile:
The System Profile is found in '/tmp/system-profile.tar.gz'
May need to use the split command to size down the compressed Profile.
Please send each 'part[x]' in a separate PM.
Thanks,
Perry
Thanks for reaching out, want to get the following:
- Java: [list]
Code: Select all
which java
Code: Select all
java -version
Code: Select all
whereis java
Code: Select all
sestatus
Code: Select all
ps -aux | grep -Ei 'java'
Along with that let's also get the log server system profile:
Code: Select all
/usr/local/nagioslogserver/scripts/profile.sh
May need to use the split command to size down the compressed Profile.
Code: Select all
split -b 40M profile.tar.gz part
Thanks,
Perry
Re: Elastic search will not start - Java may be the cause.
Command outputs below
[root@PGE-NAGIOSLOG ~]# which java
/usr/bin/java
[root@PGE-NAGIOSLOG ~]# java -version
openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8-LTS, mixed mode, sharing)
[root@PGE-NAGIOSLOG ~]# whereis java
java: /usr/bin/java /usr/lib/java /etc/java /usr/share/java /usr/share/man/man1/ java.1.gz
[root@PGE-NAGIOSLOG ~]# sestatus
SELinux status: disabled
[root@PGE-NAGIOSLOG ~]# ps -aux | grep -Ei 'java'
root 44285 0.0 0.0 112812 1004 pts/0 S+ 08:27 0:00 grep --color=au to -Ei java
[root@PGE-NAGIOSLOG ~]# which java
/usr/bin/java
[root@PGE-NAGIOSLOG ~]# java -version
openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8-LTS, mixed mode, sharing)
[root@PGE-NAGIOSLOG ~]# whereis java
java: /usr/bin/java /usr/lib/java /etc/java /usr/share/java /usr/share/man/man1/ java.1.gz
[root@PGE-NAGIOSLOG ~]# sestatus
SELinux status: disabled
[root@PGE-NAGIOSLOG ~]# ps -aux | grep -Ei 'java'
root 44285 0.0 0.0 112812 1004 pts/0 S+ 08:27 0:00 grep --color=au to -Ei java
Re: Elastic search will not start - Java may be the cause.
Hello @isaacl86
Thanks for following and sending over the System Profile, see that it is unable to establish connection to port number 9200.
Restart the following services:
Thanks,
Perry
Thanks for following and sending over the System Profile, see that it is unable to establish connection to port number 9200.
Please make the necessary adjustments to security rules and follow up with updated System Profile.
Failed connect to localhost:9200; Connection refused
Restart the following services:
Code: Select all
systemctl restart elasticsearch logstash httpd
Perry
Re: Elastic search will not start - Java may be the cause.
Hi Perry,
Thanks for your help. I am sending the updated system profile now.
Thanks for your help. I am sending the updated system profile now.
Re: Elastic search will not start - Java may be the cause.
Hello @isaacl86
Thanks for following up elasticsearh.service is executing /bin/java command line. What version here:
Let us know what that looks like,
Perry
Thanks for following up elasticsearh.service is executing /bin/java command line. What version here:
Code: Select all
/bin/java -version
Perry
Re: Elastic search will not start - Java may be the cause.
[root@PGE-NAGIOSLOG ~]# /bin/java -version
openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8-LTS, mixed mode, sharing)
[root@PGE-NAGIOSLOG ~]#
openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8-LTS, mixed mode, sharing)
[root@PGE-NAGIOSLOG ~]#
Re: Elastic search will not start - Java may be the cause.
Hello @isaacl86
Want to have you run through the Nagios log Server Elasticserch installer by downloading the matching version:
https://assets.nagios.com/downloads/nag ... rsions.php
[*]systemctl restart elasticsearch[/*]
[*]systemctl status elasticsearch[/*][/list]
Let us know how things look, there is also a ./upgrade option as well found in the '/nagioslogserver/subcomponents/elasticsearch' directory.
Thanks,
Perry
Want to have you run through the Nagios log Server Elasticserch installer by downloading the matching version:
https://assets.nagios.com/downloads/nag ... rsions.php
- Extract the contents of the installer: [list]
- cd /nagioslogserver/subcomponents/elasticsearch
- ./install
[*]systemctl restart elasticsearch[/*]
[*]systemctl status elasticsearch[/*][/list]
Let us know how things look, there is also a ./upgrade option as well found in the '/nagioslogserver/subcomponents/elasticsearch' directory.
Thanks,
Perry
Re: Elastic search will not start - Java may be the cause.
Hi,
Please see the result below.
[root@PGE-NAGIOSLOG elasticsearch]# ./install
Installing Elasticsearch...
Elasticsearch installed OK
[root@PGE-NAGIOSLOG elasticsearch]# systemctl deamonreload
Unknown operation 'deamonreload'.
[root@PGE-NAGIOSLOG elasticsearch]# systemctl deamon-reload
Unknown operation 'deamon-reload'.
[root@PGE-NAGIOSLOG elasticsearch]# systemctl daemon-reload
[root@PGE-NAGIOSLOG elasticsearch]# systemctl restart elasticsearch
[root@PGE-NAGIOSLOG elasticsearch]# systemctl status elasticsearch
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
Active: active (exited) since Tue 2022-02-15 09:53:21 EST; 7s ago
Docs: man:systemd-sysv-generator(8)
Process: 34257 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 34269 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Stopped LSB: This service manages the elasticsearch daemon.
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl runuser[34286]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[34269]: Starting elasticsearch: [ OK ]
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Started LSB: This service manages the elasticsearch daemon.
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[34269]: Unrecognized VM option 'UseParNewGC'
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[34269]: Error: Could not create the Java Virtual Machine.
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[34269]: Error: A fatal exception has occurred. Program will exit.
[root@PGE-NAGIOSLOG elasticsearch]#
Please see the result below.
[root@PGE-NAGIOSLOG elasticsearch]# ./install
Installing Elasticsearch...
Elasticsearch installed OK
[root@PGE-NAGIOSLOG elasticsearch]# systemctl deamonreload
Unknown operation 'deamonreload'.
[root@PGE-NAGIOSLOG elasticsearch]# systemctl deamon-reload
Unknown operation 'deamon-reload'.
[root@PGE-NAGIOSLOG elasticsearch]# systemctl daemon-reload
[root@PGE-NAGIOSLOG elasticsearch]# systemctl restart elasticsearch
[root@PGE-NAGIOSLOG elasticsearch]# systemctl status elasticsearch
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
Active: active (exited) since Tue 2022-02-15 09:53:21 EST; 7s ago
Docs: man:systemd-sysv-generator(8)
Process: 34257 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 34269 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Stopped LSB: This service manages the elasticsearch daemon.
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl runuser[34286]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[34269]: Starting elasticsearch: [ OK ]
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Started LSB: This service manages the elasticsearch daemon.
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[34269]: Unrecognized VM option 'UseParNewGC'
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[34269]: Error: Could not create the Java Virtual Machine.
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[34269]: Error: A fatal exception has occurred. Program will exit.
[root@PGE-NAGIOSLOG elasticsearch]#
Re: Elastic search will not start - Java may be the cause.
Hello @isaacI86
Thanks for following up, did some digging and you are correct that Java version 11 is not officially supported. Ran some tests on my VM and installed OpenJDK:
Thanks,
Perry
Thanks for following up, did some digging and you are correct that Java version 11 is not officially supported. Ran some tests on my VM and installed OpenJDK:
See that the java parm; "JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC" is no longer compatible with OpenJDK 11. The substitution that seems to work:There are 2 programs which provide 'java'.Code: Select all
alternatives --config java
Selection Command
-----------------------------------------------
* 1 java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/bin/java)
+ 2 java-11-openjdk.x86_64 (/usr/lib/jvm/java-11-openjdk-11.0.14.0.9-2.el8_5.x86_64/bin/java)
And:# There are no JAVA_OPTS set from the client, we set a predefinedCode: Select all
vi /usr/local/nagioslogserver/logstash/bin/logstash.lib.sh
# set of options that think are good in general
#JAVA_OPTS="-XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC"
Since is not a tested solution please verify. The alternative is the go-ahead and install java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/bin/java).#JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC"Code: Select all
vi /usr/local/nagioslogserver/elasticsearch/bin/elasticsearch.in.sh
#JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
Thanks,
Perry