Nagios Support Forum

Hi Everyone,
I have been trying to fix some vulnerabilities found by our scanner on the nagios log server. It seems like I cannot start elasticsearch right now.
On the web page i'm getting the waiting for database startup message. When I look at the server itself i get this message from elastic search status. (see below).
Has anyone run into this issue? I believe it has to do with Java.

[root@PGE-NAGIOSLOG ~]# service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
Active: active (exited) since Mon 2022-02-07 01:43:00 EST; 9h ago
Docs: man:systemd-sysv-generator(8)
Process: 1010 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)

Feb 07 01:42:56 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
Feb 07 01:43:00 PGE-NAGIOSLOG.powergrid.lcl runuser[1245]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
Feb 07 01:43:00 PGE-NAGIOSLOG.powergrid.lcl runuser[1245]: pam_unix(runuser:session): session closed for user nagios
Feb 07 01:43:00 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[1010]: Starting elasticsearch: [ OK ]
Feb 07 01:43:00 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Started LSB: This service manages the elasticsearch daemon.
Feb 07 01:42:04 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[1010]: Unrecognized VM option 'UseParNewGC'
Feb 07 01:42:04 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[1010]: Error: Could not create the Java Virtual Machine.
Feb 07 01:42:04 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[1010]: Error: A fatal exception has occurred. Program will exit.
[root@PGE-NAGIOSLOG ~]#

Hello @isaacl86

Thanks for reaching out, want to get the following:

• Java:
[list]

• Code: Select all

  which java

• Code: Select all

  java -version

• Code: Select all

  whereis java

• Code: Select all

  sestatus

• Code: Select all

  ps -aux | grep -Ei 'java'

[/list]

Along with that let's also get the log server system profile:
The System Profile is found in '/tmp/system-profile.tar.gz'

May need to use the split command to size down the compressed Profile.
Please send each 'part[x]' in a separate PM.

Thanks,
Perry

Command outputs below
[root@PGE-NAGIOSLOG ~]# which java
/usr/bin/java

[root@PGE-NAGIOSLOG ~]# java -version
openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8-LTS, mixed mode, sharing)

[root@PGE-NAGIOSLOG ~]# whereis java
java: /usr/bin/java /usr/lib/java /etc/java /usr/share/java /usr/share/man/man1/ java.1.gz

[root@PGE-NAGIOSLOG ~]# sestatus
SELinux status: disabled

[root@PGE-NAGIOSLOG ~]# ps -aux | grep -Ei 'java'
root 44285 0.0 0.0 112812 1004 pts/0 S+ 08:27 0:00 grep --color=au to -Ei java

Hello @isaacl86

Thanks for following and sending over the System Profile, see that it is unable to establish connection to port number 9200.

Failed connect to localhost:9200; Connection refused

Please make the necessary adjustments to security rules and follow up with updated System Profile.

Restart the following services: Thanks,
Perry

Hi Perry,
Thanks for your help. I am sending the updated system profile now.

Hello @isaacl86

Thanks for following up elasticsearh.service is executing /bin/java command line. What version here:
Let us know what that looks like,
Perry

[root@PGE-NAGIOSLOG ~]# /bin/java -version
openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8-LTS, mixed mode, sharing)
[root@PGE-NAGIOSLOG ~]#

Hello @isaacl86

Want to have you run through the Nagios log Server Elasticserch installer by downloading the matching version:

https://assets.nagios.com/downloads/nag ... rsions.php

• Extract the contents of the installer:
[list]
• cd /nagioslogserver/subcomponents/elasticsearch
• ./install

[*]systemctl deamon-reload[/*]
[*]systemctl restart elasticsearch[/*]
[*]systemctl status elasticsearch[/*][/list]

Let us know how things look, there is also a ./upgrade option as well found in the '/nagioslogserver/subcomponents/elasticsearch' directory.

Thanks,
Perry

Hi,
Please see the result below.


[root@PGE-NAGIOSLOG elasticsearch]# ./install
Installing Elasticsearch...
Elasticsearch installed OK
[root@PGE-NAGIOSLOG elasticsearch]# systemctl deamonreload
Unknown operation 'deamonreload'.
[root@PGE-NAGIOSLOG elasticsearch]# systemctl deamon-reload
Unknown operation 'deamon-reload'.
[root@PGE-NAGIOSLOG elasticsearch]# systemctl daemon-reload
[root@PGE-NAGIOSLOG elasticsearch]# systemctl restart elasticsearch
[root@PGE-NAGIOSLOG elasticsearch]# systemctl status elasticsearch
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
Active: active (exited) since Tue 2022-02-15 09:53:21 EST; 7s ago
Docs: man:systemd-sysv-generator(8)
Process: 34257 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 34269 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)

Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Stopped LSB: This service manages the elasticsearch daemon.
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl runuser[34286]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[34269]: Starting elasticsearch: [ OK ]
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl systemd[1]: Started LSB: This service manages the elasticsearch daemon.
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[34269]: Unrecognized VM option 'UseParNewGC'
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[34269]: Error: Could not create the Java Virtual Machine.
Feb 15 09:53:21 PGE-NAGIOSLOG.powergrid.lcl elasticsearch[34269]: Error: A fatal exception has occurred. Program will exit.
[root@PGE-NAGIOSLOG elasticsearch]#

Hello @isaacI86

Thanks for following up, did some digging and you are correct that Java version 11 is not officially supported. Ran some tests on my VM and installed OpenJDK:

Code: Select all

alternatives --config java

There are 2 programs which provide 'java'.

Selection Command
-----------------------------------------------
* 1 java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/bin/java)
+ 2 java-11-openjdk.x86_64 (/usr/lib/jvm/java-11-openjdk-11.0.14.0.9-2.el8_5.x86_64/bin/java)

See that the java parm; "JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC" is no longer compatible with OpenJDK 11. The substitution that seems to work:

Code: Select all

vi /usr/local/nagioslogserver/logstash/bin/logstash.lib.sh

# There are no JAVA_OPTS set from the client, we set a predefined
# set of options that think are good in general
#JAVA_OPTS="-XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC"

And:

Code: Select all

vi /usr/local/nagioslogserver/elasticsearch/bin/elasticsearch.in.sh

#JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC"
#JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"

Since is not a tested solution please verify. The alternative is the go-ahead and install java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/bin/java).

Thanks,
Perry