Problems with bringing up

[HIINNS]

All,

I have re-installed Nagios Logserver (it was working, but for some reason it wouldn't bring up the login screen, but for some reason I can't log in. The server was SELINUX and I have turned it off. When I execute the http://<NagioslogserverIP>/nagioslogserver/ command instead of being prompted for the nagiosadmin password, it brings me to the login screen. I have executed the /usr/local/nagioslogserver/scripts/reset_nagiosadmin_password.sh to reset the password, but I still can't log in. Also, the application runs extremely slow, it takes over 3 minutes to bring up the login screen. Please help.

[HIINNS]

I have found that the elasticsearch service is not running. See below.

sudo systemctl status -l elasticsearch.service
Last login: Tue Jun 1 07:15:01 EDT 2021 on cron
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2021-06-01 07:08:20 EDT; 7min ago
Docs: https://www.elastic.co
Process: 20048 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 20048 (code=exited, status=1/FAILURE)
CGroup: /system.slice/elasticsearch.service
└─2276 /bin/java -Xms11973m -Xmx11973m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Des.cluster.name=41b0d0a1-5530-4aa5-ae49-0530d464f00e -Des.node.name=046cec17-2e8a-4581-bcda-4680bf033297 -Des.discovery.zen.ping.unicast.hosts=localhost -Des.path.repo=/ -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/local/nagioslogserver/elasticsearch -cp :/usr/local/nagioslogserver/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/local/nagioslogserver/elasticsearch/lib/*:/usr/local/nagioslogserver/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/local/nagioslogserver/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/usr/local/nagioslogserver/elasticsearch/data -Des.default.path.work=/usr/local/nagioslogserver/tmp/elasticsearch -Des.default.path.conf=/usr/local/nagioslogserver/elasticsearch/config org.elasticsearch.bootstrap.Elasticsearch

Jun 01 07:08:20 nnagsr10 systemd[1]: Ignoring invalid environment assignment 'if [ "$GET_ES_CONFIG_RETURN" !=0]; then': /etc/sysconfig/elasticsearch
Jun 01 07:08:20 nnagsr10 systemd[1]: Starting Elasticsearch...
Jun 01 07:08:20 nnagsr10 systemd-entrypoint[20048]: /usr/share/elasticsearch/bin/elasticsearch-env: line 87: /etc/sysconfig/elasticsearch: Permission denied
Jun 01 07:08:20 nnagsr10 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 01 07:08:20 nnagsr10 systemd[1]: Failed to start Elasticsearch.
Jun 01 07:08:20 nnagsr10 systemd[1]: Unit elasticsearch.service entered failed state.
Jun 01 07:08:20 nnagsr10 systemd[1]: elasticsearch.service failed.

[gsmith]

Hi

Please go to /etc/sysconfig and verify the permissions on elasticsearch,
they should be:

-rw-r--r--. 1 root root 1716 Mar 24 15:30 elasticsearch

Fix them if they aren't correct, and if you had to correct them then you
should check the file permissions of the other files in the /etc/sysconfig
directory - I have attached a screen capture so you can make the comparison.

Let me know what you find out.

Thanks

[HIINNS]

Fixed ownership and permissions on file /etc/sysconfig/elasticsearch. New error shown below:

sudo systemctl status -l elasticsearch.service
Last login: Wed Jun 2 05:32:49 EDT 2021 on pts/0
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2021-06-02 05:32:49 EDT; 10s ago
Docs: https://www.elastic.co
Process: 11128 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 11128 (code=exited, status=1/FAILURE)
CGroup: /system.slice/elasticsearch.service
└─2276 /bin/java -Xms11973m -Xmx11973m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Des.cluster.name=41b0d0a1-5530-4aa5-ae49-0530d464f00e -Des.node.name=046cec17-2e8a-4581-bcda-4680bf033297 -Des.discovery.zen.ping.unicast.hosts=localhost -Des.path.repo=/ -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/local/nagioslogserver/elasticsearch -cp :/usr/local/nagioslogserver/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/local/nagioslogserver/elasticsearch/lib/*:/usr/local/nagioslogserver/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/local/nagioslogserver/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/usr/local/nagioslogserver/elasticsearch/data -Des.default.path.work=/usr/local/nagioslogserver/tmp/elasticsearch -Des.default.path.conf=/usr/local/nagioslogserver/elasticsearch/config org.elasticsearch.bootstrap.Elasticsearch

Jun 02 05:32:49 nnagsr10 systemd-entrypoint[11128]: Error: A fatal exception has occurred. Program will exit.
Jun 02 05:32:49 nnagsr10 systemd-entrypoint[11128]: at org.elasticsearch.tools.launchers.JvmOption.flagsFinal(JvmOption.java:119)
Jun 02 05:32:49 nnagsr10 systemd-entrypoint[11128]: at org.elasticsearch.tools.launchers.JvmOption.findFinalOptions(JvmOption.java:81)
Jun 02 05:32:49 nnagsr10 systemd-entrypoint[11128]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:38)
Jun 02 05:32:49 nnagsr10 systemd-entrypoint[11128]: at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:135)
Jun 02 05:32:49 nnagsr10 systemd-entrypoint[11128]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86)
Jun 02 05:32:49 nnagsr10 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 02 05:32:49 nnagsr10 systemd[1]: Failed to start Elasticsearch.
Jun 02 05:32:49 nnagsr10 systemd[1]: Unit elasticsearch.service entered failed state.
Jun 02 05:32:49 nnagsr10 systemd[1]: elasticsearch.service failed.

[gsmith]

Hi,

How much memory does this server have?
What OS is the server running?

elasticsearch is reserving 12 GB of it (/bin/java -Xms11973m -Xmx11973m)

Thanks

[HIINNS]

The server is virtual. It has 4 processor cores and 24 GB of RAM. The OS is Redhat7 FIPS.

[gsmith]

hi

Please run:

Code: Select all

ls -ld /var/run/elasticsearch >> /tmp/62721.log
ls -l /var/run/elasticsearch/elasticsearch.pid >> /tmp/62721.log
ls -ld /usr/local/nagioslogserver/elasticsearch >> /tmp/62721.log
ls -l /usr/local/nagioslogserver/elasticsearch/lib/elasticsearch-1.7.6.jar >> /tmp/62721.log
ls -ld /usr/local/nagioslogserver/elasticsearch/lib/* >> /tmp/62721.log
ls -ld /usr/local/nagioslogserver/elasticsearch/lib/sigar/* >> /tmp/62721.log
ls -ld /usr/local/nagioslogserver/elasticsearch >> /tmp/62721.log
ls -ld /var/log/elasticsearch >> /tmp/62721.log
ls -ld /usr/local/nagioslogserver/elasticsearch/data >> /tmp/62721.log
ls -ld /usr/local/nagioslogserver/tmp/elasticsearch >> /tmp/62721.log
ls -ld /usr/local/nagioslogserver/elasticsearch/config >> /tmp/62721.log
cd /tmp
tar cvzf 62721.tar.gz 62721.log

And send me the 62721.tar.gz file either as an attachment here or as an attachment
to a PM

Thanks

[HIINNS]

File sent in PM. Let me know if not received.

[gsmith]

Hi,

So what I am trying to do is figure out which java option it's not liking.

Error: A fatal exception has occurred. Program will exit.
Jun 02 05:32:49 nnagsr10 systemd-entrypoint[11128]: at org.elasticsearch.tools.launchers.JvmOption.flagsFinal(JvmOption.java:119)
Jun 02 05:32:49 nnagsr10 systemd-entrypoint[11128]: at org.elasticsearch.tools.launchers.JvmOption.findFinalOptions(JvmOption.java:81)
Jun 02 05:32:49 nnagsr10 systemd-entrypoint[11128]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:38)
Jun 02 05:32:49 nnagsr10 systemd-entrypoint[11128]: at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:135)
Jun 02 05:32:49 nnagsr10 systemd-entrypoint[11128]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86)

The only difference between my and your system is the heap size settings: -Xms11973m -Xmx11973m, but the install calculates
those values to be 1/2 of the total physical memory.

That's why I was looking at the permissions on the directories referenced by the java opts in the launch
command. The last few in question didn't get picked up by those commands I sent you so let's
double-check them please:

Code: Select all

cd /var/run/elasticsearch 
and verify permissions on the elasticsearch directory, should be:
drwxr-xr-x  2 nagios nagios    60 Jun  2 14:04 elasticsearch/


cd /usr/local/nagioslogserver/elasticsearch
and verify permissions on the data directory, should be:
drwxr-xr-x  3 nagios nagios  4096 Jun  2 14:04 data/


cd /usr/local/nagioslogserver/
and verify permissions on the tmp directory, should be:
drwxrwxr-x  2 nagios www-data 4096 Jun  2 14:06 tmp/
(Note: notice the group is www-data on this one)

If those need changes go ahead and make them, and then try to start the logserver.

Let me know what happens.

Thanks

[HIINNS]

The data directory was missing in /usr/local/nagioslogserver/elasticsearch. I created the directory and changed ownership and permissions, then tried starting the elasticsearch service. I got the following error.

sudo systemctl -l status elasticsearch.service
Last login: Fri Jun 4 05:30:03 EDT 2021
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2021-06-04 05:20:30 EDT; 14min ago
Docs: https://www.elastic.co
Process: 17008 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 17008 (code=exited, status=1/FAILURE)
CGroup: /system.slice/elasticsearch.service
└─2276 /bin/java -Xms11973m -Xmx11973m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Des.cluster.name=41b0d0a1-5530-4aa5-ae49-0530d464f00e -Des.node.name=046cec17-2e8a-4581-bcda-4680bf033297 -Des.discovery.zen.ping.unicast.hosts=localhost -Des.path.repo=/ -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/local/nagioslogserver/elasticsearch -cp :/usr/local/nagioslogserver/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/local/nagioslogserver/elasticsearch/lib/*:/usr/local/nagioslogserver/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/local/nagioslogserver/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/usr/local/nagioslogserver/elasticsearch/data -Des.default.path.work=/usr/local/nagioslogserver/tmp/elasticsearch -Des.default.path.conf=/usr/local/nagioslogserver/elasticsearch/config org.elasticsearch.bootstrap.Elasticsearch

Jun 04 05:20:30 nnagsr10 systemd-entrypoint[17008]: Error: A fatal exception has occurred. Program will exit.
Jun 04 05:20:30 nnagsr10 systemd-entrypoint[17008]: at org.elasticsearch.tools.launchers.JvmOption.flagsFinal(JvmOption.java:119)
Jun 04 05:20:30 nnagsr10 systemd-entrypoint[17008]: at org.elasticsearch.tools.launchers.JvmOption.findFinalOptions(JvmOption.java:81)
Jun 04 05:20:30 nnagsr10 systemd-entrypoint[17008]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:38)
Jun 04 05:20:30 nnagsr10 systemd-entrypoint[17008]: at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:135)
Jun 04 05:20:30 nnagsr10 systemd-entrypoint[17008]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86)
Jun 04 05:20:30 nnagsr10 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 04 05:20:30 nnagsr10 systemd[1]: Failed to start Elasticsearch.
Jun 04 05:20:30 nnagsr10 systemd[1]: Unit elasticsearch.service entered failed state.
Jun 04 05:20:30 nnagsr10 systemd[1]: elasticsearch.service failed.