Nagios Support Forum

When did
[root@a-sesklnglsprd1 ~]# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
curl: (7) Failed connect to localhost:9200; Connection refused

out put for the command
"tar czf ~/nagioslogserverlogs.tar.gz /var/log/logstash/* /var/log/elasticsearch/* "

{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-05-19T03:35:22.284000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}


Elasticsearch service status on the Nagios Log server

"elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
Active: active (exited) since Fri 2017-05-19 10:09:06 GMT; 19min ago
Docs: man:systemd-sysv-generator(8)
Process: 4553 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 4642 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)

May 19 10:09:06 a-sesklnglsprd1.astrazeneca.net systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
May 19 10:09:06 a-sesklnglsprd1.astrazeneca.net runuser[4653]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 19 10:09:06 a-sesklnglsprd1.astrazeneca.net elasticsearch[4642]: Starting elasticsearch: /bin/java: error while loading shared libraries: libjli.so: cannot open shared object file:... directory
May 19 10:09:06 a-sesklnglsprd1.astrazeneca.net elasticsearch[4642]: [ OK ]
May 19 10:09:06 a-sesklnglsprd1.astrazeneca.net systemd[1]: Started LSB: This service manages the elasticsearch daemon.
Hint: Some lines were ellipsized, use -l to show in full. \"

What could be the reason behind .

It looks like a libjli.so is missing. Try the following to verify/locate it:
What OS is this on? On my CentOS 7 install this file found under:

[root@a-sesklnglsprd1 logstash]# yum -y install mlocate
Loaded plugins: langpacks, product-id, rhnplugin, search-disabled-repos, subscription-manager, versionlock
This system is receiving updates from RHN Classic or Red Hat Satellite.
Package mlocate-0.26-6.el7.x86_64 already installed and latest version
Nothing to do
You have new mail in /var/spool/mail/root
[root@a-sesklnglsprd1 logstash]# updatedb
[root@a-sesklnglsprd1 logstash]# locate libjli.so
/opt/morpheus-node/embedded/java/lib/amd64/jli/libjli.so
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.41.x86_64/jre/lib/amd64/jli/libjli.so
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el7_3.x86_64/jre/lib/amd64/jli/libjli.so
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/lib/amd64/jli/libjli.so

What else has been installed on the server? The output would suggest that this is not a minimal install of the OS which is one of the requirements of. It is likely that something additional has caused conflict in the Java configuration.

Open /etc/java/java.conf with an editor like vi. It should have a line like:
Try changing it to:
and then restarting the service:

No change!

[root@a-sesklnglsprd1 ~]# service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
Active: active (exited) since Fri 2017-05-19 18:46:25 GMT; 7s ago
Docs: man:systemd-sysv-generator(8)
Process: 21465 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 21472 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)

May 19 18:46:25 a-sesklnglsprd1.astrazeneca.net systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
May 19 18:46:25 a-sesklnglsprd1.astrazeneca.net runuser[21483]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 19 18:46:25 a-sesklnglsprd1.astrazeneca.net elasticsearch[21472]: Starting elasticsearch: /bin/java: error while loading shared libraries: libjli.so: cannot open shared object file... directory
May 19 18:46:25 a-sesklnglsprd1.astrazeneca.net elasticsearch[21472]: [ OK ]
May 19 18:46:25 a-sesklnglsprd1.astrazeneca.net systemd[1]: Started LSB: This service manages the elasticsearch daemon.
Hint: Some lines were ellipsized, use -l to show in full.

I see you have also opened an email ticket for this, so we will be locking this thread and continuing in the email ticket. In the future please only open one or the other so we are not duplicating our efforts.