Elasticsearch service failure on Nagios Logserver

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: Elasticsearch service failure on Nagios Logserver

Post by pbroste »

Hello @HIINNS

Thanks for following up, were you able to find the path location on the 'elasticsearch' service executable? I notice that on my test VM it is located in and can start with:
/etc/init.d/elasticsearch start


Circling back we see that you have a running java process with elasticsearch parms:
ps -aux | grep -Ei 'java'
nagios 16768 0.3 52.3 16131124 12843288 ? SLl Nov09 37:12 /bin/java -Xms11973m -Xmx11973m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Des.cluster.name=c0d6d20a-0a3c-4d14-9cda-1f2f4fcb6b55 -Des.node.name=bec4d3fc-0bab-49f5-88cf-fb1094c85cfd -Des.discovery.zen.ping.unicast.hosts=localhost -Des.path.repo=/ -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/local/nagioslogserver/elasticsearch -cp :/usr/local/nagioslogserver/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/local/nagioslogserver/elasticsearch/lib/*:/usr/local/nagioslogserver/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/local/nagioslogserver/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/usr/local/nagioslogserver/elasticsearch/data -Des.default.path.work=/usr/local/nagioslogserver/tmp/elasticsearch -Des.default.path.conf=/usr/local/nagioslogserver/elasticsearch/config org.elasticsearch.bootstrap.Elasticsearch
The java process listed is running elasticsearch with parms, so we should see that elasticsearch is serving requests (using curl on Linux). To verify:

Code: Select all

curl -XGET http://localhost:9200/_status?pretty
curl -XGET 'localhost:9200/_cluster/health?pretty'
Let us know what you are seeing,
Perry
HIINNS
Posts: 172
Joined: Wed Mar 14, 2018 9:43 am

Re: Elasticsearch service failure on Nagios Logserver

Post by HIINNS »

sudo curl -XGET http://localhost:9200/_status?pretty
Last login: Fri Nov 19 05:20:02 EST 2021 on cron
{
"_shards" : {
"total" : 0,
"successful" : 0,
"failed" : 0
},
"indices" : { }
HIINNS
Posts: 172
Joined: Wed Mar 14, 2018 9:43 am

Re: Elasticsearch service failure on Nagios Logserver

Post by HIINNS »

sudo curl -XGET 'localhost:9200/_cluster/health?pretty'
Last login: Fri Nov 19 05:25:26 EST 2021 on pts/0
{
"cluster_name" : "c0d6d20a-0a3c-4d14-9cda-1f2f4fcb6b55",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: Elasticsearch service failure on Nagios Logserver

Post by pbroste »

Hello @HIINNS

Thanks for following up with the results on the api's, which confirm that elasticsearch, and logstash are functioning set with status green.

Minus the elasticsearch, since that is running process java with parameters, let's verify that Apache and logstash are running:

Code: Select all

systemctl status httpd logstash  #Centos
or

Code: Select all

systemctl status apache2.service logstash  #Debian
Verify that elasticsearch is running:

Code: Select all

ps -aux | grep -Ei 'elasticsearch'
That looks good, bring up the web console: https://yourhostaddress/nagioslogserver

Let us know how that looks,
Perry
HIINNS
Posts: 172
Joined: Wed Mar 14, 2018 9:43 am

Re: Elasticsearch service failure on Nagios Logserver

Post by HIINNS »

sudo systemctl status httpd logstash
Last login: Mon Nov 22 05:30:04 EST 2021
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-11-08 12:19:57 EST; 1 weeks 6 days ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 9890 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
Main PID: 18084 (httpd)
Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /system.slice/httpd.service
├─ 9895 /usr/sbin/httpd -DFOREGROUND
├─ 9896 /usr/sbin/httpd -DFOREGROUND
├─ 9897 /usr/sbin/httpd -DFOREGROUND
├─ 9898 /usr/sbin/httpd -DFOREGROUND
├─ 9899 /usr/sbin/httpd -DFOREGROUND
└─18084 /usr/sbin/httpd -DFOREGROUND

Nov 22 03:34:01 nnagsr10 systemd[1]: Reloading The Apache HTTP Server.
Nov 22 03:34:02 nnagsr10 systemd[1]: Reloaded The Apache HTTP Server.

● logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
Active: active (exited) since Mon 2021-11-08 12:20:02 EST; 1 weeks 6 days ago
Docs: man:systemd-sysv-generator(8)
Process: 18156 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
HIINNS
Posts: 172
Joined: Wed Mar 14, 2018 9:43 am

Re: Elasticsearch service failure on Nagios Logserver

Post by HIINNS »

ps -aux | grep -Ei 'elasticsearch'
myuid 5334 0.0 0.0 112812 1008 pts/0 S+ 06:43 0:00 grep --color=auto -Ei elasticsearch
nagios 16768 0.3 52.4 16131124 12861728 ? SLl Nov09 63:24 /bin/java -Xms11973m -Xmx11973m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Des.cluster.name=c0d6d20a-0a3c-4d14-9cda-1f2f4fcb6b55 -Des.node.name=bec4d3fc-0bab-49f5-88cf-fb1094c85cfd -Des.discovery.zen.ping.unicast.hosts=localhost -Des.path.repo=/ -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/local/nagioslogserver/elasticsearch -cp :/usr/local/nagioslogserver/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/local/nagioslogserver/elasticsearch/lib/*:/usr/local/nagioslogserver/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/local/nagioslogserver/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/usr/local/nagioslogserver/elasticsearch/data -Des.default.path.work=/usr/local/nagioslogserver/tmp/elasticsearch -Des.default.path.conf=/usr/local/nagioslogserver/elasticsearch/config org.elasticsearch.bootstrap.Elasticsearch
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: Elasticsearch service failure on Nagios Logserver

Post by pbroste »

Hello @HINNS

That looks good, what do you see when you bring up the web console: https://yourhostaddress/nagioslogserver
  • Are you able to view and add Log Sources?
Please send over the System Profile if see that you are unable to access the log server web console.

Code: Select all

/usr/local/nagioslogserver/scripts/profile.sh[/url]

The System Profile is found in '/tmp/system-profile.tar.gz'

Thanks,
Perry
HIINNS
Posts: 172
Joined: Wed Mar 14, 2018 9:43 am

Re: Elasticsearch service failure on Nagios Logserver

Post by HIINNS »

Hmmm… can't reach this page
myserver refused to connect.

Check PM, let me know if the file doesn't make it.
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: Elasticsearch service failure on Nagios Logserver

Post by pbroste »

Hello @HIINNS

Thanks for following up, looks like the Profile did not make the trip. May need to use the split command to size down the compressed Profile.

Code: Select all

split -b 40M profile.tar.gz part
Please send each 'part[x]' in a separate PM.

The following api command will provide connection feedback on the connection to the api backend on the apache:

Code: Select all

curl -XGET "http://xxx.xxx.xxx.xxx/nagioslogserver/api/backend/nagioslogserver_history/_search?pretty=true" -k
Should receive a message error: true
{"error":true,"type":"authentication","message":"Could not authenticate. Invalid token given."}
Thanks,
Perry
HIINNS
Posts: 172
Joined: Wed Mar 14, 2018 9:43 am

Re: Elasticsearch service failure on Nagios Logserver

Post by HIINNS »

sudo curl -XGET "http://xxx.xxx.xxx.xxx/nagioslogserver/ ... retty=true" -k
Last login: Tue Nov 30 08:20:01 EST 2021 on cron
Locked