elasticsearch service exited

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:
Contact scottwilkerson

Re: elasticsearch service exited

Post by scottwilkerson »

Oh, so someone has already modified this, and at 20480m I could see how you could get errors when you only had 32GB memory.

I would still try going down to 16g and see how that works our for you.
Former Nagios employee
Creator:
Human Design Website
Get Your Human Design Chart
Top
Sampath.Basireddy
Posts: 252
Joined: Wed Dec 14, 2016 12:30 pm

Re: elasticsearch service exited

Post by Sampath.Basireddy »

@scottwilkerson,

Even after setting up ES_HEAP_SIZE limit, elasticsearch service continues to exit. Not as frequent as earlier though.

Code: Select all

root@nagiosls1 ~]# cat /etc/sysconfig/elasticsearch
# Directory where the Elasticsearch binary distribution resides
APP_DIR="/usr/local/nagioslogserver"
ES_HOME="$APP_DIR/elasticsearch"

# Heap Size (defaults to 256m min, 1g max)
# Nagios Log Server Default to 0.5 physical Memory
ES_HEAP_SIZE=16g

# Heap new generation
#ES_HEAP_NEWSIZE=
And, every time the service exists and restarts, server is going into hung state as java process is taking lot of memory & cpu and load average too spikes.
Top
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: elasticsearch service exited

Post by ssax »

What is the output of this command again (when it's having the issue):

Code: Select all

ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -10
Include the last 200 or so lines from your elasticsearch logfile again, are they the same errors?

Please PM one of us a FRESH system profile from Admin > System Status > Download System Profile as well.
Top
Sampath.Basireddy
Posts: 252
Joined: Wed Dec 14, 2016 12:30 pm

Re: elasticsearch service exited

Post by Sampath.Basireddy »

One other strange behavior I noticed since last 2days is, each time service exits and after restarting both elasticsearch ang logstash service, server is going into hung state and I have reboot the server to get response. And again service goes down in under less than hour.

Output of ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -10

Code: Select all

[root@nagiosls1 elasticsearch]# ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -10
 0.7 89.4 8290880 8084 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx1024m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
%MEM %CPU    VSZ   PID CMD
 0.0  1.9      0 10419 [kworker/4:0]
 0.0  1.7 292812 15991 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
 0.0  1.5 292684 15990 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
 0.0  0.7 191516     1 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
 0.0  0.7      0    65 [kswapd0]
 0.0  0.6 2446144 7548 /opt/rapid7/ir_agent/components/insight_agent/2.6.7.9/ir_agent
 0.0  0.5 588660  8050 /opt/microsoft/configmgr/bin/ccmexec.bin
 0.0  0.3 614684  8815 /opt/rapid7/ir_agent/components/insight_agent/2.6.7.9/ir_agent --multiprocessing-fork tracker_fd=16 pipe_handle=20
[root@nagiosls1 elasticsearch]#
Last 200 lines from /var/log/elasticsearch/92ab601c-2645-49fa-98ab-52d987a0a647.log

Code: Select all

[root@nagiosls1 elasticsearch]# tail -200 /var/log/elasticsearch/92ab601c-2645-49fa-98ab-52d987a0a647.log
	at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
	at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2019-06-18 00:30:27,730][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2019-06-18 00:30:27,743][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2019-06-18 00:30:27,758][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2019-06-18 00:30:27,758][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2019-06-18 00:31:05,739][INFO ][cluster.metadata         ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18] update_mapping [%{type}] (dynamic)
[2019-06-18 00:39:02,880][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][2] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto4ZXevl0EXvzLdGgC], source[{"message":"{\"auditId\":\"b743fd8d-09e2-42cb-b45a-dc1931237950\",\"correlationId\":\"AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:38:54\",\"userIP\":\"10.10.38.207\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.207\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"details\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/001/00001850198/details\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:39:02.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:39:02","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"b743fd8d-09e2-42cb-b45a-dc1931237950","correlationId":"AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:38:54","userIP":"10.10.38.207","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.207","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"details","uri":"/api-trinet-auth/services/v2.0/employee/001/00001850198/details","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[2019-06-18 00:39:02,880][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][3] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto4ZXevl0EXvzLdGgB], source[{"message":"{\"auditId\":\"0587eb83-013e-40ff-9c6d-8e0f840215da\",\"correlationId\":\"AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:38:54\",\"userIP\":\"10.10.38.205\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.205\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"details\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/001/00001850198/details\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:39:02.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:39:02","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"0587eb83-013e-40ff-9c6d-8e0f840215da","correlationId":"AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:38:54","userIP":"10.10.38.205","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.205","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"details","uri":"/api-trinet-auth/services/v2.0/employee/001/00001850198/details","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[2019-06-18 00:39:02,882][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][3] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto4ZXevl0EXvzLdGgG], source[{"message":"{\"auditId\":\"f0ca56a1-18b1-493a-ab55-fe083b6484ee\",\"correlationId\":\"AQIC5wM2LY4SfczVCf2Y6qytfbjYxoRTKNE0qXkyY8N8Gg8.*AAJTSQACMTAAAlNLABQtNjIyMDI2OTEwOTc3MjY5OTU4NgACUzEAAjAz*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:38:54\",\"userIP\":\"10.10.38.207\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.207\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"details\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/TUE/00002078745/details\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:39:02.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:39:02","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"f0ca56a1-18b1-493a-ab55-fe083b6484ee","correlationId":"AQIC5wM2LY4SfczVCf2Y6qytfbjYxoRTKNE0qXkyY8N8Gg8.*AAJTSQACMTAAAlNLABQtNjIyMDI2OTEwOTc3MjY5OTU4NgACUzEAAjAz*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:38:54","userIP":"10.10.38.207","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.207","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"details","uri":"/api-trinet-auth/services/v2.0/employee/TUE/00002078745/details","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4SfczVCf2Y6qytfbjYxoRTKNE0qXkyY8N8Gg8.*AAJTSQACMTAAAlNLABQtNjIyMDI2OTEwOTc3MjY5OTU4NgACUzEAAjAz*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[2019-06-18 00:39:02,881][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][4] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto4ZXevl0EXvzLdGgF], source[{"message":"{\"auditId\":\"b418cba4-8c89-4b31-9ae3-8a954ed54f61\",\"correlationId\":\"AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:38:54\",\"userIP\":\"10.10.38.205\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.205\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"employee-privileges\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/001/00001850198/employee-privileges\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:39:02.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:39:02","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"b418cba4-8c89-4b31-9ae3-8a954ed54f61","correlationId":"AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:38:54","userIP":"10.10.38.205","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.205","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"employee-privileges","uri":"/api-trinet-auth/services/v2.0/employee/001/00001850198/employee-privileges","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[2019-06-18 00:39:22,664][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][2] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto4eMhvl0EXvzLdH6T], source[{"message":"{\"auditId\":\"f60d9ae3-f856-4e40-b5ce-54c631280ab1\",\"correlationId\":\"AQIC5wM2LY4Sfcwp5oaCePrRz6HrebYiQuzP-ltQYWiaK3g.*AAJTSQACMTAAAlNLABM0MDYwNzU2MDYyMDY1Mjg5NzUwAAJTMQACMDM.*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:39:13\",\"userIP\":\"10.10.38.207\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.207\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"employee-privileges\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/37H/00001014268/employee-privileges\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:39:22.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:39:22","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"f60d9ae3-f856-4e40-b5ce-54c631280ab1","correlationId":"AQIC5wM2LY4Sfcwp5oaCePrRz6HrebYiQuzP-ltQYWiaK3g.*AAJTSQACMTAAAlNLABM0MDYwNzU2MDYyMDY1Mjg5NzUwAAJTMQACMDM.*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:39:13","userIP":"10.10.38.207","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.207","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"employee-privileges","uri":"/api-trinet-auth/services/v2.0/employee/37H/00001014268/employee-privileges","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4Sfcwp5oaCePrRz6HrebYiQuzP-ltQYWiaK3g.*AAJTSQACMTAAAlNLABM0MDYwNzU2MDYyMDY1Mjg5NzUwAAJTMQACMDM.*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[2019-06-18 00:39:22,664][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][3] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto4eMhvl0EXvzLdH6S], source[{"message":"{\"auditId\":\"81d672df-f638-484e-8f9c-fea26380179f\",\"correlationId\":\"AQIC5wM2LY4Sfcwp5oaCePrRz6HrebYiQuzP-ltQYWiaK3g.*AAJTSQACMTAAAlNLABM0MDYwNzU2MDYyMDY1Mjg5NzUwAAJTMQACMDM.*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:39:12\",\"userIP\":\"10.10.38.207\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.207\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"details\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/37H/00001014268/details\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:39:22.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:39:22","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"81d672df-f638-484e-8f9c-fea26380179f","correlationId":"AQIC5wM2LY4Sfcwp5oaCePrRz6HrebYiQuzP-ltQYWiaK3g.*AAJTSQACMTAAAlNLABM0MDYwNzU2MDYyMDY1Mjg5NzUwAAJTMQACMDM.*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:39:12","userIP":"10.10.38.207","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.207","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"details","uri":"/api-trinet-auth/services/v2.0/employee/37H/00001014268/details","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4Sfcwp5oaCePrRz6HrebYiQuzP-ltQYWiaK3g.*AAJTSQACMTAAAlNLABM0MDYwNzU2MDYyMDY1Mjg5NzUwAAJTMQACMDM.*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[2019-06-18 00:47:24,161][WARN ][monitor.jvm              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [gc][young][1134][140] duration [1s], collections [1]/[1.5s], total [1s]/[12.4s], memory [1.8gb]->[1.4gb]/[15.9gb], all_pools {[young] [404.1mb]->[7.8mb]/[532.5mb]}{[survivor] [33.3mb]->[33.7mb]/[66.5mb]}{[old] [1.4gb]->[1.4gb]/[15.3gb]}
[2019-06-18 00:47:24,161][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][3] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto6Teevl0EXvzLdnay], source[{"message":"{\"auditId\":\"2ec6b78c-b5f8-48f4-b2e1-a0a6a94c532d\",\"correlationId\":\"AQIC5wM2LY4SfcwzvLTfDtbzsoiX-wNo5Xtm6FGkEgrB13Q.*AAJTSQACMTAAAlNLABQtODkyMzY2NTc2MzcwMTY0NzY3NgACUzEAAjAy*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:47:12\",\"userIP\":\"10.10.38.208\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.208\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"details\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/001/00001850198/details\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:47:22.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:47:22","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"2ec6b78c-b5f8-48f4-b2e1-a0a6a94c532d","correlationId":"AQIC5wM2LY4SfcwzvLTfDtbzsoiX-wNo5Xtm6FGkEgrB13Q.*AAJTSQACMTAAAlNLABQtODkyMzY2NTc2MzcwMTY0NzY3NgACUzEAAjAy*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:47:12","userIP":"10.10.38.208","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.208","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"details","uri":"/api-trinet-auth/services/v2.0/employee/001/00001850198/details","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4SfcwzvLTfDtbzsoiX-wNo5Xtm6FGkEgrB13Q.*AAJTSQACMTAAAlNLABQtODkyMzY2NTc2MzcwMTY0NzY3NgACUzEAAjAy*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[root@nagiosls1 elasticsearch]#
You do not have the required permissions to view the files attached to this post.
Top
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: elasticsearch service exited

Post by cdienger »

The profile was unable to be generated properly and contains the message:
Unable to generate system profile!<br>Please try manually running:<br><pre>sudo /usr/local/nagioslogserver/scripts/profile.sh</pre>
/etc/sudoers should have a section like so:

Code: Select all

User_Alias NAGIOSLOGSERVER=nagios
User_Alias NAGIOSLOGSERVERWEB=apache
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/change_timezone.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/get_logstash_ports.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/profile.sh
Please verify this file and try gathering a profile from the command line with:

sudo /usr/local/nagioslogserver/scripts/profile.sh

This should create /tmp/system-profile.tar.gz.

Note that this file can be very large and may not be able to be uploaded due to size. This is usually due to the logs in the logstash and/or elasticseach directories found in it. If it is too large, please open the profile, extract these directories/files and send them separately.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Top
Sampath.Basireddy
Posts: 252
Joined: Wed Dec 14, 2016 12:30 pm

Re: elasticsearch service exited

Post by Sampath.Basireddy »

I checked /etc/sudoers, I see all the lines you listed except NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/profile.sh

I tried gathering profile from command line, but the file ended up being 300+mb.

Are the any specific files you want from the ZIP file or its entire contents?
Top
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: elasticsearch service exited

Post by cdienger »

It'd be best to get everything. If you don't want to send the files separately then please upload them to a secure file sharing site and provide a link that we can use to download it.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Top
Sampath.Basireddy
Posts: 252
Joined: Wed Dec 14, 2016 12:30 pm

Re: elasticsearch service exited

Post by Sampath.Basireddy »

What is the maximum size of attachments I can upload here?

There are some zip files within the zip files which are more than 20 to 30mb in size. Extracting each file and separating then is going to take for ever and lot of files.

I don't have any secure file sharing site. :(


Can I create a support request and upload the files there?
Top
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: elasticsearch service exited

Post by cdienger »

Yes, please open a ticket at https://support.nagios.com/tickets/.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Top
Locked

Return to “Nagios Log Server”