Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

elasticsearch service exited

https://support.nagios.com/forum/viewtopic.php?t=54084

Page 2 of 2

Re: elasticsearch service exited

Posted: Tue Jun 11, 2019 3:45 pm
by scottwilkerson
Oh, so someone has already modified this, and at 20480m I could see how you could get errors when you only had 32GB memory.

I would still try going down to 16g and see how that works our for you.

Re: elasticsearch service exited

Posted: Mon Jun 17, 2019 9:34 am
by Sampath.Basireddy
@scottwilkerson,

Even after setting up ES_HEAP_SIZE limit, elasticsearch service continues to exit. Not as frequent as earlier though.

Code: Select all

root@nagiosls1 ~]# cat /etc/sysconfig/elasticsearch
# Directory where the Elasticsearch binary distribution resides
APP_DIR="/usr/local/nagioslogserver"
ES_HOME="$APP_DIR/elasticsearch"

# Heap Size (defaults to 256m min, 1g max)
# Nagios Log Server Default to 0.5 physical Memory
ES_HEAP_SIZE=16g

# Heap new generation
#ES_HEAP_NEWSIZE=
And, every time the service exists and restarts, server is going into hung state as java process is taking lot of memory & cpu and load average too spikes.

Re: elasticsearch service exited

Posted: Mon Jun 17, 2019 4:25 pm
by ssax
What is the output of this command again (when it's having the issue):

Code: Select all

ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -10
Include the last 200 or so lines from your elasticsearch logfile again, are they the same errors?

Please PM one of us a FRESH system profile from Admin > System Status > Download System Profile as well.

Re: elasticsearch service exited

Posted: Tue Jun 18, 2019 12:08 am
by Sampath.Basireddy
One other strange behavior I noticed since last 2days is, each time service exits and after restarting both elasticsearch ang logstash service, server is going into hung state and I have reboot the server to get response. And again service goes down in under less than hour.

Output of ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -10

Code: Select all

[root@nagiosls1 elasticsearch]# ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -10
 0.7 89.4 8290880 8084 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx1024m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
%MEM %CPU    VSZ   PID CMD
 0.0  1.9      0 10419 [kworker/4:0]
 0.0  1.7 292812 15991 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
 0.0  1.5 292684 15990 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
 0.0  0.7 191516     1 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
 0.0  0.7      0    65 [kswapd0]
 0.0  0.6 2446144 7548 /opt/rapid7/ir_agent/components/insight_agent/2.6.7.9/ir_agent
 0.0  0.5 588660  8050 /opt/microsoft/configmgr/bin/ccmexec.bin
 0.0  0.3 614684  8815 /opt/rapid7/ir_agent/components/insight_agent/2.6.7.9/ir_agent --multiprocessing-fork tracker_fd=16 pipe_handle=20
[root@nagiosls1 elasticsearch]#
Last 200 lines from /var/log/elasticsearch/92ab601c-2645-49fa-98ab-52d987a0a647.log

Code: Select all

[root@nagiosls1 elasticsearch]# tail -200 /var/log/elasticsearch/92ab601c-2645-49fa-98ab-52d987a0a647.log
	at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
	at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2019-06-18 00:30:27,730][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2019-06-18 00:30:27,743][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2019-06-18 00:30:27,758][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2019-06-18 00:30:27,758][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2019-06-18 00:31:05,739][INFO ][cluster.metadata         ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18] update_mapping [%{type}] (dynamic)
[2019-06-18 00:39:02,880][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][2] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto4ZXevl0EXvzLdGgC], source[{"message":"{\"auditId\":\"b743fd8d-09e2-42cb-b45a-dc1931237950\",\"correlationId\":\"AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:38:54\",\"userIP\":\"10.10.38.207\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.207\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"details\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/001/00001850198/details\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:39:02.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:39:02","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"b743fd8d-09e2-42cb-b45a-dc1931237950","correlationId":"AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:38:54","userIP":"10.10.38.207","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.207","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"details","uri":"/api-trinet-auth/services/v2.0/employee/001/00001850198/details","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[2019-06-18 00:39:02,880][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][3] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto4ZXevl0EXvzLdGgB], source[{"message":"{\"auditId\":\"0587eb83-013e-40ff-9c6d-8e0f840215da\",\"correlationId\":\"AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:38:54\",\"userIP\":\"10.10.38.205\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.205\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"details\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/001/00001850198/details\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:39:02.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:39:02","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"0587eb83-013e-40ff-9c6d-8e0f840215da","correlationId":"AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:38:54","userIP":"10.10.38.205","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.205","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"details","uri":"/api-trinet-auth/services/v2.0/employee/001/00001850198/details","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[2019-06-18 00:39:02,882][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][3] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto4ZXevl0EXvzLdGgG], source[{"message":"{\"auditId\":\"f0ca56a1-18b1-493a-ab55-fe083b6484ee\",\"correlationId\":\"AQIC5wM2LY4SfczVCf2Y6qytfbjYxoRTKNE0qXkyY8N8Gg8.*AAJTSQACMTAAAlNLABQtNjIyMDI2OTEwOTc3MjY5OTU4NgACUzEAAjAz*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:38:54\",\"userIP\":\"10.10.38.207\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.207\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"details\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/TUE/00002078745/details\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:39:02.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:39:02","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"f0ca56a1-18b1-493a-ab55-fe083b6484ee","correlationId":"AQIC5wM2LY4SfczVCf2Y6qytfbjYxoRTKNE0qXkyY8N8Gg8.*AAJTSQACMTAAAlNLABQtNjIyMDI2OTEwOTc3MjY5OTU4NgACUzEAAjAz*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:38:54","userIP":"10.10.38.207","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.207","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"details","uri":"/api-trinet-auth/services/v2.0/employee/TUE/00002078745/details","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4SfczVCf2Y6qytfbjYxoRTKNE0qXkyY8N8Gg8.*AAJTSQACMTAAAlNLABQtNjIyMDI2OTEwOTc3MjY5OTU4NgACUzEAAjAz*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[2019-06-18 00:39:02,881][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][4] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto4ZXevl0EXvzLdGgF], source[{"message":"{\"auditId\":\"b418cba4-8c89-4b31-9ae3-8a954ed54f61\",\"correlationId\":\"AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:38:54\",\"userIP\":\"10.10.38.205\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.205\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"employee-privileges\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/001/00001850198/employee-privileges\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:39:02.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:39:02","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"b418cba4-8c89-4b31-9ae3-8a954ed54f61","correlationId":"AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:38:54","userIP":"10.10.38.205","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.205","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"employee-privileges","uri":"/api-trinet-auth/services/v2.0/employee/001/00001850198/employee-privileges","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4SfcwQxY_QNRNEv9z5NcopMqvCPOpMjvjbfYM.*AAJTSQACMTAAAlNLABQtNDEyNjAxMDkzNjE0MjY0MzI3MgACUzEAAjAx*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[2019-06-18 00:39:22,664][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][2] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto4eMhvl0EXvzLdH6T], source[{"message":"{\"auditId\":\"f60d9ae3-f856-4e40-b5ce-54c631280ab1\",\"correlationId\":\"AQIC5wM2LY4Sfcwp5oaCePrRz6HrebYiQuzP-ltQYWiaK3g.*AAJTSQACMTAAAlNLABM0MDYwNzU2MDYyMDY1Mjg5NzUwAAJTMQACMDM.*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:39:13\",\"userIP\":\"10.10.38.207\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.207\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"employee-privileges\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/37H/00001014268/employee-privileges\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:39:22.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:39:22","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"f60d9ae3-f856-4e40-b5ce-54c631280ab1","correlationId":"AQIC5wM2LY4Sfcwp5oaCePrRz6HrebYiQuzP-ltQYWiaK3g.*AAJTSQACMTAAAlNLABM0MDYwNzU2MDYyMDY1Mjg5NzUwAAJTMQACMDM.*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:39:13","userIP":"10.10.38.207","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.207","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"employee-privileges","uri":"/api-trinet-auth/services/v2.0/employee/37H/00001014268/employee-privileges","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4Sfcwp5oaCePrRz6HrebYiQuzP-ltQYWiaK3g.*AAJTSQACMTAAAlNLABM0MDYwNzU2MDYyMDY1Mjg5NzUwAAJTMQACMDM.*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[2019-06-18 00:39:22,664][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][3] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto4eMhvl0EXvzLdH6S], source[{"message":"{\"auditId\":\"81d672df-f638-484e-8f9c-fea26380179f\",\"correlationId\":\"AQIC5wM2LY4Sfcwp5oaCePrRz6HrebYiQuzP-ltQYWiaK3g.*AAJTSQACMTAAAlNLABM0MDYwNzU2MDYyMDY1Mjg5NzUwAAJTMQACMDM.*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:39:12\",\"userIP\":\"10.10.38.207\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.207\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"details\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/37H/00001014268/details\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:39:22.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:39:22","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"81d672df-f638-484e-8f9c-fea26380179f","correlationId":"AQIC5wM2LY4Sfcwp5oaCePrRz6HrebYiQuzP-ltQYWiaK3g.*AAJTSQACMTAAAlNLABM0MDYwNzU2MDYyMDY1Mjg5NzUwAAJTMQACMDM.*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:39:12","userIP":"10.10.38.207","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.207","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"details","uri":"/api-trinet-auth/services/v2.0/employee/37H/00001014268/details","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4Sfcwp5oaCePrRz6HrebYiQuzP-ltQYWiaK3g.*AAJTSQACMTAAAlNLABM0MDYwNzU2MDYyMDY1Mjg5NzUwAAJTMQACMDM.*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[2019-06-18 00:47:24,161][WARN ][monitor.jvm              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [gc][young][1134][140] duration [1s], collections [1]/[1.5s], total [1s]/[12.4s], memory [1.8gb]->[1.4gb]/[15.9gb], all_pools {[young] [404.1mb]->[7.8mb]/[532.5mb]}{[survivor] [33.3mb]->[33.7mb]/[66.5mb]}{[old] [1.4gb]->[1.4gb]/[15.3gb]}
[2019-06-18 00:47:24,161][DEBUG][action.bulk              ] [745e0cf8-855b-48c9-91c0-3973b12ae3bf] [logstash-2019.06.18][3] failed to execute bulk item (index) index {[logstash-2019.06.18][syslog][AWto6Teevl0EXvzLdnay], source[{"message":"{\"auditId\":\"2ec6b78c-b5f8-48f4-b2e1-a0a6a94c532d\",\"correlationId\":\"AQIC5wM2LY4SfcwzvLTfDtbzsoiX-wNo5Xtm6FGkEgrB13Q.*AAJTSQACMTAAAlNLABQtODkyMzY2NTc2MzcwMTY0NzY3NgACUzEAAjAy*\",\"employeeId\":null,\"companyId\":null,\"proxyEmployeeId\":null,\"proxyPersonId\":null,\"proxyCompanyId\":null,\"timeStamp\":\"2019-06-18 00:47:12\",\"userIP\":\"10.10.38.208\",\"browserInfo\":\"Java/1.8.0_162\",\"osInfo\":\"Linux\",\"deviceInfo\":null,\"environment\":\"10.10.38.208\",\"serverIP\":\"microval.hrpassport.com\",\"application\":null,\"feature\":\"v2.0\",\"resource\":\"details\",\"uri\":\"/api-trinet-auth/services/v2.0/employee/001/00001850198/details\",\"bizEvent\":null,\"method\":\"GET\",\"statusCode\":null,\"errorCode\":null,\"request\":null,\"response\":null}\n","@version":"1","@timestamp":"2019-06-18T04:47:22.000Z","type":"syslog","host":"10.10.38.196","priority":187,"timestamp":"Jun 18 00:47:22","logsource":"val01brmssec01","program":"apiAudit","severity":3,"facility":23,"facility_label":"local7","auditId":"2ec6b78c-b5f8-48f4-b2e1-a0a6a94c532d","correlationId":"AQIC5wM2LY4SfcwzvLTfDtbzsoiX-wNo5Xtm6FGkEgrB13Q.*AAJTSQACMTAAAlNLABQtODkyMzY2NTc2MzcwMTY0NzY3NgACUzEAAjAy*","employeeId":null,"companyId":null,"proxyEmployeeId":null,"proxyPersonId":null,"proxyCompanyId":null,"timeStamp":"2019-06-18 00:47:12","userIP":"10.10.38.208","browserInfo":"Java/1.8.0_162","osInfo":"Linux","deviceInfo":null,"environment":"10.10.38.208","serverIP":"microval.hrpassport.com","application":null,"feature":"v2.0","resource":"details","uri":"/api-trinet-auth/services/v2.0/employee/001/00001850198/details","bizEvent":null,"method":"GET","statusCode":null,"errorCode":null,"request":null,"response":null}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [correlationId]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:465)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AQIC5wM2LY4SfcwzvLTfDtbzsoiX-wNo5Xtm6FGkEgrB13Q.*AAJTSQACMTAAAlNLABQtODkyMzY2NTc2MzcwMTY0NzY3NgACUzEAAjAy*"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.parseLong(Long.java:631)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
	at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
[root@nagiosls1 elasticsearch]#

Re: elasticsearch service exited

Posted: Tue Jun 18, 2019 10:43 am
by cdienger
The profile was unable to be generated properly and contains the message:
Unable to generate system profile!<br>Please try manually running:<br><pre>sudo /usr/local/nagioslogserver/scripts/profile.sh</pre>
/etc/sudoers should have a section like so:

Code: Select all

User_Alias NAGIOSLOGSERVER=nagios
User_Alias NAGIOSLOGSERVERWEB=apache
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/change_timezone.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/get_logstash_ports.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/profile.sh
Please verify this file and try gathering a profile from the command line with:

sudo /usr/local/nagioslogserver/scripts/profile.sh

This should create /tmp/system-profile.tar.gz.

Note that this file can be very large and may not be able to be uploaded due to size. This is usually due to the logs in the logstash and/or elasticseach directories found in it. If it is too large, please open the profile, extract these directories/files and send them separately.

Re: elasticsearch service exited

Posted: Tue Jun 18, 2019 12:17 pm
by Sampath.Basireddy
I checked /etc/sudoers, I see all the lines you listed except NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/profile.sh

I tried gathering profile from command line, but the file ended up being 300+mb.

Are the any specific files you want from the ZIP file or its entire contents?

Re: elasticsearch service exited

Posted: Tue Jun 18, 2019 2:27 pm
by cdienger
It'd be best to get everything. If you don't want to send the files separately then please upload them to a secure file sharing site and provide a link that we can use to download it.

Re: elasticsearch service exited

Posted: Tue Jun 18, 2019 3:10 pm
by Sampath.Basireddy
What is the maximum size of attachments I can upload here?

There are some zip files within the zip files which are more than 20 to 30mb in size. Extracting each file and separating then is going to take for ever and lot of files.

I don't have any secure file sharing site. :(


Can I create a support request and upload the files there?

Re: elasticsearch service exited

Posted: Wed Jun 19, 2019 11:38 am
by cdienger
Yes, please open a ticket at https://support.nagios.com/tickets/.
All times are UTC-05:00
Page 2 of 2

Powered by phpBB® Forum Software © phpBB Limited