New install Logstash failing

[mcapra]

Send over these two pretty please:

Code: Select all

-rw-r--r--. 1 nagios users   691 Sep 10 06:07 043f21b0-1d53-4881-8f3e-9c58be028b8d.log-20160910.gz
-rw-r--r--. 1 nagios users 26834 Sep 21 18:00 1c6d88c6-e9ca-480d-8c85-8bd97ef41adf.log

[BanditBBS]

Here ya go....and FYI - No crash since yesterday afternoon. Should I continue with the special running logstash?

043f21b0-1d53-4881-8f3e-9c58be028b8d.log-20160910.gz

1c6d88c6-e9ca-480d-8c85-8bd97ef41adf.log-20160921.gz

[mcapra]

The old adage of "if it ain't broke, don't fix it" seems appropriate, but the init.d script we use for logstash does some special stuff behind the scenes to optimize memory usage on the machine that may become relevant if the node ever reaches capacity.

Also if you intend to run logstash this way for a long time, I would remove the debug output.

[BanditBBS]

Just connected to the screen session to kill it and saw this. Looks like every 4 hours I get a list of these and then nothing for 4 more hours

Code: Select all

Sep 22, 2016 12:00:11 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [9453] handler org.elasticsearch.action.TransportActionNodeProxy$1@16102657, error [true], resetting
Sep 22, 2016 12:00:11 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [9478] handler org.elasticsearch.action.TransportActionNodeProxy$1@1e23f4f4, error [true], resetting
Sep 22, 2016 12:00:11 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [9479] handler org.elasticsearch.action.TransportActionNodeProxy$1@291d8a98, error [true], resetting
Sep 22, 2016 12:00:11 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [9386] handler org.elasticsearch.action.TransportActionNodeProxy$1@3a605c15, error [true], resetting
Sep 22, 2016 12:00:12 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [9455] handler org.elasticsearch.action.TransportActionNodeProxy$1@fbf52fa, error [true], resetting
Sep 22, 2016 12:00:12 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [9480] handler org.elasticsearch.action.TransportActionNodeProxy$1@3595a991, error [true], resetting
Sep 22, 2016 2:00:05 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11496] handler org.elasticsearch.action.TransportActionNodeProxy$1@4041cd8c, error [true], resetting
Sep 22, 2016 2:00:06 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11539] handler org.elasticsearch.action.TransportActionNodeProxy$1@68a695fc, error [true], resetting
Sep 22, 2016 2:00:06 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11497] handler org.elasticsearch.action.TransportActionNodeProxy$1@58f10c9d, error [true], resetting
Sep 22, 2016 2:00:07 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11540] handler org.elasticsearch.action.TransportActionNodeProxy$1@204aeadd, error [true], resetting
Sep 22, 2016 2:00:07 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11498] handler org.elasticsearch.action.TransportActionNodeProxy$1@4ef0f21d, error [true], resetting
Sep 22, 2016 2:00:08 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11542] handler org.elasticsearch.action.TransportActionNodeProxy$1@66b68b9e, error [true], resetting
Sep 22, 2016 2:00:08 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11500] handler org.elasticsearch.action.TransportActionNodeProxy$1@7898bae1, error [true], resetting
Sep 22, 2016 2:00:09 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11543] handler org.elasticsearch.action.TransportActionNodeProxy$1@478d5c82, error [true], resetting
Sep 22, 2016 2:00:09 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11501] handler org.elasticsearch.action.TransportActionNodeProxy$1@4f0d8122, error [true], resetting
Sep 22, 2016 2:00:10 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11544] handler org.elasticsearch.action.TransportActionNodeProxy$1@33ab9953, error [true], resetting
Sep 22, 2016 2:00:10 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11502] handler org.elasticsearch.action.TransportActionNodeProxy$1@2dba6d22, error [true], resetting
Sep 22, 2016 2:00:11 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11545] handler org.elasticsearch.action.TransportActionNodeProxy$1@38710954, error [true], resetting
Sep 22, 2016 2:00:11 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11503] handler org.elasticsearch.action.TransportActionNodeProxy$1@25606a84, error [true], resetting
Sep 22, 2016 2:00:12 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11546] handler org.elasticsearch.action.TransportActionNodeProxy$1@d4a5c99, error [true], resetting
Sep 22, 2016 2:00:12 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11505] handler org.elasticsearch.action.TransportActionNodeProxy$1@7e6b20b5, error [true], resetting
Sep 22, 2016 2:00:13 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11548] handler org.elasticsearch.action.TransportActionNodeProxy$1@616d6ee5, error [true], resetting
Sep 22, 2016 2:00:13 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11506] handler org.elasticsearch.action.TransportActionNodeProxy$1@fc321a2, error [true], resetting
Sep 22, 2016 2:00:14 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11549] handler org.elasticsearch.action.TransportActionNodeProxy$1@709dd657, error [true], resetting
Sep 22, 2016 6:00:13 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [15756] handler org.elasticsearch.action.TransportActionNodeProxy$1@49357bd5, error [true], resetting
Sep 22, 2016 6:00:14 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [15757] handler org.elasticsearch.action.TransportActionNodeProxy$1@75c115a4, error [true], resetting
Sep 22, 2016 6:00:15 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [15758] handler org.elasticsearch.action.TransportActionNodeProxy$1@a841d4c, error [true], resetting
Sep 22, 2016 10:00:12 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19690] handler org.elasticsearch.action.TransportActionNodeProxy$1@5bfc8b2, error [true], resetting
Sep 22, 2016 10:00:13 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19691] handler org.elasticsearch.action.TransportActionNodeProxy$1@4136a820, error [true], resetting
Sep 22, 2016 10:00:14 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19693] handler org.elasticsearch.action.TransportActionNodeProxy$1@2ae403ca, error [true], resetting
Sep 22, 2016 10:00:15 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19694] handler org.elasticsearch.action.TransportActionNodeProxy$1@6ee13f89, error [true], resetting
Sep 22, 2016 10:00:16 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19695] handler org.elasticsearch.action.TransportActionNodeProxy$1@3335752, error [true], resetting
Sep 22, 2016 10:00:17 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19696] handler org.elasticsearch.action.TransportActionNodeProxy$1@364e8f9c, error [true], resetting
Sep 22, 2016 10:00:18 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19697] handler org.elasticsearch.action.TransportActionNodeProxy$1@4248577e, error [true], resetting

[mcapra]

I would spin up a 3rd node if possible and add it to the cluster. See if it encounters the same problems.

We can move this to email and schedule a remote if you like as well. This would probably be easier to troubleshoot in real-time.