Nagios Support Forum

Posted: **Thu Sep 22, 2016 9:06 am**

Send over these two pretty please:

-rw-r--r--. 1 nagios users   691 Sep 10 06:07 043f21b0-1d53-4881-8f3e-9c58be028b8d.log-20160910.gz
-rw-r--r--. 1 nagios users 26834 Sep 21 18:00 1c6d88c6-e9ca-480d-8c85-8bd97ef41adf.log

Posted: **Thu Sep 22, 2016 9:13 am**

Here ya go....and FYI - No crash since yesterday afternoon. Should I continue with the special running logstash?

043f21b0-1d53-4881-8f3e-9c58be028b8d.log-20160910.gz

1c6d88c6-e9ca-480d-8c85-8bd97ef41adf.log-20160921.gz

Posted: **Thu Sep 22, 2016 11:24 am**

The old adage of "if it ain't broke, don't fix it" seems appropriate, but the init.d script we use for logstash does some special stuff behind the scenes to optimize memory usage on the machine that may become relevant if the node ever reaches capacity.

Also if you intend to run logstash this way for a long time, I would remove the debug output.

Posted: **Thu Sep 22, 2016 11:28 am**

Just connected to the screen session to kill it and saw this. Looks like every 4 hours I get a list of these and then nothing for 4 more hours

Code: Select all

Sep 22, 2016 12:00:11 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [9453] handler org.elasticsearch.action.TransportActionNodeProxy$1@16102657, error [true], resetting
Sep 22, 2016 12:00:11 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [9478] handler org.elasticsearch.action.TransportActionNodeProxy$1@1e23f4f4, error [true], resetting
Sep 22, 2016 12:00:11 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [9479] handler org.elasticsearch.action.TransportActionNodeProxy$1@291d8a98, error [true], resetting
Sep 22, 2016 12:00:11 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [9386] handler org.elasticsearch.action.TransportActionNodeProxy$1@3a605c15, error [true], resetting
Sep 22, 2016 12:00:12 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [9455] handler org.elasticsearch.action.TransportActionNodeProxy$1@fbf52fa, error [true], resetting
Sep 22, 2016 12:00:12 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [9480] handler org.elasticsearch.action.TransportActionNodeProxy$1@3595a991, error [true], resetting
Sep 22, 2016 2:00:05 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11496] handler org.elasticsearch.action.TransportActionNodeProxy$1@4041cd8c, error [true], resetting
Sep 22, 2016 2:00:06 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11539] handler org.elasticsearch.action.TransportActionNodeProxy$1@68a695fc, error [true], resetting
Sep 22, 2016 2:00:06 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11497] handler org.elasticsearch.action.TransportActionNodeProxy$1@58f10c9d, error [true], resetting
Sep 22, 2016 2:00:07 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11540] handler org.elasticsearch.action.TransportActionNodeProxy$1@204aeadd, error [true], resetting
Sep 22, 2016 2:00:07 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11498] handler org.elasticsearch.action.TransportActionNodeProxy$1@4ef0f21d, error [true], resetting
Sep 22, 2016 2:00:08 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11542] handler org.elasticsearch.action.TransportActionNodeProxy$1@66b68b9e, error [true], resetting
Sep 22, 2016 2:00:08 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11500] handler org.elasticsearch.action.TransportActionNodeProxy$1@7898bae1, error [true], resetting
Sep 22, 2016 2:00:09 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11543] handler org.elasticsearch.action.TransportActionNodeProxy$1@478d5c82, error [true], resetting
Sep 22, 2016 2:00:09 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11501] handler org.elasticsearch.action.TransportActionNodeProxy$1@4f0d8122, error [true], resetting
Sep 22, 2016 2:00:10 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11544] handler org.elasticsearch.action.TransportActionNodeProxy$1@33ab9953, error [true], resetting
Sep 22, 2016 2:00:10 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11502] handler org.elasticsearch.action.TransportActionNodeProxy$1@2dba6d22, error [true], resetting
Sep 22, 2016 2:00:11 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11545] handler org.elasticsearch.action.TransportActionNodeProxy$1@38710954, error [true], resetting
Sep 22, 2016 2:00:11 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11503] handler org.elasticsearch.action.TransportActionNodeProxy$1@25606a84, error [true], resetting
Sep 22, 2016 2:00:12 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11546] handler org.elasticsearch.action.TransportActionNodeProxy$1@d4a5c99, error [true], resetting
Sep 22, 2016 2:00:12 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11505] handler org.elasticsearch.action.TransportActionNodeProxy$1@7e6b20b5, error [true], resetting
Sep 22, 2016 2:00:13 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11548] handler org.elasticsearch.action.TransportActionNodeProxy$1@616d6ee5, error [true], resetting
Sep 22, 2016 2:00:13 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11506] handler org.elasticsearch.action.TransportActionNodeProxy$1@fc321a2, error [true], resetting
Sep 22, 2016 2:00:14 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [11549] handler org.elasticsearch.action.TransportActionNodeProxy$1@709dd657, error [true], resetting
Sep 22, 2016 6:00:13 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [15756] handler org.elasticsearch.action.TransportActionNodeProxy$1@49357bd5, error [true], resetting
Sep 22, 2016 6:00:14 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [15757] handler org.elasticsearch.action.TransportActionNodeProxy$1@75c115a4, error [true], resetting
Sep 22, 2016 6:00:15 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [15758] handler org.elasticsearch.action.TransportActionNodeProxy$1@a841d4c, error [true], resetting
Sep 22, 2016 10:00:12 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19690] handler org.elasticsearch.action.TransportActionNodeProxy$1@5bfc8b2, error [true], resetting
Sep 22, 2016 10:00:13 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19691] handler org.elasticsearch.action.TransportActionNodeProxy$1@4136a820, error [true], resetting
Sep 22, 2016 10:00:14 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19693] handler org.elasticsearch.action.TransportActionNodeProxy$1@2ae403ca, error [true], resetting
Sep 22, 2016 10:00:15 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19694] handler org.elasticsearch.action.TransportActionNodeProxy$1@6ee13f89, error [true], resetting
Sep 22, 2016 10:00:16 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19695] handler org.elasticsearch.action.TransportActionNodeProxy$1@3335752, error [true], resetting
Sep 22, 2016 10:00:17 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19696] handler org.elasticsearch.action.TransportActionNodeProxy$1@364e8f9c, error [true], resetting
Sep 22, 2016 10:00:18 AM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [19697] handler org.elasticsearch.action.TransportActionNodeProxy$1@4248577e, error [true], resetting

Posted: **Thu Sep 22, 2016 11:33 am**

I would spin up a 3rd node if possible and add it to the cluster. See if it encounters the same problems.

We can move this to email and schedule a remote if you like as well. This would probably be easier to troubleshoot in real-time.

Nagios Support Forum

New install Logstash failing

Re: New install Logstash failing

Re: New install Logstash failing

Re: New install Logstash failing

Re: New install Logstash failing

Re: New install Logstash failing