Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Logstash logs - growing too big, too fast.

https://support.nagios.com/forum/viewtopic.php?t=39598

Page 1 of 2

Logstash logs - growing too big, too fast.

Posted: Mon Aug 01, 2016 9:01 am
by polarbear1
Greetings,

Running into a bit of trouble here with logstash being extremely chatty in /var/log/logstash/logstash.log . Over and over again I am seeing that log file get huge (sometimes to the tune of 4Gb/hour), which fills up my /var partition, and then the logstash service crashes. Deleting the offending log file to make some room and restarting the service works, but only until next time we fill up /var.

Few questions then --
  • I know how fast logstash logs grow really depends on activity so its hard to answer if 4gb/hr is too much - but what is reasonable?
  • On that note - how big should the /var parition be?
  • Is there any way to make logstash less chatty to slow the growth of the logstash.log file? We don't really look too much into that file anyway.
I'll take any other half way relevant advice here too.

I have 2 servers in a clusters, and typically they do this together - so there goes any redundancy anyway.

Re: Logstash logs - growing too big, too fast.

Posted: Mon Aug 01, 2016 9:14 am
by eloyd
Not sure where /var/log/logstash.log comes from, but if you mean /var/log/logstash/logstash.log, then I'd say you have a serious problem. Ours is 1.4 MB for the day, so far, and all that's in it is an entry as follows, repeated over and over again, that we're too lazy to fix:

Code: Select all

:message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Aug  1 03:45:05", :exception=>java.lang.IllegalArgumentException: Invalid format: "Aug  1 03:45:05", :level=>:warn}

Re: Logstash logs - growing too big, too fast.

Posted: Mon Aug 01, 2016 9:20 am
by polarbear1
eloyd wrote:Not sure where /var/log/logstash.log comes from, but if you mean /var/log/logstash/logstash.log
Yes I did... fixed in OP. Coffee didn't kick in yet.

Re: Logstash logs - growing too big, too fast.

Posted: Mon Aug 01, 2016 9:42 am
by rkennedy
Can you run the following on the machine and post back?

Code: Select all

tail -n100 /var/log/logstash/logstash.log
It sounds like a reoccurring issue is causing it to become quite large. I just checked mine on a few systems, and it isn't more than a few meg. I imagine once we fix the core issue it'll return back to normal.

Re: Logstash logs - growing too big, too fast.

Posted: Mon Aug 01, 2016 9:49 am
by polarbear1
I had to delete out the logstash.log to make room yesterday and as of right now the files are behaving. Let's let this thread sit for a bit - I am sure it's only a matter of days before it blows up again. Then we can continue the troubleshooting.


Also to confirm - I actually have 4 clusters (of 2 nodes each) and only one of the clusters is misbehaving. So I am sure you're right that there is some underlying nonsense.

Re: Logstash logs - growing too big, too fast.

Posted: Mon Aug 01, 2016 10:25 am
by rkennedy
Got it. I'll watch for a response when it comes back.

I suspect so, could be a parse failure or one specific machine sending in a log in a certain way. We'll be able to see once it re-generates.

Re: Logstash logs - growing too big, too fast.

Posted: Tue Aug 09, 2016 10:44 am
by polarbear1
Like clockwork, another week - another failure. It is actually generating at an alarmingly fast rate and I am blanking it out every few hours (from 5GB).

Looking at the file from yesterday --- ehh, here's a short tail, it's more or less just more of the same for all 5 gigs of it... (spaces inserted to separate each new line from the file

Code: Select all

{:timestamp=>"2016-08-08T03:36:28.630000-0500", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2016.08.08\", :_type=>\"eventlog\", :_routing=>nil}, #<LogStash::Event:0x2b71d025 @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0x3f24523 @store={\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, @lut={\"host\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, \"host\"], \"type\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, \"type\"], \"[program]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, \"program\"], \"[SourceModuleName]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, \"SourceModuleName\"], \"message\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, \"message\"], \"timestamp\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, \"timestamp\"]}>, @data={\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, @metadata_accessors=#<LogStash::Util::Accessors:0x6800cb00 @store={\"retry_count\"=>0}, @lut={}>, @cancelled=false>]", :level=>:warn}

{:timestamp=>"2016-08-08T03:36:28.631000-0500", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2016.08.08\", :_type=>\"eventlog\", :_routing=>nil}, #<LogStash::Event:0x73ece740 @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0x2d29d59c @store={\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, @lut={\"host\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, \"host\"], \"type\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, \"type\"], \"[program]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, \"program\"], \"[SourceModuleName]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, \"SourceModuleName\"], \"message\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, \"message\"], \"timestamp\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, \"timestamp\"]}>, @data={\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, @metadata_accessors=#<LogStash::Util::Accessors:0x49ec7571 @store={\"retry_count\"=>0}, @lut={}>, @cancelled=false>]", :level=>:warn}

{:timestamp=>"2016-08-08T03:36:28.632000-0500", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2016.08.08\", :_type=>\"eventlog\", :_routing=>nil}, #<LogStash::Event:0x380abf46 @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0x7d571094 @store={\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, @lut={\"host\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, \"host\"], \"type\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, \"type\"], \"[program]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, \"program\"], \"[SourceModuleName]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, \"SourceModuleName\"], \"message\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, \"message\"], \"timestamp\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, \"timestamp\"]}>, @data={\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, @metadata_accessors=#<LogStash::Util::Accessors:0x13752371 @store={\"retry_count\"=>0}, @lut={}>, @cancelled=false>]", :level=>:warn}

{:timestamp=>"2016-08-08T03:36:29.531000-0500", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2016.08.08\", :_type=>\"eventlog\", :_routing=>nil}, #<LogStash::Event:0x4dc59195 @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0x6c2ab97 @store={\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, @lut={\"host\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, \"host\"], \"type\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, \"type\"], \"[program]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, \"program\"], \"[SourceModuleName]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, \"SourceModuleName\"], \"message\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, \"message\"], \"timestamp\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, \"timestamp\"]}>, @data={\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, @metadata_accessors=#<LogStash::Util::Accessors:0x216044df @store={\"retry_count\"=>0}, @lut={}>, @cancelled=false>]", :level=>:warn}

{:timestamp=>"2016-08-08T03:36:30.487000-0500", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2016.08.08\", :_type=>\"eventlog\", :_routing=>nil}, #<LogStash::Event:0x5c396c6c @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0x430911b2 @store={\"EventReceivedTime\"=>\"2016-08-08 03:36:29\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_netclearedconvergencebiddingawards.log\", \"message\"=>\"2016/08/08 03:36:28.796 | netclearedconvergencebiddingawards | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\netclearedconvergencebiddingawards\\\\Process\\\\CAISO_CBIDAWARDS_20160804.zip with size of: 91,967.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:29.896Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.796\"}, @lut={\"host\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:29\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_netclearedconvergencebiddingawards.log\", \"message\"=>\"2016/08/08 03:36:28.796 | netclearedconvergencebiddingawards | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\netclearedconvergencebiddingawards\\\\Process\\\\CAISO_CBIDAWARDS_20160804.zip with size of: 91,967.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:29.896Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.796\"}, \"host\"], \"type\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:29\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_netclearedconvergencebiddingawards.log\", \"message\"=>\"2016/08/08 03:36:28.796 | netclearedconvergencebiddingawards | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\netclearedconvergencebiddingawards\\\\Process\\\\CAISO_CBIDAWARDS_20160804.zip with size of: 91,967.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:29.896Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.796\"}, \"type\"], \"[program]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:29\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_netclearedconvergencebiddingawards.log\", \"message\"=>\"2016/08/08 03:36:28.796 | netclearedconvergencebiddingawards | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\netclearedconvergencebiddingawards\\\\Process\\\\CAISO_CBIDAWARDS_20160804.zip with size of: 91,967.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:29.896Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.796\"}, \"program\"], \"[SourceModuleName]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:29\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_netclearedconvergencebiddingawards.log\", \"message\"=>\"2016/08/08 03:36:28.796 | netclearedconvergencebiddingawards | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\netclearedconvergencebiddingawards\\\\Process\\\\CAISO_CBIDAWARDS_20160804.zip with size of: 91,967.0 kb\"

Doing a quick google on the code 400 - looks like many reasons. Many threads I see have something to do with crappy filters. So would it possibly be some crappy grok filters? I do have a few on this cluster that I don't on my other clusters (which are behaving).


Also -- my versions:

Code: Select all

Nagios Log Server	1.4.1
Elasticsearch	1.6.0
Logstash	1.5.1
Kibana	3.1.1-nagios3

Re: Logstash logs - growing too big, too fast.

Posted: Tue Aug 09, 2016 10:47 am
by hsmith
Can we see a screenshot of your global configuration page?

Re: Logstash logs - growing too big, too fast.

Posted: Tue Aug 09, 2016 2:23 pm
by polarbear1
Yes you can. Attached. I only expected the configs for the non-default groks I added. Everything else is default.

As you can probably tell from teh context - the "ISO Parser" config is the only relevant one to the 400 Errors I posted above. The middletier config is not applicable for those specific errors, but I am not saying it's not a contributing factor.

Re: Logstash logs - growing too big, too fast.

Posted: Tue Aug 09, 2016 4:29 pm
by hsmith
What kind of device are these logs coming from? Sometimes if the timestamp is not in a format that your logserver is expecting, the logs will be dropped due to the syslog input.

To modify the syslog input, or create a new one for your syslogs, take a look at this post.
All times are UTC-05:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited