Logstash logs - growing too big, too fast.

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
polarbear1
Posts: 73
Joined: Mon Apr 13, 2015 4:26 pm

Logstash logs - growing too big, too fast.

Post by polarbear1 »

Greetings,

Running into a bit of trouble here with logstash being extremely chatty in /var/log/logstash/logstash.log . Over and over again I am seeing that log file get huge (sometimes to the tune of 4Gb/hour), which fills up my /var partition, and then the logstash service crashes. Deleting the offending log file to make some room and restarting the service works, but only until next time we fill up /var.

Few questions then --
  • I know how fast logstash logs grow really depends on activity so its hard to answer if 4gb/hr is too much - but what is reasonable?
  • On that note - how big should the /var parition be?
  • Is there any way to make logstash less chatty to slow the growth of the logstash.log file? We don't really look too much into that file anyway.
I'll take any other half way relevant advice here too.

I have 2 servers in a clusters, and typically they do this together - so there goes any redundancy anyway.
Last edited by polarbear1 on Mon Aug 01, 2016 9:20 am, edited 1 time in total.
Top
User avatar
eloyd
Cool Title Here
Posts: 2190
Joined: Thu Sep 27, 2012 9:14 am
Location: Rochester, NY
Contact:
Contact eloyd

Re: Logstash logs - growing too big, too fast.

Post by eloyd »

Not sure where /var/log/logstash.log comes from, but if you mean /var/log/logstash/logstash.log, then I'd say you have a serious problem. Ours is 1.4 MB for the day, so far, and all that's in it is an entry as follows, repeated over and over again, that we're too lazy to fix:

Code: Select all

:message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Aug  1 03:45:05", :exception=>java.lang.IllegalArgumentException: Invalid format: "Aug  1 03:45:05", :level=>:warn}
Image
Eric Loyd • http://everwatch.global • 844.240.EVER • @EricLoyd
I'm a Nagios Fanatic! • Join our public Nagios Discord Server!
Top
polarbear1
Posts: 73
Joined: Mon Apr 13, 2015 4:26 pm

Re: Logstash logs - growing too big, too fast.

Post by polarbear1 »

eloyd wrote:Not sure where /var/log/logstash.log comes from, but if you mean /var/log/logstash/logstash.log
Yes I did... fixed in OP. Coffee didn't kick in yet.
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Logstash logs - growing too big, too fast.

Post by rkennedy »

Can you run the following on the machine and post back?

Code: Select all

tail -n100 /var/log/logstash/logstash.log
It sounds like a reoccurring issue is causing it to become quite large. I just checked mine on a few systems, and it isn't more than a few meg. I imagine once we fix the core issue it'll return back to normal.
Former Nagios Employee
Top
polarbear1
Posts: 73
Joined: Mon Apr 13, 2015 4:26 pm

Re: Logstash logs - growing too big, too fast.

Post by polarbear1 »

I had to delete out the logstash.log to make room yesterday and as of right now the files are behaving. Let's let this thread sit for a bit - I am sure it's only a matter of days before it blows up again. Then we can continue the troubleshooting.


Also to confirm - I actually have 4 clusters (of 2 nodes each) and only one of the clusters is misbehaving. So I am sure you're right that there is some underlying nonsense.
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Logstash logs - growing too big, too fast.

Post by rkennedy »

Got it. I'll watch for a response when it comes back.

I suspect so, could be a parse failure or one specific machine sending in a log in a certain way. We'll be able to see once it re-generates.
Former Nagios Employee
Top
polarbear1
Posts: 73
Joined: Mon Apr 13, 2015 4:26 pm

Re: Logstash logs - growing too big, too fast.

Post by polarbear1 »

Like clockwork, another week - another failure. It is actually generating at an alarmingly fast rate and I am blanking it out every few hours (from 5GB).

Looking at the file from yesterday --- ehh, here's a short tail, it's more or less just more of the same for all 5 gigs of it... (spaces inserted to separate each new line from the file

Code: Select all

{:timestamp=>"2016-08-08T03:36:28.630000-0500", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2016.08.08\", :_type=>\"eventlog\", :_routing=>nil}, #<LogStash::Event:0x2b71d025 @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0x3f24523 @store={\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, @lut={\"host\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, \"host\"], \"type\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, \"type\"], \"[program]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, \"program\"], \"[SourceModuleName]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, \"SourceModuleName\"], \"message\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, \"message\"], \"timestamp\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, \"timestamp\"]}>, @data={\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.455 | rttransmissioninterfaceusage | ISO | Setting up directories.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.455\"}, @metadata_accessors=#<LogStash::Util::Accessors:0x6800cb00 @store={\"retry_count\"=>0}, @lut={}>, @cancelled=false>]", :level=>:warn}

{:timestamp=>"2016-08-08T03:36:28.631000-0500", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2016.08.08\", :_type=>\"eventlog\", :_routing=>nil}, #<LogStash::Event:0x73ece740 @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0x2d29d59c @store={\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, @lut={\"host\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, \"host\"], \"type\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, \"type\"], \"[program]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, \"program\"], \"[SourceModuleName]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, \"SourceModuleName\"], \"message\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, \"message\"], \"timestamp\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, \"timestamp\"]}>, @data={\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_rttransmissioninterfaceusage.log\", \"message\"=>\"2016/08/08 03:36:26.471 | rttransmissioninterfaceusage | Getting Files.\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.471\"}, @metadata_accessors=#<LogStash::Util::Accessors:0x49ec7571 @store={\"retry_count\"=>0}, @lut={}>, @cancelled=false>]", :level=>:warn}

{:timestamp=>"2016-08-08T03:36:28.632000-0500", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2016.08.08\", :_type=>\"eventlog\", :_routing=>nil}, #<LogStash::Event:0x380abf46 @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0x7d571094 @store={\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, @lut={\"host\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, \"host\"], \"type\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, \"type\"], \"[program]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, \"program\"], \"[SourceModuleName]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, \"SourceModuleName\"], \"message\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, \"message\"], \"timestamp\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, \"timestamp\"]}>, @data={\"EventReceivedTime\"=>\"2016-08-08 03:36:27\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_daintertieconstraintshadowprices.log\", \"message\"=>\"2016/08/08 03:36:26.923 | daintertieconstraintshadowprices | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\daintertieconstraintshadowprices\\\\Process\\\\CAISO_DAINTERT_20160809.zip with size of: 651.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:27.824Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:26.923\"}, @metadata_accessors=#<LogStash::Util::Accessors:0x13752371 @store={\"retry_count\"=>0}, @lut={}>, @cancelled=false>]", :level=>:warn}

{:timestamp=>"2016-08-08T03:36:29.531000-0500", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2016.08.08\", :_type=>\"eventlog\", :_routing=>nil}, #<LogStash::Event:0x4dc59195 @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0x6c2ab97 @store={\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, @lut={\"host\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, \"host\"], \"type\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, \"type\"], \"[program]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, \"program\"], \"[SourceModuleName]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, \"SourceModuleName\"], \"message\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, \"message\"], \"timestamp\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, \"timestamp\"]}>, @data={\"EventReceivedTime\"=>\"2016-08-08 03:36:28\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_haspnomogram.log\", \"message\"=>\"2016/08/08 03:36:28.281 | haspnomogram | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\haspnomogram\\\\Process\\\\CAISO_HASPNOMO_20160805.zip with size of: 984.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:28.841Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.281\"}, @metadata_accessors=#<LogStash::Util::Accessors:0x216044df @store={\"retry_count\"=>0}, @lut={}>, @cancelled=false>]", :level=>:warn}

{:timestamp=>"2016-08-08T03:36:30.487000-0500", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2016.08.08\", :_type=>\"eventlog\", :_routing=>nil}, #<LogStash::Event:0x5c396c6c @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0x430911b2 @store={\"EventReceivedTime\"=>\"2016-08-08 03:36:29\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_netclearedconvergencebiddingawards.log\", \"message\"=>\"2016/08/08 03:36:28.796 | netclearedconvergencebiddingawards | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\netclearedconvergencebiddingawards\\\\Process\\\\CAISO_CBIDAWARDS_20160804.zip with size of: 91,967.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:29.896Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.796\"}, @lut={\"host\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:29\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_netclearedconvergencebiddingawards.log\", \"message\"=>\"2016/08/08 03:36:28.796 | netclearedconvergencebiddingawards | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\netclearedconvergencebiddingawards\\\\Process\\\\CAISO_CBIDAWARDS_20160804.zip with size of: 91,967.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:29.896Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.796\"}, \"host\"], \"type\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:29\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_netclearedconvergencebiddingawards.log\", \"message\"=>\"2016/08/08 03:36:28.796 | netclearedconvergencebiddingawards | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\netclearedconvergencebiddingawards\\\\Process\\\\CAISO_CBIDAWARDS_20160804.zip with size of: 91,967.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:29.896Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.796\"}, \"type\"], \"[program]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:29\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_netclearedconvergencebiddingawards.log\", \"message\"=>\"2016/08/08 03:36:28.796 | netclearedconvergencebiddingawards | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\netclearedconvergencebiddingawards\\\\Process\\\\CAISO_CBIDAWARDS_20160804.zip with size of: 91,967.0 kb\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-08-08T08:36:29.896Z\", \"host\"=>\"192.168.1.110\", \"type\"=>\"eventlog\", \"timestamp\"=>\"16/08/08 03:36:28.796\"}, \"program\"], \"[SourceModuleName]\"=>[{\"EventReceivedTime\"=>\"2016-08-08 03:36:29\", \"SourceModuleName\"=>\"iso\", \"SourceModuleType\"=>\"im_file\", \"Hostname\"=>\"SCHPISO1\", \"Program\"=>\"CAISOexecutable_CAISO_netclearedconvergencebiddingawards.log\", \"message\"=>\"2016/08/08 03:36:28.796 | netclearedconvergencebiddingawards | File Downloaded: D:\\\\dataservices\\\\dataprograms\\\\CAISO\\\\Application\\\\Data\\\\netclearedconvergencebiddingawards\\\\Process\\\\CAISO_CBIDAWARDS_20160804.zip with size of: 91,967.0 kb\"

Doing a quick google on the code 400 - looks like many reasons. Many threads I see have something to do with crappy filters. So would it possibly be some crappy grok filters? I do have a few on this cluster that I don't on my other clusters (which are behaving).


Also -- my versions:

Code: Select all

Nagios Log Server	1.4.1
Elasticsearch	1.6.0
Logstash	1.5.1
Kibana	3.1.1-nagios3
Top
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:
Contact hsmith

Re: Logstash logs - growing too big, too fast.

Post by hsmith »

Can we see a screenshot of your global configuration page?
Former Nagios Employee.
me.
Top
polarbear1
Posts: 73
Joined: Mon Apr 13, 2015 4:26 pm

Re: Logstash logs - growing too big, too fast.

Post by polarbear1 »

Yes you can. Attached. I only expected the configs for the non-default groks I added. Everything else is default.

As you can probably tell from teh context - the "ISO Parser" config is the only relevant one to the 400 Errors I posted above. The middletier config is not applicable for those specific errors, but I am not saying it's not a contributing factor.
You do not have the required permissions to view the files attached to this post.
Top
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:
Contact hsmith

Re: Logstash logs - growing too big, too fast.

Post by hsmith »

What kind of device are these logs coming from? Sometimes if the timestamp is not in a format that your logserver is expecting, the logs will be dropped due to the syslog input.

To modify the syslog input, or create a new one for your syslogs, take a look at this post.
Former Nagios Employee.
me.
Top
Locked

Return to “Nagios Log Server”