logstash warning Failed Parsing Date Invalid format, again

[GhostRider2110]

Nagios Log Server Cluster:
Have added System Profiles from both systems as attachments.

iganagioslog - CentOS release 6.10 (Final)
This is the first system setup when we started using NLS, I believe it was a VMware image from Nagios.
NLS 2.1.3

iganagioslog01 - Red Hat Enterprise Linux Server release 7.7 (Maipo)
Install from downloaded tar file.

I've had this problem before, but seems to have cropped up again. I have searched and can't find what has changed on the systems being logged.

In logstash.log from iganagioslog:
{:timestamp=>"2020-01-21T11:08:19.843000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"21/Jan/2020:11:08:18 -0500", :exception=>"Invalid format: \"21/Jan/2020:11:08:18 -0500\"", :config_parsers=>"MMM dd HH:mm:ss", :config_locale=>"en", :level=>:warn}
{:timestamp=>"2020-01-21T11:08:19.844000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"21/Jan/2020:11:08:19 -0500", :exception=>"Invalid format: \"21/Jan/2020:11:08:19 -0500\"", :config_parsers=>"MMM dd HH:mm:ss", :config_locale=>"en", :level=>:warn}
{:timestamp=>"2020-01-21T11:08:19.845000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"21/Jan/2020:11:08:19 -0500", :exception=>"Invalid format: \"21/Jan/2020:11:08:19 -0500\"", :config_parsers=>"MMM dd HH:mm:ss", :config_locale=>"en", :level=>:warn}
{:timestamp=>"2020-01-21T11:08:19.846000-0500", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"21/Jan/2020:11:08:19 -0500", :exception=>"Invalid format: \"21/Jan/2020:11:08:19 -0500\"", :config_parsers=>"MMM dd HH:mm:ss", :config_locale=>"en", :level=>:warn}

Not seeing them in the logstash.log on iganagiosls01

Many moons ago you helped me get a similar problem fixed, by adding custom syslog and apache log input filters

iganagiosls01-config-syslog-apache-001.png

I have found the entries and see where they are coming from, just not how to fix it...

The other strange thing, right now I am concentrating on the 3 webcache servers, igapubwebcache01/02/03. Configured the same, yet I can't get any apache_access logs or apache_error logs to show up in a search for igapubwebcache01. I get syslog and sudo logs, but neither of the apache logs. In addition, I'm only seen the data parse failure in only one of the logstash files. The one on iganagioslog. I have a couple more screen shots, but can only attach 3.

Thanks
Mitch

[GhostRider2110]

Couple more screen shots:

Nagiosls01-error-string-search01.png

Nagiosls01-dateparsefailure-02.png

Thanks
Mitch

[GhostRider2110]

Found these entries in
/var/log/elasticsearch/25e0abdc-5b56-4815-adcb-4239555d0899.log

[2020-01-21 10:17:56,259][WARN ][indices.breaker ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [FIELDDATA] New used memory 7658589285 [7.1gb] from field [message.raw] would be larger than configured breaker: 7566183628 [7gb], breaking
[2020-01-21 10:17:56,618][WARN ][indices.breaker ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [FIELDDATA] New used memory 7586146942 [7gb] from field [@timestamp] would be larger than configured breaker: 7566183628 [7gb], breaking
[2020-01-21 10:17:56,632][WARN ][indices.breaker ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [FIELDDATA] New used memory 7586161669 [7gb] from field [@timestamp] would be larger than configured breaker: 7566183628 [7gb], breaking
[2020-01-21 10:17:56,747][WARN ][indices.breaker ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [FIELDDATA] New used memory 7586463908 [7gb] from field [@timestamp] would be larger than configured breaker: 7566183628 [7gb], breaking
[2020-01-21 10:17:56,765][WARN ][indices.breaker ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [FIELDDATA] New used memory 7586483647 [7gb] from field [@timestamp] would be larger than configured breaker: 7566183628 [7gb], breaking
[2020-01-21 10:17:58,618][WARN ][indices.breaker ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [FIELDDATA] New used memory 7588960497 [7gb] from field [message.raw] would be larger than configured breaker: 7566183628 [7gb], breaking
[2020-01-21 10:17:58,650][WARN ][indices.breaker ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [FIELDDATA] New used memory 7588957427 [7gb] from field [message.raw] would be larger than configured breaker: 7566183628 [7gb], breaking
[2020-01-21 10:17:58,776][WARN ][indices.breaker ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [FIELDDATA] New used memory 7588944746 [7gb] from field [message.raw] would be larger than configured breaker: 7566183628 [7gb], breaking
[2020-01-21 10:17:59,559][WARN ][indices.breaker ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [FIELDDATA] New used memory 7588951930 [7gb] from field [message.raw] would be larger than configured breaker: 7566183628 [7gb], breaking

[cdienger]

These all could be related to the inability to parse the timestamp. The issue is that it is currently configured to find a timestamp in the format of "MMM dd HH:mm:ss" but something is sending the date over in the format of "dd/MMM/yyyy:HH:mm:ss Z'. To account for this variation you can change the date filter from:

Code: Select all

        date {
	   locale => "en"
            match => [ 'timestamp', 'MMM dd HH:mm:ss' ]
        }

to:

Code: Select all

 date {
	   locale => "en"
            match => [ 'timestamp', 'MMM dd HH:mm:ss', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }

[GhostRider2110]

That fixed that. Still getting some other errors,

Will open another thread for that one. Thanks

Mitch

[scottwilkerson]

That fixed that. Still getting some other errors,

Will open another thread for that one. Thanks

Mitch

Great!

Locking thread