Parsing 12 hour format timestamps

[CFT6Server]

Seeking some guidance here on how to parse and create a grok pattern/filter for AD DNS debugging logs. The time format is in a 12hour time format and I am not sure not to convert this for proper timestamp due to the AM/PM.

Sample line.

9/24/2015 11:23:52 AM 0F1C PACKET 00000000031A1170 UDP Rcv 172.26.250.44 69d4 Q [0001 D NOERROR] PTR (2)22(3)128(2)75(2)10(7)in-addr(4)arpa(0)

I need to get the time 11:23:52 AM converted to something that can be used as timestamp.

[jolson]

The date filter is capable of recognizing 12-hour timestamps.

https://www.elastic.co/guide/en/logstas ... -date.html

You can match the date filter against any format supplied by the JODA library, which includes 12-hour timestamps. You might try something like this:

Code: Select all

date {
match => [ "datefield", "M/DD/YYYY hh.mm.ss a" ]
}

Give the above a shot and let me know if it works!

[CFT6Server]

This is assuming I can grab the whole date into a field?
I am looking how to capture 9/24/2015 11:23:52 AM into a field with grok. I can grab everything except for the AM|PM. I must be missing something minor.

Grok pattern I am trying to use:
%{DATE_US} %{TIME} (?:AM|PM)

[lgroschen]

Looks good in the debugger:

Code: Select all

{
  "DATE_US": [
    [
      "9/24/2015"
    ]
  ],
  "MONTHNUM": [
    [
      "9"
    ]
  ],
  "MONTHDAY": [
    [
      "24"
    ]
  ],
  "YEAR": [
    [
      "2015"
    ]
  ],
  "TIME": [
    [
      "11:23:52"
    ]
  ],
  "HOUR": [
    [
      "11"
    ]
  ],
  "MINUTE": [
    [
      "23"
    ]
  ],
  "SECOND": [
    [
      "52"
    ]
  ]
}

[CFT6Server]

what will be the field name that it is parsing? Time? But time doesn't include AM/PM so how will it read the field? In the Elasticsearch example assumes there's a field logdate, but how would put this all in one field for it to add? or that's not how it works?

[jolson]

The easiest way that I can think of approaching this problem is as follows.

1. Get your timestamp into a field called '12_time'

2. Convert the 12_time field into a datestamp.

3. Wipe the '12_time' field appropriately.

The filter:

Code: Select all

filter {
 grok {
 match => [ "message", "%{DATE} %{GREEDYDATA:12_time},%{GREEDYDATA:otherdata}" ]

 }
 date {
 match => [ "12_time", "hh:mm:ss a" ]
 target => "12_time"

 timezone => "+00:00"
 }
 grok {
 match => { "12_time" => "%{GREEDYDATA} %{GREEDYDATA:time} " }
 overwrite => [ "12_time" ]
 }
}

Let me know if the above works for you. Thanks!

[CFT6Server]

So this is what I've done instead. There's no punctuation so it makes it a bit harder to grab the time fields. Here's the grok pattern:

Message:

Code: Select all

9/24/2015 11:23:52 AM 0F1C PACKET 00000000031A1170 UDP Rcv 172.26.250.44 69d4 Q [0001 D NOERROR] PTR (2)22(3)128(2)75(2)10(7)in-addr(4)arpa(0)

Grok Pattern:

Code: Select all

grok {
        match => [ 'message', '%{DATE_US:date} %{TIME:time1} %{WORD:time2} %{WORD:dns_thread_id} %{WORD:dns_context}%{SPACE}%{WORD:dns_packet_id} %{WORD:protocol} %{WORD:dns_direction} %{IP:src_IP}%{SPACE}%{WORD:dns_xid}%{SPACE}(?:Q|R Q|R U|U)%{SPACE}\[%{WORD:dns_hex_flags}%{SPACE}%{WORD:dns_hex_flags}%{SPACE}%{GREEDYDATA:dns_response}\]%{SPACE}%{WORD:dns_recordtype}%{SPACE}%{GREEDYDATA:dns_query_name}']
    }

Now take time1/2 to form the 12h format along with the date

Code: Select all

mutate {
	    add_field => { "12time" => "%{date};%{time1} %{time2}" }
		}
	date {
		match => [ "12time", "MM/dd/YYYY;hh:mm:ss a" ]
    }	
	mutate {
		remove_field => [ "date","time1",time2" ]
		}

Without the date, it cannot replace the timestamp field and will error and cause logstash to just dump the log entries.

here's the result of this:

date.JPG

The message was 10/6/2015 9:50:34 AM 0C54 PACKET 000000001DD19210 UDP Snd ..............

[jolson]

Well done! This looks proper to me - it looks like your timestamps differ, but that's because the @timestamp field can be displayed in UTC (which is looks like it is, in your case). Do you have any further questions?