Page 1 of 1
Parsing 12 hour format timestamps
Posted: Thu Sep 24, 2015 1:25 pm
by CFT6Server
Seeking some guidance here on how to parse and create a grok pattern/filter for AD DNS debugging logs. The time format is in a 12hour time format and I am not sure not to convert this for proper timestamp due to the AM/PM.
Sample line.
9/24/2015 11:23:52 AM 0F1C PACKET 00000000031A1170 UDP Rcv 172.26.250.44 69d4 Q [0001 D NOERROR] PTR (2)22(3)128(2)75(2)10(7)in-addr(4)arpa(0)
I need to get the time 11:23:52 AM converted to something that can be used as timestamp.
Re: Parsing 12 hour format timestamps
Posted: Thu Sep 24, 2015 4:23 pm
by jolson
The date filter is capable of recognizing 12-hour timestamps.
https://www.elastic.co/guide/en/logstas ... -date.html
You can match the date filter against any format supplied by the
JODA library, which includes 12-hour timestamps. You might try something like this:
Code: Select all
date {
match => [ "datefield", "M/DD/YYYY hh.mm.ss a" ]
}
Give the above a shot and let me know if it works!
Re: Parsing 12 hour format timestamps
Posted: Tue Sep 29, 2015 3:10 pm
by CFT6Server
This is assuming I can grab the whole date into a field?
I am looking how to capture 9/24/2015 11:23:52 AM into a field with grok. I can grab everything except for the AM|PM. I must be missing something minor.
Grok pattern I am trying to use:
%{DATE_US} %{TIME} (?:AM|PM)
Re: Parsing 12 hour format timestamps
Posted: Wed Sep 30, 2015 11:52 am
by lgroschen
Looks good in the debugger:
Code: Select all
{
"DATE_US": [
[
"9/24/2015"
]
],
"MONTHNUM": [
[
"9"
]
],
"MONTHDAY": [
[
"24"
]
],
"YEAR": [
[
"2015"
]
],
"TIME": [
[
"11:23:52"
]
],
"HOUR": [
[
"11"
]
],
"MINUTE": [
[
"23"
]
],
"SECOND": [
[
"52"
]
]
}
Re: Parsing 12 hour format timestamps
Posted: Fri Oct 02, 2015 3:53 pm
by CFT6Server
what will be the field name that it is parsing? Time? But time doesn't include AM/PM so how will it read the field? In the Elasticsearch example assumes there's a field logdate, but how would put this all in one field for it to add? or that's not how it works?
Re: Parsing 12 hour format timestamps
Posted: Mon Oct 05, 2015 10:13 am
by jolson
The easiest way that I can think of approaching this problem is as follows.
1. Get your timestamp into a field called '12_time'
2. Convert the 12_time field into a datestamp.
3. Wipe the '12_time' field appropriately.
The filter:
Code: Select all
filter {
grok {
match => [ "message", "%{DATE} %{GREEDYDATA:12_time},%{GREEDYDATA:otherdata}" ]
}
date {
match => [ "12_time", "hh:mm:ss a" ]
target => "12_time"
timezone => "+00:00"
}
grok {
match => { "12_time" => "%{GREEDYDATA} %{GREEDYDATA:time} " }
overwrite => [ "12_time" ]
}
}
Let me know if the above works for you. Thanks!
Re: Parsing 12 hour format timestamps
Posted: Tue Oct 06, 2015 11:52 am
by CFT6Server
So this is what I've done instead. There's no punctuation so it makes it a bit harder to grab the time fields. Here's the grok pattern:
Message:
Code: Select all
9/24/2015 11:23:52 AM 0F1C PACKET 00000000031A1170 UDP Rcv 172.26.250.44 69d4 Q [0001 D NOERROR] PTR (2)22(3)128(2)75(2)10(7)in-addr(4)arpa(0)
Grok Pattern:
Code: Select all
grok {
match => [ 'message', '%{DATE_US:date} %{TIME:time1} %{WORD:time2} %{WORD:dns_thread_id} %{WORD:dns_context}%{SPACE}%{WORD:dns_packet_id} %{WORD:protocol} %{WORD:dns_direction} %{IP:src_IP}%{SPACE}%{WORD:dns_xid}%{SPACE}(?:Q|R Q|R U|U)%{SPACE}\[%{WORD:dns_hex_flags}%{SPACE}%{WORD:dns_hex_flags}%{SPACE}%{GREEDYDATA:dns_response}\]%{SPACE}%{WORD:dns_recordtype}%{SPACE}%{GREEDYDATA:dns_query_name}']
}
Now take time1/2 to form the 12h format along with the date
Code: Select all
mutate {
add_field => { "12time" => "%{date};%{time1} %{time2}" }
}
date {
match => [ "12time", "MM/dd/YYYY;hh:mm:ss a" ]
}
mutate {
remove_field => [ "date","time1",time2" ]
}
Without the date, it cannot replace the timestamp field and will error and cause logstash to just dump the log entries.
here's the result of this:
date.JPG
The message was 10/6/2015 9:50:34 AM 0C54 PACKET 000000001DD19210 UDP Snd ..............
Re: Parsing 12 hour format timestamps
Posted: Tue Oct 06, 2015 1:44 pm
by jolson
Well done! This looks proper to me - it looks like your timestamps differ, but that's because the @timestamp field can be displayed in UTC (which is looks like it is, in your case). Do you have any further questions?