NLS 2.1.7 - filter timestamp in microseconds
Posted: Fri Aug 07, 2020 12:45 am
My input:
28500 2020-06-08T19:31:50.739496 lab-vm2 f0bcc1e028f2[23452]: InstanceInfo: bos.tcp-proxy.Local.0
My filter:
if [program] == "import_raw" {
grok {
match => [ 'message', '^%{NUMBER:ser} %{NOTSPACE:timestamp} *%{NOTSPACE:node} *%{GREEDYDATA:log_message}' ]
}
mutate {
replace => [ 'type', 'test_log' ]
}
}
And ingest using:
python shipper.py -f '/tmp/test.logs' program:import_raw | nc 127.0.0.1 2057
This works, however I would like to parse the timestamp ("2020-06-08T19:31:50.739496" in the example above), including the microseconds. I read somewhere that I could convert to Unix epoch time and just use %{NUMBER:timestamp} and the conversion would happen. I tried doing that - didn't get recognized as a timestamp. I was unable to find ms/us precision in the grok docs. I can convert this to any format prior to ingesting. Is this even possible? Thanks in advance
28500 2020-06-08T19:31:50.739496 lab-vm2 f0bcc1e028f2[23452]: InstanceInfo: bos.tcp-proxy.Local.0
My filter:
if [program] == "import_raw" {
grok {
match => [ 'message', '^%{NUMBER:ser} %{NOTSPACE:timestamp} *%{NOTSPACE:node} *%{GREEDYDATA:log_message}' ]
}
mutate {
replace => [ 'type', 'test_log' ]
}
}
And ingest using:
python shipper.py -f '/tmp/test.logs' program:import_raw | nc 127.0.0.1 2057
This works, however I would like to parse the timestamp ("2020-06-08T19:31:50.739496" in the example above), including the microseconds. I read somewhere that I could convert to Unix epoch time and just use %{NUMBER:timestamp} and the conversion would happen. I tried doing that - didn't get recognized as a timestamp. I was unable to find ms/us precision in the grok docs. I can convert this to any format prior to ingesting. Is this even possible? Thanks in advance