Page 1 of 1
Timestamp manipulation
Posted: Mon Jul 08, 2019 3:50 pm
by SteveBeauchemin
My IIS logs in NLS are using the log scrape "date" and the @timestamp is not using the date and time from the log line. It is using the collection time. It is not using the true event time.
The date field is like this 2019-07-08
The time field is like this 17:03:43
I am trying to use this filter
Code: Select all
mutate {
add_field => ["ts", "%{date} %{time}"]
}
date {
match => ["ts", "yyyy-MM-dd HH:mm:ss"]
target => "@timestamp"
}
mutate {
remove_field => ["ts", "date", "time", "EventReceivedTime"]
}
When I do this, the IIS data just stops coming in.
If I remove the date {} section it shows up again.
Is there a preferred syntax for date?
Please advise.
Thanks
Steve B
Re: Timestamp manipulation
Posted: Tue Jul 09, 2019 9:44 am
by SteveBeauchemin
Strange but true...
If I mutate / convert the date to a 'string' the @timestamp works properly.
I took 2 fields and combined them to create a pseudo time stamp. To convert it to the real @timestamp it needs to be a string first.
So this fixed it.
Code: Select all
mutate {
add_field => ["ts", "%{date} %{time}"]
}
mutate {
convert => { "ts" => "string" }
}
date {
match => ["ts", "yyyy-MM-dd HH:mm:ss"]
target => "@timestamp"
}
This can be closed.
Thanks
Steve B
Re: Timestamp manipulation
Posted: Tue Jul 09, 2019 10:02 am
by cdienger
What does the input configuration look like? Are there any other filters for this data? Behavior like this is usually due to a parsing issue, but the config you provided looks good.
You can enable debug logging by editing /etc/init.d/logstash and changing line 64 from:
Code: Select all
DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"
to:
Code: Select all
DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS} --debug"
and then restart the service:
Code: Select all
systemctl daemon-reload
service logstash restart
Let it run this way just long enough to allow events to come in and hit this filter, then disable debugging, and gather the /var/logstash/logstash.log.
Re: Timestamp manipulation
Posted: Tue Jul 09, 2019 12:19 pm
by SteveBeauchemin
So, my timestamp problem is okay now. Once I turned it into a string.
But. I am having some trouble getting ruby to work in a filter.
I am not so sure that the new config is getting read properly after an update is committed.
I say that because I have a line that removes 4 unused fields. Not in ruby. The previous version of config removed 3 of them. I added to the same line a 4th. Yet the 3 are removed but the 4th is not.
So, I'm still trying to get a good understanding of how it all works. If I update a config, it updates on all 4 servers. And then I see the effect in the GUI sometimes. Not always. As in this case where I am expecting 4 fields to be removed but only 3 are gone.
So. You just now provided a way to debug. I will start using that tomorrow and see if I can get this squared away.
I am also logging in to only one specific server of the 4 to do this work.
Thanks
Steve B
Re: Timestamp manipulation
Posted: Tue Jul 09, 2019 4:21 pm
by cdienger
Sounds good. Keep us posted.
Re: Timestamp manipulation
Posted: Thu Jul 11, 2019 10:41 am
by SteveBeauchemin
We should close this. I'll start a new thread for my other items. The timestamps are good now.
Thanks
Steve B
Re: Timestamp manipulation
Posted: Thu Jul 11, 2019 10:59 am
by scottwilkerson
SteveBeauchemin wrote:We should close this. I'll start a new thread for my other items. The timestamps are good now.
Thanks
Steve B
Great!
Locking