Timestamp manipulation

[SteveBeauchemin]

My IIS logs in NLS are using the log scrape "date" and the @timestamp is not using the date and time from the log line. It is using the collection time. It is not using the true event time.

The date field is like this 2019-07-08
The time field is like this 17:03:43

I am trying to use this filter

Code: Select all

  mutate {
    add_field => ["ts", "%{date} %{time}"]
  }
  date {
    match => ["ts", "yyyy-MM-dd HH:mm:ss"]
    target => "@timestamp"
  }
  mutate {
    remove_field => ["ts", "date", "time", "EventReceivedTime"]
  }

When I do this, the IIS data just stops coming in.
If I remove the date {} section it shows up again.

Is there a preferred syntax for date?

Please advise.

Thanks

Steve B

[SteveBeauchemin]

Strange but true...

If I mutate / convert the date to a 'string' the @timestamp works properly.

I took 2 fields and combined them to create a pseudo time stamp. To convert it to the real @timestamp it needs to be a string first.

So this fixed it.

Code: Select all

  mutate {
    add_field => ["ts", "%{date} %{time}"]
  }
  mutate {
    convert => { "ts" => "string" }
  }
  date {
    match => ["ts", "yyyy-MM-dd HH:mm:ss"]
    target => "@timestamp"
  }

This can be closed.

Thanks

Steve B

[cdienger]

What does the input configuration look like? Are there any other filters for this data? Behavior like this is usually due to a parsing issue, but the config you provided looks good.

You can enable debug logging by editing /etc/init.d/logstash and changing line 64 from:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

to:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS} --debug"

and then restart the service:

Code: Select all

systemctl daemon-reload
service logstash restart

Let it run this way just long enough to allow events to come in and hit this filter, then disable debugging, and gather the /var/logstash/logstash.log.

[SteveBeauchemin]

So, my timestamp problem is okay now. Once I turned it into a string.

But. I am having some trouble getting ruby to work in a filter.
I am not so sure that the new config is getting read properly after an update is committed.

I say that because I have a line that removes 4 unused fields. Not in ruby. The previous version of config removed 3 of them. I added to the same line a 4th. Yet the 3 are removed but the 4th is not.

So, I'm still trying to get a good understanding of how it all works. If I update a config, it updates on all 4 servers. And then I see the effect in the GUI sometimes. Not always. As in this case where I am expecting 4 fields to be removed but only 3 are gone.

So. You just now provided a way to debug. I will start using that tomorrow and see if I can get this squared away.

I am also logging in to only one specific server of the 4 to do this work.

Thanks

Steve B

[cdienger]

Sounds good. Keep us posted.

[SteveBeauchemin]

We should close this. I'll start a new thread for my other items. The timestamps are good now.

Thanks

Steve B

[scottwilkerson]

We should close this. I'll start a new thread for my other items. The timestamps are good now.

Thanks

Steve B

Great!

Locking