Adding a date to time-only log entries?

[gsl_ops_practice]

Hello,

I have a log file where the developers chose not to put the current date, only the current time at the beginning of each line.

Code: Select all

00:00:00.215

Is there some sort of an input filter I can create that will put the current date in front so I can then import this log properly?

Thank you,
Alex

[cdienger]

@timestamp should automatically be filed with the time that the event came into NLS which may not match up with when the actual event occurred. Not ideal, but is this effectively what you're trying to do with a filter? Are the logs not importing at all?

How are the logs being sent over? Ideally you'd want to make this change on the client side - either making sure the logs include the timestamp or the client sending the data timestamps it.

[gsl_ops_practice]

Hi,

The issue is when using the %{TIMESTAMP_ISO8601:logdate} filter to pull out the date (so that the events display properly over time), the parsing fails and all subsequent filters fail as well.

Your assistance in achieving this would be appreciated.

Thank you,
Alex

[cdienger]

Can you attach a copy of the input, filter, and a sample of the log(sanitized if needed)? Also, are the logs coming over via syslog or nxlog? It'd be good to gather either the rsyslog.conf or nxlog.conf file as well.

[gsl_ops_practice]

Hello,

I was trying to do this in rsyslog, it would be the simplest solution so far but rsyslog seems to be cryptic and buggy. In the end I resorted to an ugly hack of running a screen session with tail output with awk to create a new file, but this is not a long term solution as it dies every day at 0000 when the logs rotate. Enough for a proof of concept but nowhere near production-grade.

Code: Select all

tail -f /mnt/prod1/transaction.log | awk '{ print strftime("%F"), $0; fflush() }' > /home/nagios/transaction.log

effectively turning

Code: Select all

00:00:00.759

into

Code: Select all

2017-05-10 00:00:00.759

[cdienger]

Yeah, that does not seem ideal. Does your rsyslog config specify a format? This is found in the CentOS7 default rsyslog.conf:

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

[gsl_ops_practice]

I've been through the rsyslog5 and rsyslog7 config and man pages for this on Centos6, didn't really get anywhere because it seems it is cryptic and ill behaved, which is unfortunate.

I would prefer to do this with an input filter in NagiosLog, can you please advise exactly what is needed for this?

Thanks,
Alex

[cdienger]

You can insert the timestamp into the message with:

mutate {
    replace => { "message" => "%{@timestamp}: %{message}" }
}