quick question about timestamps

[benhank]

Hey fellas I have a quick question
I have 2 timestamps on the same logfile:

Code: Select all

"message": "<14>1 2019-04-10T10:58:50.146967-04

2019-04-10T14:58:50.206Z",

The top line shows the syslog message timestamp of 10:58:50 the second line shows the logstash timestamp of 14:58:50.206.
Am I correct to assume that the syslog message timestamp reflects the time on the server when the file was created and the NLS timestamp indicates when NLS received the logfile?
And furthermore IF my statement is correct, AND we take into account that both servers are both located within the same timezone then we can conclude that the sending server is in deed guilty of...(dramatice pause) INCORRECT TIMEZONE CONFIGURATION! Slaps big stack of papers down on desk, as the crowd in the courtroom erupts!
Sending Server leaps to its feet and runs towards NLS server screaming 'I'll get you for this ! I'll get you!"

[cdienger]

...(the plot thickens)...

Elasticsearch will convert timestamps and store them into UTC time.

That said, even accounting for this conversion 10:58:50.146967-04 != 14:58:50.206.

It's not clear to me where these values are getting pulled from directly, but if you see "<14>1 2019-04-10....." in the message field when viewing the event in the dashboard, then it isn't getting parsed correctly by logstashs' syslog input - a proper parsing will leave the message field with just the meat of the event and remove the syslog header info.

Looking over some valid syslog traffic and a quick review of https://www.ietf.org/rfc/rfc3164.txt, it would appear the "1" found in the syslog header is breaking a formatting requirement(https://www.elastic.co/guide/en/logstas ... yslog.html), and is the true culprit.

[benhank]

AHA! Just as I thought along! The true culprit is ME being stupid when I set up the filter!
You'll never take me alive!

sounds of breaking glass, a few seconds later the sound of a parachute deploying
hahahaahahaahh till next time!
then a thud..the proceedings took place on the second floor...

The moral of the story:
Never get a law degree from Leroys Law Lernin Skewl....


On a serious note. so I just have to set up a filter to correctly parse the timestamp data. The syslogs in question are being sent from an ubunto 14.x machine to centos, but I don't see how that would mess things up.

[cdienger]

A filter that applies to just this host(assuming the problem isn't impacting a large number) or finding out which configuration file on the client needs to be tweaked to correct the formating. If you want to provide us with the full message text, we can help with a grok filter if you want to go that direction. Otherwise I'd be curious to see the client configuration(/etc/rsyslog.conf and /etc/rsyslod.d/).

The filter would look something like:

Code: Select all

if [host] == '1.2.3.4' {
    grok {
match => { "message" => "GROKPATTERN" } //with the help of http://grokdebug.herokuapp.com
}
}

[benhank]

I had to re read the replies over and over to finally get an Idea of what you are saying.
First, I don't have any filters configured to parse syslog messages specifically.
I do have a filter that parses messages from a specific host:

Code: Select all

if [host] == '172.30.100.226' {

grok {

match => { "message" => "result=\"%{WORD:result}\" ip=\"%{IP:IP}\" action=\"%{WORD:action}\" params=\"Username: %{USER:params}\" user=\"%{USER:user}\" tenant=\"%{WORD:tenant}\""}
match => { "message" => "result=\"%{WORD:result}\" ip=\"%{IP:IP}\" action=\"%{WORD:action}\" params=\"Username=%{USER:params}\" user=\"%{USER:user}\" tenant=\"%{WORD:tenant}\""}
}
mutate {
    replace => { "Ipaddress" => "%{IP}" }
  }
geoip {
database => "/usr/share/GeoIP/GeoLiteCity.dat"
source => "Ipaddress"
}
}

looking over my syslog files I see that I have different types of logs:
Cisco asa:

Code: Select all

 	<164>Mar 18 2019 15:09:58: %ASA-4-113019: Group = DuoVPN, Username = ywalsh, IP = 64.212.90.254, Session disconnected. Session Type: SSL, Duration: 0h:00m:34s, Bytes xmt: 139896, Bytes rcv: 107255, Reason: User Requested

Centos boxes:

Code: Select all

<133>Apr 11 12:00:51 Lkennagiost02 Nagios_ACCESS: 127.0.0.1 - - [11/Apr/2019:12:00:43 -0400] "POST /nagiosxi/backend/ HTTP/1.1" 200 816 "-" "BinGet/1.00.A (http://www.bin-co.com/php/scripts/load/)"

And my ubuntu 14.x box:

Code: Select all

 	<14>1 2019-04-11T10:41:33.984187-04:00 vidyoprtl1 java - - -  VidyoPortal [audit result="SUCCESS" ip="172.22.23.124" action="Login" params="Username: candreoli" user="candreoli" tenant="Connect" timestamp="Thu Apr 11 10:41:33 EDT 2019"]

The ubuntoi machine is the one that is actually the one I am concerned about the most

[cdienger]

Sorry if I've been a bit confusing.

Going forward, I'd recommend creating a new input for this host and others like it:

Code: Select all

tcp {
    type => 'SYSLOG5424BASE2'
    port => 2059
}

and use this filter to parse these logs:

Code: Select all

if [type] == 'SYSLOG5424BASE2' {
    grok {
match => { "message" => "^%{SYSLOG5424BASE}" }
}
}

[benhank]

one question my syslogs come in on port 514, not 2059, does it make a difference?

[cdienger]

Yes, if the logs are not following rfc3164 then they'll need to use a different input(tcp) and port(any unused port. If a port is below 1024 then additional steps are needed to open it.).

[benhank]

wow this stuff is complicated lol... you guys can lock it

[cdienger]

Case closed.