Multiple time zones in one NLS

[eloyd]

We're just starting to notice something that's been with us from day one, and the solution needs to be something other than "change /etc/localtime" Let me 'splain:

We have multiple Linux boxes mostly in Eastern time. We also have Linux boxes in Center, Mountain, and Pacific time but we've got their time zones set to Eastern because it's easier that way. Now, we added a new box in Pacific time, but we need that box to have a Pacific timezone. All of these boxes are sending syslog data to a NLS box in Eastern time.

Our NLS box, however, is showing the events from our new Pacific box as showing up three hours behind, which makes "show me the past hour's worth of events across all boxes" awkward.

Did we miss a memo somewhere? How can we do this?

[mcapra]

All I had to do was add a template to my messages that included %timegenerated%.


In /etc/rsyslog.d/99-nagioslogserver.conf:

Code: Select all

# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional       NAGIOSLOGSERVER
$template add_time,"[%timegenerated%] %msg%"
*.* @@192.168.67.200:5545;add_time                                               

And logstash was able to pick up on it just fine without additional filtering.

Remote system:

Code: Select all

root@ubuntu:/etc/rsyslog.d# date
Tue Sep  6 11:30:50 PDT 2016

NLS system:

Code: Select all

[root@localhost init.d]# date
Tue Sep  6 13:31:33 CDT 2016

Before remote system template:

2016_09_06_13_33_17_Dashboard_Nagios_Log_Server.png

After remote system template:

2016_09_06_13_33_44_Dashboard_Nagios_Log_Server.png

[eloyd]

Changing the message sent is not practical. Ideally, I want to teach NLS that logsource X has a time offset of Y compared to localtime on the NLS server. Assuming that I can't do that, the only solution we've come up with is to convert all timestamps to UTC based on host, but this is ugly and stupid. I realize it's logstash under the hood, but was wondering if anyone else had come across this kind of usage case.

[mcapra]

It comes up frequently enough for sure. It's a limitation with rsyslog (< v8) as much as it is a limitation with (our outdated version of) logstash. rsyslog 8+ has some conventions for properly handling the timezone, but most recent OS versions are distributing 7.4 via apt and yum.

You can do clever things with date filters to set the timezone, but I don't think you can have a date filter match the @timestamp itself. You would need to parse the time into a separate field then run a date filter on it setting the timezone appropriately.

[eloyd]

@timestamp is inherent and easily searchable with "now:now-1hour" so we'll just proceed with changing timezones on the remote host rsyslog feeds. Thanks. You can close.