Page 1 of 1
Multiple time zones in one NLS
Posted: Tue Sep 06, 2016 11:00 am
by eloyd
We're just starting to notice something that's been with us from day one, and the solution needs to be something other than "change /etc/localtime" Let me 'splain:
We have multiple Linux boxes mostly in Eastern time. We also have Linux boxes in Center, Mountain, and Pacific time but we've got their time zones set to Eastern because it's easier that way. Now, we added a new box in Pacific time, but we need that box to have a Pacific timezone. All of these boxes are sending syslog data to a NLS box in Eastern time.
Our NLS box, however, is showing the events from our new Pacific box as showing up three hours behind, which makes "show me the past hour's worth of events across all boxes" awkward.
Did we miss a memo somewhere? How can we do this?
Re: Multiple time zones in one NLS
Posted: Tue Sep 06, 2016 1:29 pm
by mcapra
All I had to do was add a template to my messages that included %timegenerated%.
In
/etc/rsyslog.d/99-nagioslogserver.conf:
Code: Select all
# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional NAGIOSLOGSERVER
$template add_time,"[%timegenerated%] %msg%"
*.* @@192.168.67.200:5545;add_time
And logstash was able to pick up on it just fine without additional filtering.
Remote system:
Code: Select all
root@ubuntu:/etc/rsyslog.d# date
Tue Sep 6 11:30:50 PDT 2016
NLS system:
Code: Select all
[root@localhost init.d]# date
Tue Sep 6 13:31:33 CDT 2016
Before remote system template:
2016_09_06_13_33_17_Dashboard_Nagios_Log_Server.png
After remote system template:
2016_09_06_13_33_44_Dashboard_Nagios_Log_Server.png
Re: Multiple time zones in one NLS
Posted: Tue Sep 06, 2016 1:47 pm
by eloyd
Changing the message sent is not practical. Ideally, I want to teach NLS that logsource X has a time offset of Y compared to localtime on the NLS server. Assuming that I can't do that, the only solution we've come up with is to convert all timestamps to UTC based on host, but this is ugly and stupid. I realize it's logstash under the hood, but was wondering if anyone else had come across this kind of usage case.
Re: Multiple time zones in one NLS
Posted: Tue Sep 06, 2016 2:10 pm
by mcapra
It comes up frequently enough for sure. It's a limitation with rsyslog (< v8) as much as it is a limitation with (our outdated version of) logstash. rsyslog 8+ has some conventions for properly handling the timezone, but most recent OS versions are distributing 7.4 via apt and yum.
You can do clever things with date filters to set the timezone, but I don't think you can have a date filter match the @timestamp itself. You would need to parse the time into a separate field then run a date filter on it setting the timezone appropriately.
Re: Multiple time zones in one NLS
Posted: Tue Sep 06, 2016 2:20 pm
by eloyd
@timestamp is inherent and easily searchable with "now:now-1hour" so we'll just proceed with changing timezones on the remote host rsyslog feeds. Thanks. You can close.