Nagios Support Forum

Hi
i am evaluating NLS for syslog inputs (syslogs from network devices like cisco FW)
i have managed to use to get syslogs using high ports, displayed in kibana webui

now i am concerned to store syslogs also in flat file format before any change done by logstash

i was thinking of using rsyslog (which is already running on NLS) to be the first input point from network device, then rsyslog will both output to a local flat file on disk (with log file naming and logrotate, ...) and also send it to logstash for "classic" log parsing/filtering, and storing in elastic

[networkdevice]----(syslog-protocol)----[rsyslog]----[flat-file-on-disk]+[logstash]----[elastic]----[kibana]

How can we achieve that ?
Can we just change rsyslog config without the risk that NLS config management will overwrite the rsyslog config file ?
Is there any risk with compatibility in future updates from NLS ?
Any recommendation to achieve that ?

Thanks

While you could probably do this on the NLS server, we couldn't guarantee the changes wouldn't get overwritten ever and it wouldn't be supported. Instead I think your best bet would bet setting up a syslog proxy/relay:

log source ---> syslog relay ---> NLS
|
local copy

How you forward and store a local copy will depend on what you use for a relay, but generally speaking rsyslog can take specify multiple outputs:

*.* @NLS:5544
*.* /var/log/messages

You may also need to play around with the settings on the syslog relay/NLS server to get them to play nicely - the relay may tack on it's own syslog info which could cause NLS to parse IPs, timestamps, etc... incorrectly.

Thank you, this is a good question and answer.

What if,

(say, maybe you have 25 small remote locations, and want to limit traffic from them to HQ DC where Log Server is, and wanted all of the 'original' logs stored at HQ DC too)

...you could do two things at once at the Cluster:

Remote Sources -----------> Cluster ---> Logstash -----> *Filter/parse* ----> Elasticsearch .
............................................................. ........||
...................................................... .................||
.......................................................... *do not parse* forward unmodified (Output?)-----> some other store

Is this possible? I guess this would increase load on Log Server, and I'm not sure if it is. Filter AND skip the Filter, then Output.

Anyway, sorry to barge in, but I thought this question was along parallel.

The syslog output option may provide you with what you're looking for:

https://www.elastic.co/guide/en/logstas ... yslog.html

Thank you cdienger, that syslog Output looks promising in the context I outlined.

Thanks for all inputs
I will try to use the file output and perhaps also the syslog output of logstash
I also understand the "front" syslog relay
I was looking for a more reliable syslog input than logstash and also something that won't stop storing logs when changing grok filters, or when there is an issue with config or service
(Yesterday syslog input and storage stops after 2 days of good work, even after service logstash rrestart, it doesn't came back. I had to reboot the vm ... that's why i would have prefered to store syslog with rsyslog in a file (we are sure to store logs even if parsing is causing issues) and logstash read the file, but all on the same server

Any recommandation if i want to use the local rsyslog daemon ?

Thanks

Essentially the rsyslog forwarder could live on the same box as Nagios Log Server, write messages to files, then Logstash could have a file input responsible for consuming those files. rsyslog on it's own sort of already works this way; When it can't contact the remote syslog server (Nagios Log Server / Logstash), it spools messages up on disk to be shipped later on. The difference in running a local rsyslog forwarder (on the Nagios Log Server machine) would be that you're spooling to the Nagios Log Server machine's disk rather than the original source machine's disk.
http://www.rsyslog.com/storing-and-forw ... -messages/

Some people put a Redis instance in front of Logstash, then have Logstash read from Redis:
https://www.elastic.co/guide/en/logstas ... redis.html

You could do the same with RabbitMQ as well, where Logstash would act as a consumer:
https://www.elastic.co/guide/en/logstas ... bitmq.html

In both cases, Redis/RabbitMQ acts as a broker for Logstash and can collect events without issues should Logstash die. You would obviously still need to respond to the Logstash failure relatively quickly, but adding a broker can give you more time to respond.

That's all a bit beyond a vanilla Nagios Log Server setup, though.

Thanks for the input, mcapra!