Forwarding address defined in one place

[gormank]

I have thousands of config files in /etc/rsyslog.d across hundreds of RHEL hosts in several environments. All are managed by various puppet systems in the environments. All use similar setups and logs are forwarded with rsyslog. The destination addresses are defined in each file.

That said, I'd like to set up rsyslog forwarding in a single file rather than in each file. For example in a 001_destination.addresses.conf, which would theoretically be the first file read out of the .d dir.

Here are examples of my not too bright .conf files. The first is forwarding all in rsyslog.conf, and the second is one of many files forwarded.

Code: Select all

$WorkDirectory /var/lib/rsyslog                              # Where spool files will live
$ActionQueueFileName nlsFwdRule0                             # Unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g                                  # 1GB space limit (use as much as possible)
$ActionQueueSaveOnShutdown on                                # Save messages to disk on shutdown
$ActionQueueType LinkedList                                  # Use asynchronous processing
$ActionResumeRetryCount -1                                   # Infinite retries if host is down

*.* @@[2001:4888:a05:3154:e0:ff2:0:b01]:5544                 # *.*: send everything in rsyslog.conf, @@: use TCP
$ActionExecOnlyWhenPreviousIsSuspended on                    # If the 1st log server is down send to the next, etc.
& @@[2001:4888:a05:3154:e0:ff2:0:b02]:5544
& @@[2001:4888:a05:3154:e0:ff2:0:b03]:5544
$ActionExecOnlyWhenPreviousIsSuspended off

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog
$InputFileName /var/log/mcs/tomcat01/tc_wi_task.out
$InputFileTag tc_wi_task.out
$InputFileStateFile nls-state-tc_wi_task.out
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
if $programname == 'tc_wi_task.out' then @@[2001:4888:a05:3154:e0:ff2:0:b01]:5544
$ActionExecOnlyWhenPreviousIsSuspended on
& @@[2001:4888:a05:3154:e0:ff2:0:b02]:5544
& @@[2001:4888:a05:3154:e0:ff2:0:b03]:5544
$ActionExecOnlyWhenPreviousIsSuspended off
if $programname == 'tc_wi_task.out' then stop

[cdienger]

Using your examples as a template, I was able to condense a couple of configs and the rsyslog.conf into a single file:

Code: Select all

$WorkDirectory /var/lib/rsyslog                              # Where spool files will live
$ActionQueueFileName nlsFwdRule0                             # Unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g                                  # 1GB space limit (use as much as possible)
$ActionQueueSaveOnShutdown on                                # Save messages to disk on shutdown
$ActionQueueType LinkedList                                  # Use asynchronous processing
$ActionResumeRetryCount -1                                   # Infinite retries if host is down

*.* @@[192.168.3.42]:5544                 # *.*: send everything in rsyslog.conf, @@: use TCP
$ActionExecOnlyWhenPreviousIsSuspended on                    # If the 1st log server is down send to the next, etc.
& @@[192.168.3.44]:5544
& @@[192.168.3.45]:5544
$ActionExecOnlyWhenPreviousIsSuspended off


$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

$InputFileName /var/log/test1.out
$InputFileTag test1.out
$InputFileStateFile nls-state-test1.out
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
if $programname == 'test1.out' then @@[192.168.3.42]:5544
$ActionExecOnlyWhenPreviousIsSuspended on
& @@[192.168.3.43]:5544
& @@[192.168.3.44]:5544
$ActionExecOnlyWhenPreviousIsSuspended off
if $programname == 'test1.out' then stop

$InputFileName /var/log/test2.out
$InputFileTag test2.out
$InputFileStateFile nls-state-test2.out
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
if $programname == 'test2.out' then @@[192.168.3.42]:5544
$ActionExecOnlyWhenPreviousIsSuspended on
& @@[192.168.3.42]:5544
& @@[192.168.3.42]:5544
$ActionExecOnlyWhenPreviousIsSuspended off
if $programname == 'test2.out' then stop

Hope this helps! Let me know if you have any further questions.

[gormank]

I don't think things are clear.
I want to separate the destination addresses from the individual conf files and put them in a single file. Your example shows the addresses defined twice, and my config even before being distributed on one environment has 100 conf files.

[cdienger]

Ah. I see the goal but don't see a way to accomplish it unfortunately.

[mcapra]

Apologies if I'm misunderstanding things; I come from the worlds of Chef and Ansible.

Is there a particular reason you're not having Puppet be responsible for generating a single configuration file for each machine in a dynamic way? Within my org we've had great success with letting Chef worry about distilling down environments into a single configuration set for a given machine with respect to our agents/loggers.

[gormank]

Puppet variables actually become constants once defined. Thus w/ ~100 log files, puppet is going to be very hard to use. Different host roles log to various files and paths, etc.

[cdienger]

Unfortunately this is a bit beyond the scope of NLS and what we typically deal with when setting up hosts to forward data to NLS. Perhaps the documentation or maintainers of the specific version of rsyslog can be of better help.

[gormank]

I guess I'll put strings to be replaced in the individual files and replace with an exec sed in each. I was hoping to make things even more generic by using a single template .conf file but that seems to be a bridge too far.
Feel free to close this.

[tmcdonald]

Yea, unfortunately sometimes we are beholden to the lower levels of the software stack, and this is one of those times.