Using NLS to log dynamic docker hosts

[roddergreg]

Hello,

We are running containers inside a docker host. When a new docker host is created a different hostid hash is created. We are trying to build and elastic type apache environment. Depending on CPU Load an application called marathon will fire up another apache container to help with the load.

The question I have is: Is there a way to configure the rsyslog daemon on each docker to use a dyamic name when sending the logs?

So we have 1 Docker Host. In the host we are running about 4 apache containers with rsyslog running.

Inside the container we are writing apache logs to /var/log/httpd/$HOSTNAME/access. I want to ship these logs with a dynamic hostname to our NLS. Currently it appears that we would have to setup rsyslog on each container to ship them to the Log Server. This sounds like an administrative nightmare. Especially if the marathon process decides to start up another instance.

Is there any way to setup rsyslog to use dynamic names?

Thank you in advance,

Greg Cooper

[eloyd]

Greg,

I may be missing what you're asking exactly, but rsyslog does not send information with a hostname. It is receiving side that identifies the IP of the sender and optionally assigns a hostname. Since rsyslog is a file-level monitoring service, then yes, if you want to send /var/log/httpd/<dynamic thing>/access to NLS, you would have to set up rsyslog to send each <dynamic thing> log file, as you say.

rsyslogd does support wildcards for filenames (http://www.rsyslog.com/doc/master/confi ... mfile.html) but not for directories. Which, for me, means if you can change /var/log/httpd/$HOSTNAME/access to /var/log/httpd/$HOSTNAME.access, then you could monitor /var/log/httpd/* via rsyslog and get all your container access logs via the wildcard processing.

[hsmith]

Thanks, Eric!

@ roddergreg, does this answer your question?

[prhunixadmin]

Yes this answers my question but we've decided to go another route.

I do have another questions that involves ENV variables.

Since we are running docker containers I'd like to include the MARATHON_APP_ID variable as part of the tag for the syslog messages being shipped to nagioslogserver.

Is there any way to pass an environment variable that rsyslog could read as part of the tag?

[eloyd]

Environment variables are not part of the tag. At least, not dynamically assigned ones. You'd have to reconfigure the rsyslog sender tag each time the variable was set/changed and restart rsyslog.

A potentially better solution is to teach NLS that data from certain hosts, sender types, or matched data (or a combination of things) belongs to a certain tag, and tag it in the input filter.

[rkennedy]

Thanks @eloyd! As mentioned, your best bet is going to be tagging an additional field at the input level.

Let us know if you have any further questions.