Drop messages from central syslog server

[krobertson71]

Linux team reported this to me today. They have enabled auditing on all linux servers which is sending a boat load of data. I am dropping a lot of it at the front door.

They are reporting the following errors on their end.

Code: Select all

Sep 23 08:37:32 logserver1 rsyslogd-2177: imuxsock lost 1003 messages from pid 23612 due to rate-limiting
Sep 23 08:42:35 logserver1 rsyslogd-2177: imuxsock begins to drop messages from pid 24186 due to rate-limiting
Sep 23 09:42:35 logserver1 rsyslogd-2177: imuxsock begins to drop messages from pid 2141 due to rate-limiting
Sep 23 09:58:16 logserver1 rsyslogd-2177: imuxsock lost 1021 messages from pid 5151 due to rate-limiting
Sep 23 10:42:35 logserver1 rsyslogd-2177: imuxsock begins to drop messages from pid 11922 due to rate-limiting

Code: Select all

Sep 23 10:41:19 syslogserver syslog-ng[9208]: Syslog connection failed; fd='408', server='AF_INET(1.1.1.1:2999)', error='Connection timed out (110)', time_reopen='10'

[mcapra]

They have enabled auditing on all linux servers which is sending a boat load of data.

Do you have a rsyslog forwarder that passes the events off to Nagios Log Server?

By default, rsyslog typically drops messages if the rate is more than ~40/second. You can override this in the /etc/rsyslog.conf file with the following directives:

Code: Select all

$SystemLogRateLimitInterval 10
$SystemLogRateLimitBurst 500

Increasing the Interval and Burst as necessary. You could also just turn off rate limiting entirely by setting the interval to 0:

Code: Select all

$SystemLogRateLimitInterval 0

Be sure to restart the rsyslog service after making changes.

[krobertson71]

They say rate limit is set to 0. Is there something on the NLS side on how much it can process at one time?

[mcapra]

There are a few logstash runtime variables that could be causing this, yeah. Are you able to share the /var/log/logstash/logstash.log file? If it's a size 0 file, I would check the most recent .tar.gz archive of the log.

[krobertson71]

Code: Select all

{:timestamp=>"2016-09-22T13:31:29.305000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:31:29 ctmsssfv05 kernel: 00000000 00000000 00000000 2cbb0001 . . . . . . . . . . . . . . \\xBB ,\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:31:36.214000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:31:36 ctmsssfv06 kernel: 27000000 424d53ff 0000e324 c00298c0 . . . ' \\xFF S M B $ \\xE3 . . \\xC0 . . \\xC0\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:31:45.715000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:31:45 ctmsssfv06 kernel: 27000000 424d53ff 0000e324 c00298c0 . . . ' \\xFF S M B $ \\xE3 . . \\xC0 . . \\xC0\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:31:45.719000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:31:45 ctmsssfv06 kernel: 01a50001 00000000 . . \\xA5 . .\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:31:47.016000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:31:47 ctmsssfv06 kernel: 27000000 424d53ff 0000e324 c00298c0 . . . ' \\xFF S M B $ \\xE3 . . \\xC0 . . \\xC0\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:31:47.022000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:31:47 ctmsssfv06 kernel: 01f10001 00000000 . . \\xF1 . .\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:33:22.915000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:33:22 ctmsssfv05 kernel: 27000000 424d53ff 0000e324 c00298c0 . . . ' \\xFF S M B $ \\xE3 . . \\xC0 . . \\xC0\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:33:22.917000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:33:22 ctmsssfv05 kernel: 00000000 00000000 00000000 34bb0001 . . . . . . . . . . . . . . \\xBB 4\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T14:20:43.768000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 14:20:43 ctmsssfp01 kernel: 27000000 424d53ff 0000e324 c00298c0 . . . ' \\xFF S M B $ \\xE3 . . \\xC0 . . \\xC0\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T14:20:43.770000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 14:20:43 ctmsssfp01 kernel: 00000000 00000000 00000000 4aa00001 . . . . . . . . . . . . . . \\xA0 J\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-23T06:26:13.783000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 23 06:26:13 ctmsssfv05 kernel: 27000000 424d53ff 0000e324 c00298c0 . . . ' \\xFF S M B $ \\xE3 . . \\xC0 . . \\xC0\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-23T06:26:13.785000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 23 06:26:13 ctmsssfv05 kernel: 00000000 00000000 00000000 2cbb0001 . . . . . . . . . . . . . . \\xBB ,\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-23T06:26:13.787000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 23 06:26:13 ctmsssfv05 kernel: 0ee90001 00000000 . . \\xE9 . .\\n", :expected_charset=>"UTF-8", :level=>:warn}

[mcapra]

I don't think the events are being rejected by Logstash. You'd see some very specific and descriptive Java exceptions in the Logstash log if that were the case.

It could just be that rsyslog is simply overloaded rather than some configured rate limit being set.

What machine is this? 1.1.1.1:2999

[krobertson71]

That is be redacting the ip. It was the ip of the NLS host.

[mcapra]

Gotcha, I see where you're coming from then.

Are all of these log sources (with the "boat load" of audit data) shipping directly to Nagios Log Server, or is there a forwarder somewhere in the mix?

[krobertson71]

All syslog data is going from other Linux nodes to a central syslog server. That server forwards all syslog event to NLS via 5599. the auditd (which is quit a bit, but nothing that log server should be complaining about) messages are coming in raw on 2999 from that same central source.

[krobertson71]

also they are using syslog-ng not rsyslog