Page 1 of 2
Drop messages from central syslog server
Posted: Fri Sep 23, 2016 9:49 am
by krobertson71
Linux team reported this to me today. They have enabled auditing on all linux servers which is sending a boat load of data. I am dropping a lot of it at the front door.
They are reporting the following errors on their end.
Code: Select all
Sep 23 08:37:32 logserver1 rsyslogd-2177: imuxsock lost 1003 messages from pid 23612 due to rate-limiting
Sep 23 08:42:35 logserver1 rsyslogd-2177: imuxsock begins to drop messages from pid 24186 due to rate-limiting
Sep 23 09:42:35 logserver1 rsyslogd-2177: imuxsock begins to drop messages from pid 2141 due to rate-limiting
Sep 23 09:58:16 logserver1 rsyslogd-2177: imuxsock lost 1021 messages from pid 5151 due to rate-limiting
Sep 23 10:42:35 logserver1 rsyslogd-2177: imuxsock begins to drop messages from pid 11922 due to rate-limiting
Code: Select all
Sep 23 10:41:19 syslogserver syslog-ng[9208]: Syslog connection failed; fd='408', server='AF_INET(1.1.1.1:2999)', error='Connection timed out (110)', time_reopen='10'
Re: Drop messages from central syslog server
Posted: Fri Sep 23, 2016 10:22 am
by mcapra
krobertson71 wrote:They have enabled auditing on all linux servers which is sending a boat load of data.
Do you have a rsyslog forwarder that passes the events off to Nagios Log Server?
By default, rsyslog typically drops messages if the rate is more than ~40/second. You can override this in the
/etc/rsyslog.conf file with the following directives:
Code: Select all
$SystemLogRateLimitInterval 10
$SystemLogRateLimitBurst 500
Increasing the Interval and Burst as necessary. You could also just turn off rate limiting entirely by setting the interval to 0:
Be sure to restart the rsyslog service after making changes.
Re: Drop messages from central syslog server
Posted: Fri Sep 23, 2016 10:50 am
by krobertson71
They say rate limit is set to 0. Is there something on the NLS side on how much it can process at one time?
Re: Drop messages from central syslog server
Posted: Fri Sep 23, 2016 11:02 am
by mcapra
There are a few logstash runtime variables that could be causing this, yeah. Are you able to share the /var/log/logstash/logstash.log file? If it's a size 0 file, I would check the most recent .tar.gz archive of the log.
Re: Drop messages from central syslog server
Posted: Fri Sep 23, 2016 11:15 am
by krobertson71
Code: Select all
{:timestamp=>"2016-09-22T13:31:29.305000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:31:29 ctmsssfv05 kernel: 00000000 00000000 00000000 2cbb0001 . . . . . . . . . . . . . . \\xBB ,\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:31:36.214000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:31:36 ctmsssfv06 kernel: 27000000 424d53ff 0000e324 c00298c0 . . . ' \\xFF S M B $ \\xE3 . . \\xC0 . . \\xC0\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:31:45.715000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:31:45 ctmsssfv06 kernel: 27000000 424d53ff 0000e324 c00298c0 . . . ' \\xFF S M B $ \\xE3 . . \\xC0 . . \\xC0\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:31:45.719000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:31:45 ctmsssfv06 kernel: 01a50001 00000000 . . \\xA5 . .\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:31:47.016000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:31:47 ctmsssfv06 kernel: 27000000 424d53ff 0000e324 c00298c0 . . . ' \\xFF S M B $ \\xE3 . . \\xC0 . . \\xC0\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:31:47.022000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:31:47 ctmsssfv06 kernel: 01f10001 00000000 . . \\xF1 . .\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:33:22.915000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:33:22 ctmsssfv05 kernel: 27000000 424d53ff 0000e324 c00298c0 . . . ' \\xFF S M B $ \\xE3 . . \\xC0 . . \\xC0\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T13:33:22.917000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 13:33:22 ctmsssfv05 kernel: 00000000 00000000 00000000 34bb0001 . . . . . . . . . . . . . . \\xBB 4\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T14:20:43.768000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 14:20:43 ctmsssfp01 kernel: 27000000 424d53ff 0000e324 c00298c0 . . . ' \\xFF S M B $ \\xE3 . . \\xC0 . . \\xC0\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-22T14:20:43.770000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 22 14:20:43 ctmsssfp01 kernel: 00000000 00000000 00000000 4aa00001 . . . . . . . . . . . . . . \\xA0 J\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-23T06:26:13.783000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 23 06:26:13 ctmsssfv05 kernel: 27000000 424d53ff 0000e324 c00298c0 . . . ' \\xFF S M B $ \\xE3 . . \\xC0 . . \\xC0\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-23T06:26:13.785000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 23 06:26:13 ctmsssfv05 kernel: 00000000 00000000 00000000 2cbb0001 . . . . . . . . . . . . . . \\xBB ,\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2016-09-23T06:26:13.787000-0400", :message=>"Received an event that has a different character encoding than you configured.", :text=>"<7>Sep 23 06:26:13 ctmsssfv05 kernel: 0ee90001 00000000 . . \\xE9 . .\\n", :expected_charset=>"UTF-8", :level=>:warn}
Re: Drop messages from central syslog server
Posted: Fri Sep 23, 2016 11:27 am
by mcapra
I don't think the events are being rejected by Logstash. You'd see some very specific and descriptive Java exceptions in the Logstash log if that were the case.
It could just be that rsyslog is simply overloaded rather than some configured rate limit being set.
What machine is this? 1.1.1.1:2999
Re: Drop messages from central syslog server
Posted: Fri Sep 23, 2016 11:29 am
by krobertson71
That is be redacting the ip. It was the ip of the NLS host.
Re: Drop messages from central syslog server
Posted: Fri Sep 23, 2016 11:32 am
by mcapra
Gotcha, I see where you're coming from then.
Are all of these log sources (with the "boat load" of audit data) shipping directly to Nagios Log Server, or is there a forwarder somewhere in the mix?
Re: Drop messages from central syslog server
Posted: Fri Sep 23, 2016 11:41 am
by krobertson71
All syslog data is going from other Linux nodes to a central syslog server. That server forwards all syslog event to NLS via 5599. the auditd (which is quit a bit, but nothing that log server should be complaining about) messages are coming in raw on 2999 from that same central source.
Re: Drop messages from central syslog server
Posted: Fri Sep 23, 2016 11:42 am
by krobertson71
also they are using syslog-ng not rsyslog