Nagios Logserver Logstash service going down frequently

[anish]

Hi ,

We actually implemented Nagios Log server in our environment . we started adding the servers to the Nagios Log server for Log monitoring .But the Logstash service is frequently dying . Tried to restart the service so many times but within short span the service is getting stopped .
Did edited the following file value .

/etc/init.d/logstash

LS_HEAP_SIZE="1000m"
LS_OPEN_FILES=65535

sill the log server is not collecting the data .
Please reach us through the email : [email protected]

[tmcdonald]

Please reach us through the email : [email protected]

If you would prefer to use email, you can reach us at [email protected] to open an email-based ticket.

[anish]

Hi ,

Please support us through forum . Please ignore the email address mentioned.

[rkennedy]

Can you please post a system profile for us to review? (Admin -> System Status -> Download System Profile)

This should have a bit more information about your environment that we'll need.

[anish]

Please find the attached System profile

system-profile (4).tar.gz

[mcapra]

Can I see all the files located in the following paths:

Code: Select all

/var/log/logstash
/var/log/elasticsearch

If you could compress them into a zip or tarball, that would be ideal.

[anish]

logstash.zip

Please check the attached requested files

[mcapra]

The logs are a few weeks old, so i'm not sure how relevant they are. One thing worth mentioning is an awful lot of these:

Code: Select all

{:timestamp=>"2016-12-16T09:04:09.467000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"{\\\"EventReceivedTime\\\":\\\"2016-12-16 23:04:08\\\",\\\"SourceModuleName\\\":\\\"file1\\\",\\\"SourceModuleType\\\":\\\"im_file\\\",\\\"message\\\":\\\"2016-12-03 01:19:30 ERROR couldn't connect to tcp socket on sgkclnglsipd01.astrazeneca.net:3515; \\x90ڑ\\xB1\\x8Dς݂̌Ăяo\\x82\\xB5\\x90悪\\x88\\xEA\\x92\\xE8\\x82̎\\x9E\\x8AԂ\\xF0\\x89߂\\xAC\\x82Ă\\xE0\\x90\\xB3\\x82\\xB5\\x82\\xAD\\x89\\x9E\\x93\\x9A\\x82\\xB5\\x82Ȃ\\xA9\\x82\\xC1\\x82\\xBD\\x82\\xBD\\x82߁A\\x90ڑ\\xB1\\x82ł\\xAB\\x82܂\\xB9\\x82\\xF1\\x82ł\\xB5\\x82\\xBD\\x81B\\x82܂\\xBD\\x82͐ڑ\\xB1\\x8Dς݂̃z\\x83X\\x83g\\x82\\xAA\\x89\\x9E\\x93\\x9A\\x82\\xB5\\x82Ȃ\\xA9\\x82\\xC1\\x82\\xBD\\x82\\xBD\\x82߁A\\x8Am\\x97\\xA7\\x82\\xB3\\x82ꂽ\\x90ڑ\\xB1\\x82͎\\xB8\\x94s\\x82\\xB5\\x82܂\\xB5\\x82\\xBD\\x81B  \\\"}\\r", :expected_charset=>"UTF-8", :level=>:warn}

I suspect the issue is one of your Windows machines has different character set configured than what the Logstash input on port 3515 is expecting.

Can you share the output of the following command:

Code: Select all

grep '' /usr/local/nagioslogserver/logstash/etc/conf.d/*

As well as any nxlog configurations from your Windows machines that are sending logs? If you've got a lot of Windows machines shipping (more than is practical to grab all the configs from), I'd be interested specifically in any non-english machines.

[anish]

output file.txt

[mcapra]

I suspect one of your Windows machines is using a different charset than CP-1252. It is then hitting this input:

Code: Select all

tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }

Which will interpret the event as CP-1252 encoded and try to convert it to UTF-8. If the event received by this input is not CP-1252, Logstash gets confused. I would verify that all your Windows machines sending to port 3515 are using CP-1252 and not some other charset. Running chcp from the Windows command line will usually tell you what codepage is being leveraged.