Page 1 of 2
Nagios Logserver Logstash service going down frequently
Posted: Thu Jan 05, 2017 9:42 am
by anish
Hi ,
We actually implemented Nagios Log server in our environment . we started adding the servers to the Nagios Log server for Log monitoring .But the Logstash service is frequently dying . Tried to restart the service so many times but within short span the service is getting stopped .
Did edited the following file value .
/etc/init.d/logstash
LS_HEAP_SIZE="1000m"
LS_OPEN_FILES=65535
sill the log server is not collecting the data .
Please reach us through the email :
[email protected]
Re: Nagios Logserver Logstash service going down frequently
Posted: Thu Jan 05, 2017 10:17 am
by tmcdonald
If you would prefer to use email, you can reach us at
[email protected] to open an email-based ticket.
Re: Nagios Logserver Logstash service going down frequently
Posted: Thu Jan 05, 2017 10:49 am
by anish
Hi ,
Please support us through forum . Please ignore the email address mentioned.
Re: Nagios Logserver Logstash service going down frequently
Posted: Thu Jan 05, 2017 10:52 am
by rkennedy
Can you please post a system profile for us to review? (Admin -> System Status -> Download System Profile)
This should have a bit more information about your environment that we'll need.
Re: Nagios Logserver Logstash service going down frequently
Posted: Thu Jan 05, 2017 11:47 am
by anish
Please find the attached System profile
system-profile (4).tar.gz
Re: Nagios Logserver Logstash service going down frequently
Posted: Thu Jan 05, 2017 5:52 pm
by mcapra
Can I see all the files located in the following paths:
Code: Select all
/var/log/logstash
/var/log/elasticsearch
If you could compress them into a zip or tarball, that would be ideal.
Re: Nagios Logserver Logstash service going down frequently
Posted: Fri Jan 06, 2017 6:11 am
by anish
logstash.zip
Please check the attached requested files
Re: Nagios Logserver Logstash service going down frequently
Posted: Fri Jan 06, 2017 12:00 pm
by mcapra
The logs are a few weeks old, so i'm not sure how relevant they are. One thing worth mentioning is an awful lot of these:
Code: Select all
{:timestamp=>"2016-12-16T09:04:09.467000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"{\\\"EventReceivedTime\\\":\\\"2016-12-16 23:04:08\\\",\\\"SourceModuleName\\\":\\\"file1\\\",\\\"SourceModuleType\\\":\\\"im_file\\\",\\\"message\\\":\\\"2016-12-03 01:19:30 ERROR couldn't connect to tcp socket on sgkclnglsipd01.astrazeneca.net:3515; \\x90ڑ\\xB1\\x8Dς݂̌Ăяo\\x82\\xB5\\x90悪\\x88\\xEA\\x92\\xE8\\x82̎\\x9E\\x8AԂ\\xF0\\x89߂\\xAC\\x82Ă\\xE0\\x90\\xB3\\x82\\xB5\\x82\\xAD\\x89\\x9E\\x93\\x9A\\x82\\xB5\\x82Ȃ\\xA9\\x82\\xC1\\x82\\xBD\\x82\\xBD\\x82߁A\\x90ڑ\\xB1\\x82ł\\xAB\\x82܂\\xB9\\x82\\xF1\\x82ł\\xB5\\x82\\xBD\\x81B\\x82܂\\xBD\\x82͐ڑ\\xB1\\x8Dς݂̃z\\x83X\\x83g\\x82\\xAA\\x89\\x9E\\x93\\x9A\\x82\\xB5\\x82Ȃ\\xA9\\x82\\xC1\\x82\\xBD\\x82\\xBD\\x82߁A\\x8Am\\x97\\xA7\\x82\\xB3\\x82ꂽ\\x90ڑ\\xB1\\x82͎\\xB8\\x94s\\x82\\xB5\\x82܂\\xB5\\x82\\xBD\\x81B \\\"}\\r", :expected_charset=>"UTF-8", :level=>:warn}
I suspect the issue is one of your Windows machines has different character set configured than what the Logstash input on port 3515 is expecting.
Can you share the output of the following command:
Code: Select all
grep '' /usr/local/nagioslogserver/logstash/etc/conf.d/*
As well as any nxlog configurations from your Windows machines that are sending logs? If you've got a lot of Windows machines shipping (more than is practical to grab all the configs from), I'd be interested specifically in any non-english machines.
Re: Nagios Logserver Logstash service going down frequently
Posted: Mon Jan 09, 2017 5:46 am
by anish
Re: Nagios Logserver Logstash service going down frequently
Posted: Mon Jan 09, 2017 1:54 pm
by mcapra
I suspect one of your Windows machines is using a different charset than CP-1252. It is then hitting this input:
Code: Select all
tcp {
type => 'eventlog'
port => 3515
codec => json {
charset => 'CP1252'
}
}
Which will interpret the event as CP-1252 encoded and try to convert it to UTF-8. If the event received by this input is not CP-1252, Logstash gets confused. I would verify that all your Windows machines sending to port 3515 are using CP-1252 and not some other charset. Running
chcp from the Windows command line will usually tell you what codepage is being leveraged.