S3 Bucket Logs to Logstash not working? Advice?

[jslawrenc]

I was hoping someone could help me out here. I have Cisco Managed S3 bucket containing gzip files which I need to pull into my CentOS Nagios LMS. Using the below GC input example, I always get the "configuration could not be verified" when doing a SAVE>>APPLY. I'm not seeing any relevant errors in logstash.log / nagios.log. I know my input parameters are correct as I can navigate around using AWS CLI. Is there another log specific to global config inputs that might help me identify the problem? I'm very new working with this. Any advice-feedback would be appreciated.

Input {
s3 {
access_key_id => ".."
bucket => ".."
region => ".."
secret_access_key => ".."
prefix => ".."
type => "s3"
}
}

[cdienger]

The input{} isn't necessary when you add the input via the web UI and the region must be one listed at https://www.elastic.co/guide/en/logstas ... ts-s3.html.

s3 {
access_key_id => ".."
bucket => ".."
region => "us-east-1"
secret_access_key => ".."
prefix => ".."
type => "s3"
}

[jslawrenc]

Thanks a lot cdienger. I did have my region set correctly, but removing the input{} fixed the initial verify error & started to producing events in logstash.log. The S3 log source is still not showing up as a source in LMS-elasticsearch and I'm now seeing "Logstash S3 input, stop reading in the middle of the file" errors. I'm guessing Logstash has some problem with the gzip log source data. I'm further along than I was thanks to your suggestion.

[cdienger]

Where are you seeing these errors exactly? Do you see anything in /var/log/logstash/logstash.log ?

[jslawrenc]

Yes- /var/log/logstash/logstash.log. I haven't seen the error reappear since defining charset => "ISO-8859-1, but still not seeing the S3 logs show up in ElasticSearch. If I can trouble you with one last question. If you don't define "Input" within Global Config input web browser, I presume I wouldn't define an output either? (every attempt to set ElasticSearch on localhost as an output fails with an error). Assuming everything else is configured correctly, should those S3 logs show up in the GUI without needing to define an output statement?

[cdienger]

The output by default is the elasticsearch database and this config can be seen on file system in/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf. You can define additional outputs under Configure > Global > Global Config > Show Outputs. You do not have to use output{} if you define additional outputs - input{},filter{}, and output{} will wrap any of the input,filters, or outputs automatically.

S3 logs should show up in the GUI without having to define an additional output statement.

[jslawrenc]

Thanks again cdienger. Greatly appreciated.

[tmcdonald]

Did you have further (related) questions or are we good to lock this up?

[jslawrenc]

Hi TMCDonald- I have been unable to resolve the S3 log pull from ElasticSearch to Logstash>>Kibana. /var/log/logstash/logstash.com warning is "Logstash S3 input, stop reading in the middle of the file, we will read it again when logstash is started". If you can offer any guidance on why this may be occurring, or see any clear problems with my input, I'd certainly appreciate Nagios support assistance.

s3 {
access_key_id => ".."
bucket => "cisco-managed-us-west-1"
region => "us-west-1"
secret_access_key => ".."
prefix => ".."
backup_add_prefix => "logstash-"
type => "s3"
}

[jslawrenc]

One update: Running logstash in debug, I observed the error "The shutdown process appears to be stalled due to busy or blocked plugins" which might suggest my S3 plugin is receiving data at a slow pace. This seems to be somewhat common with S3 bucket logs. I'm not sure how to remedy this yet, but thought I'd share in case it's useful for anyone else down the road.