Tiered NLS Architecture

[mcapra]

Can you share the contents of the elasticsearch logs from both machines? They're typically found in /var/log/elasticsearch. If you could designate which logs are from the 112 and 113 machines that would be helpful

I would also like to see /var/log/logstash/logstash.log from the 112 machine.

[animumrege]

Ill PM you the logs.

Thanks

[mcapra]

Looking over the logstash logs on the 112 machine, it looks like it can't properly find a node on the 113 machine:

Code: Select all

{:timestamp=>"2016-12-14T14:37:27.024000-0600", :message=>"Failed to flush outgoing items", :outgoing_count=>1, :exception=>org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [], :backtrace=>["org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(org/elasticsearch/client/transport/TransportClientNodesService.java:279)", "org.elasticsearch.client.transport.TransportClientNodesService.execute(org/elasticsearch/client/transport/TransportClientNodesService.java:198)", "org.elasticsearch.client.transport.support.InternalTransportClient.execute(org/elasticsearch/client/transport/support/InternalTransportClient.java:106)", "org.elasticsearch.client.support.AbstractClient.bulk(org/elasticsearch/client/support/AbstractClient.java:163)", "org.elasticsearch.client.transport.TransportClient.bulk(org/elasticsearch/client/transport/TransportClient.java:356)", "org.elasticsearch.action.bulk.BulkRequestBuilder.doExecute(org/elasticsearch/action/bulk/BulkRequestBuilder.java:164)", "org.elasticsearch.action.ActionRequestBuilder.execute(org/elasticsearch/action/ActionRequestBuilder.java:91)", "org.elasticsearch.action.ActionRequestBuilder.execute(org/elasticsearch/action/ActionRequestBuilder.java:65)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:606)", "RUBY.bulk(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.2.8-java/lib/logstash/outputs/elasticsearch/protocol.rb:224)", "RUBY.submit(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.2.8-java/lib/logstash/outputs/elasticsearch.rb:466)", "RUBY.flush(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.2.8-java/lib/logstash/outputs/elasticsearch.rb:490)", "Stud::Buffer.buffer_flush(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:219)", "Stud::Buffer.buffer_flush(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:219)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1341)", "Stud::Buffer.buffer_flush(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:216)", "Stud::Buffer.buffer_flush(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:216)", "Stud::Buffer.buffer_flush(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:193)", "Stud::Buffer.buffer_flush(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:193)", "RUBY.buffer_receive(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:159)", "RUBY.receive(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.2.8-java/lib/logstash/outputs/elasticsearch.rb:455)", "RUBY.handle(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/outputs/base.rb:88)", "RUBY.worker_setup(/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/outputs/base.rb:79)", "java.lang.Thread.run(java/lang/Thread.java:745)"], :level=>:warn}

Assuming the correct cluster and host have been properly set in the output rule on 112, you might try adding a specific node to the output definition:

Code: Select all

elasticsearch {
        cluster => 'a4e1a590-f811-4545-b7a9-c5b160afebc2'
        node_name => 'my_node_id'
        host => '192.168.67.4'
        document_type => '%{type}'
        protocol => 'transport'
        workers => 4
    }

But as far as I can tell, the 112 machine is never even speaking to the 113 machine's Elasticsearch cluster. I would double check the cluster ID and make sure 112 and 113 can talk over port 9300.

[animumrege]

That appears to be working. Awesome.

So from a top level perspective when we add mode nodes to the cluster, can we load balance connections from the relay to the top tier by connecting to any node? As we add nodes we didn't know if we could connect to any node within the cluster or just the primary node. We will probably put NGINX or another load balancer in the middle to make sure we spread the load.

Thanks

[mcapra]

Specifying the node name would likely trip up any 3rd party loadbalancer in this case. All of your inserts would be forcibly redirected to a specific node. Not really the best setup. In a situation where the host you're pointing to is the load balancer, you definitely wouldn't want to specify a node_name.

I would be interested in figuring out exactly why the following configuration wasn't working:

Code: Select all

elasticsearch {
        cluster => 'cluster_id'
        host => 'host_name'
        document_type => '%{type}'
        protocol => 'transport'
        workers => 4
    }

Did you verify the cluster id and hostname were correct prior to adding the node_name? It's very suspicious to me that hitting the cluster without specifying a node was throwing NoNodeAvailableException, but specifying a node_name eliminates that issue. That doesn't add up.

[animumrege]

No idea why it wasnt working. It appears to be now. I am open to letting you see the config first hand if interested.

[mcapra]

Well, I already have the Elasticsearch logs from the destination machine which is really all I can look at. Nothing in there indicates any obvious problems.

Did you have additional questions regarding the tiered setup that I can answer?

[animumrege]

The only additional question I have is how would I use SSL/TLS to secure this communication? Since it is outside of the configuration within NLS, we just want to make sure to configure it in the appropriate and supported manner.

We are expecting that the NLS main system will live in AWS/Azure so we need to protect the communication with encryption.

Thanks

[hsmith]

I was working on documentation for this before I left Nagios, but unfortunately never got it pushed out due to a high workload.

You'll want to take a look at a project called Lumberjack/Logstash Forwarder/Beats/Whatever Elastic Co is calling it this month.

I was able to successfully ship logs to NLS over a secure connection by making use of the following two resources:

Helpful blog post

Logstash-forwarder GitHub(I find this well documented)

I've since lost access to the server I had this set up on due to lousy management of my SSH keys, but I would imagine you can get it working with the two resources listed above.

[mcapra]

The options described by @hsmith are probably the most appropriate options. Otherwise, you can adjust the output rule I shared to use http for the protocol and encrypt the message as https using the ssl settings for the elasticsearch output plugin:
https://www.elastic.co/guide/en/logstas ... search-ssl