Extract a query to another NLS instance

[batzos]

One of the basic concerns I have regarding the tool is the possibility to assign granular access to certain people. For example, during an investigation, I would like to give access to a person only to past logs of certain servers (not all of them imported into the tool). Can we run a query on logs of these specific servers and send these logs to another NLS instance? This instance will be used only for "visitors" giving them access to these specific logs in order process them and run new queries. These specific logs have to be saved in the "visitors" NLS. This will concern only past logs and not a continuous feeding of the "visitors" server with present logs.
Another possibility could be to save the results of the query in the 1st server somewhere in a separate folder of its repository (as a snapshot?) and then assign this folder to the "visitors" NLS instance?
Some weeks ago you sent me the following procedure which is not very clear to me. Please describe to me more analytical the procedure.
Can we use an example? I want to run a query for all Linux logs with input in the primary NLS:
syslog {
type => 'syslog'
port => 5544
}

of the period from 2016-01-22 00:00:00 to 2016-01-25 00:00:00 of the servers with severity from 0 to 5 included and send the results to the "visitors" server.
Could you describe me analytically how do I have to configure the input, filter and output of both the primary NLS server and the "visitors" server?
(I want to continue receiving normally all the Linux logs in the primary server as before.)


Below are my comments to our previous communication and the solution you have presented to me.
Apply the filters where, in the query window or in global configuration?
How can I flag them?
Use where the input, filter and output? Input in the "visitors" server and filter and output in the primary NLS?
Where do I set the IP of the target "visitors" server?

And the reply from Nagios support:
"This can easily be done with Log Server. You apply all the filters to get only the desired logs and then you can flag it and use outputs in Log server and send it to a separate install of Log server where you set up inputs to accept the information. You'll want to use the TCP input and output to accomplish this.

output configuration:

To send specific events to a different Elasticsearch cluster, the
process would look like the following:

Servers --> output1 --> | --> elasticsearch1
|
------> | --> elasticsearch2

Example configuration:

input {
port => 1234
type => 'privatedata'
}

filter {

# design your own grok filter to parse the log data appropriately
grok {}

filter {
if [type] == "privatedata" {
add_field => { "privacy" => 'private' }
}
}

output {
if [privacy] == "private" {
elasticsearch {
# private elasticsearch instance
}
}
}

"

[jolson]

One of the basic concerns I have regarding the tool is the possibility to assign granular access to certain people. For example, during an investigation, I would like to give access to a person only to past logs of certain servers (not all of them imported into the tool).

Just for your reference, we are working on granular access for users now, as it's the most requested feature to have in the next version of NLS.

Let's say we have two clusters - one 'primary' cluster and one 'visitor' cluster.

You'll be sending the sensitive information to your primary cluster, and then duplicating that information over to your visitor cluster.

I set up a test environment tonight, and I'll see if data populates overnight - I'll get back to you tomorrow with results. This is what my global configuration page looks like:

2016-02-01 18_15_48-Instance Configuration • Nagios Log Server - Chromium.png

[batzos]

Thank you Jesse for your feedback.
I guess the output configuration is for the 1st cluster which is the source of the desired logs and 192.168.4.186 is the 2nd cluster which is the target.
Concerning the "custom-input" this is the input of the first cluster, because for me it looks like the input of the 2nd one. Can you define analytically both inputs for 1st and 2nd cluster?
Can you describe how "resticted-log" is produced? For my case, I need to run query for certain past logs which are already stored in the 1st cluster and then export this result to the 2nd cluster. How can I flag this query result as "resticted-log"?
How can I do the following as you said: "You'll be sending the sensitive information to your primary cluster"
Thank you in advance.

[jolson]

While I'm sure that we could come up with a way to accomplish what you're trying to design, I'd like to offer an alternative.

Because we've already confirmed granular user log access as part of our roadmap, I think it'd be best to wait until such a point that the functionality exists as-is in our software. The reason I think that you should take this route is because:
1. You will save on license costs
2. You will save on management time
3. You will not have an extra cluster that needs to be taken down once we release the granular user access update.

Does that make sense to you? Building the cluster out now and fully understanding it before giving particular users access may be the best call for the reasons mentioned above.

Thanks!

Jesse

[batzos]

I am looking forward to this functionality and of course it would simplify things in terms of cost, time and complexity, but I cannot provide this answer to my management.Right now I have to provide a solution for granular access which is a basic constraint for the success of the PoC, regardless the cost. I hope you will propose to me a way to do it. At least, can you describe me how can I extract a query? Even from the CLI, if the query results can be saved in a file and then this file could be imported into another instance.
Do you have a provisional date for the implementation of the functionality of granular access in your SW?

[rkennedy]

We do not have an estimated date available. I was able to get this working, with output to a file by using this as an output -

Code: Select all

if [type] == 'restricted-log' {
file {
path => "/tmp/restrictedlog.txt"
}
}

From there, it will output anything that has a type=restricted-log to /tmp/restricted-log.txt.
Does this work for you?

[batzos]

What I am still missing is how to "produce" the input: 'restricted-log', which has to be the result of a query. You have declared it as an input in port 2333, but how is my query forwarded to this input? For example, if I want to send logs of a server I configure its rsyslog to send the logs to the NLS IP in certain port and at the NLS I configure the input to accept these logs from this particular port and tag them as I want. Here, how do I send a query result to the input?

[rkennedy]

The type, we are defining on input. So any input on a given port, is tagged with a type. See this example -

Code: Select all

tcp {
  port => 2333
  type => 'restricted-log'
}

I think a better way to look at this is not based on the restricted-log, but based on other fields.

For example, you can match based on any of the fields. This, would log everything with a host of '192.168.3.116' -

Code: Select all

if [host] == '192.168.3.116' {
file {
path => "/tmp/restrictedlog.txt"
}
}

[batzos]

I created the following output:

Code: Select all

if [host] == '10.22.244.136' {
file {
path => "/tmp/restrictedlog2.txt"
}
}

selecting the logs of a specific host and indeed they were saved in the txt file. The problem is that I want to do this for past logs that are already stored. The solution you have proposed me saves continuously the logs into the txt file by the time I activate the output. No previous logs are saved. I want to define the period of these logs, for example I want to save to a file or forward to another instance the logs of the last month of a certain host. Is this possible? It is the same as making a query in the GUI of NLS and send this query to a file or another NLS. What I want is to do a query in stored logs. For example for the host: 10.27.1.4 with facility 3.
I set the following input, but it does not work. Could you help me with the syntax?

Code: Select all

# Read all documents from Elasticsearch matching the given query
  elasticsearch {
    type => 'Query1'
    hosts => "10.27.1.4"
    query => '{ "query": { "match": { "facility": 3 } } }'
  }

I have also set the following output:

Code: Select all

if [type] == 'query' {
file {
path => "/tmp/query.txt"
}
}

to get the txt file, but it also does not work, probably because 'query' is not a "type". Do I have to set a type or tag this input? For example are the following right?

Code: Select all

# Read all documents from Elasticsearch matching the given query
  elasticsearch {
    type => 'Query1'
    hosts => "10.27.1.4"
    query => '{ "query": { "match": { "facility": 3 } } }'
    tags => 'Q1'
  }

or

Code: Select all

# Read all documents from Elasticsearch matching the given query
  elasticsearch {
    type => 'Query1'
    hosts => '10.27.1.4'
    query => '{ 'query': { 'match': { 'facility': 3 } } }'
    tags => 'Q1'
  }

Additionally, if I want to send this query to another NLS instance (192.168.0.23), is the following output correct?

Code: Select all

if [type] == 'Query1' {
         elasticsearch{
    host => "192.168.0.23"
    port => "9300"      
    }
 }

[hsmith]

The problem is that I want to do this for past logs that are already stored

Unless I'm mistaken, there's not a way to do that other than loading the logs back in again.

The solution you have proposed me saves continuously the logs into the txt file by the time I activate the output. No previous logs are saved.

Is this with using the output to a file? It's overwriting it?

I want to define the period of these logs, for example I want to save to a file or forward to another instance the logs of the last month of a certain host. Is this possible?

This really is looking like something you're going to have to script.

What I want is to do a query in stored logs. For example for the host: 10.27.1.4 with facility 3.

Are you referring to the logs that your outputting to?

Do I have to set a type or tag this input? For example are the following right?

This is not tested by us at all, have you tried looking at Elasticsearch's documentation?

Additionally, if I want to send this query to another NLS instance (192.168.0.23), is the following output correct?

Again, not tested, but it looks right for if the type is 'Query1'.

I can spend a little time researching this tomorrow, but this is getting pretty close to being consulting. I understand that you're doing this as a workaround until we implement the ability to restrict certain logs to certain users, but it's still something that is going to take a lot of time away from other support duties for us to resolve. I'll spend some time looking at this tomorrow, but like I said, it may not be something we can spend a lot of time on, as it takes away from other support time.