Placing data into new indexes by type

[gregoryd]

What is the logserver supported method to send data of a certain type to another index?

We tried adding an if statement to the output section, but we found when we did this that data was being written to the default logstash-YYYY.MM.dd index and the syslog-YYYY.MM.dd index we created.

This looks to be happening because of the way logserver puts the default output entry into the configuration file.

Any ideas on how to properly do this?

Code: Select all

[root@hostname conf.d]# pwd
/usr/local/nagioslogserver/logstash/etc/conf.d
[root@hostname conf.d]# cat 999_outputs.conf 
# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Tue, 10 Mar 2015 13:43:44 -0500
#

#
# Required output for Nagios Log Server
#

output {
    elasticsearch {
        cluster => 'xxxxxxx-xxxxxx-xxxxxx-xxxx-xxxxxxxx'
        host => 'localhost'
        index_type => '%{type}'
        node_name =>  'xxxxxxx-xxxxxx-xxxxxx-xxxx-xxxxxxxx'
        protocol => 'transport'
        workers => 4
    }
}

#
# Global outputs
#

output {
    if [type] == 'syslog' {
       elasticsearch {
            index => 'syslog-%{+YYYY.MM.dd}'
            cluster =>  'xxxxxxx-xxxxxx-xxxxxx-xxxx-xxxxxxxx'
            host => 'localhost'
            index_type => '%{type}'
            node_name =>  'xxxxxxx-xxxxxx-xxxxxx-xxxx-xxxxxxxx'
            protocol => 'transport'
            workers => 4
        }
    }
}

#
# Local outputs
#


[root@hostname conf.d]# 

[jolson]

Just to be clear - you are trying to accept logs into Logstash from some remote sender, and after accepting those logs you want to output them to a particular Elasticsearch index? Is that correct? Or are you trying to output them to a different external device?

[gregoryd]

We want to output them to a particular Elasticsearch index without duplicating them in the default if possible.

[jolson]

After re-creating your setup, I am seeing the same results on my cluster. I haven't found a way to delete the replicate data left behind by the first output. The only way that I can imagine doing it is by adding an 'if' statement to the original output configuration (if type = syslog action = delete) to see if that resolves it. Since the file is generated dynamically, this will not work. I have made a note to continue testing this tomorrow morning.

[jolson]

Right now, it does not look like multiple outputs are possible without data duplication.

[jolson]

Hello,

I have confirmed with development that it may not ever be possible for data to not hit the default index. Unfortunately you'll either need to concatenate your data into the default index, or duplicate the data.

Best,


Jesse