Deleting data or retagging it

[eloyd]

We want to bring in some customer equipment logs to our NLS server but direct, real-time connectivity is not an option. So we looked at scp'ing log files over and importing them with shipper.py. Unfortunately, we did not realize (or maybe, did not think through properly?) that the data would be tagged as coming from import_json and that the host would be the local NLS host. It is important to us to know the host that the log information was generated from, since we want to use it for capacity planning purposes for each customer.

So the question is - is there a way to retag this data once it has been imported, or perhaps to delete it and import it again with the logsource identified as being the original machine and the timestamp being what's in the log file (instead of when it was imported)?

I'd really be interested in data management within NLS to be able to delete/roll-up records that are no longer relevant. Deleting this data would be a real good start, for instance, since it it basically useless but skewing our results.

[sreinhardt]

I'm not sure if this is available in the present incarnation of shipper.py, but it is basically a necessity to be able to specify sending host and tag. I will put in a feature request now!

[eloyd]

Thanks. Any word on how to delete this data from our existing indicies?

[scottwilkerson]

There isn't a GUI version of this but you can do it fron the console on one of the cluster instances

Run the following replacing LOGSOURCE with the text of your logsource

Code: Select all

curl -XDELETE 'http://localhost:9200/logstash/_all/_query' -d '{
    "query" : {
        "term" : { "logsource" : "LOGSOURCE" }
    }
}'