write in to a CSV file

[lukedevon]

Hi

From the following query, I am able to search specific data accurately in incoming logs.

(host="IP_1" & host="IP_2" & host="IP_3" & host="IP_4") && ("String_1:" && "String_2")

I want to write those captured data in to a csv file. I have installed CSV_OUTPUT plugin.

https://www.elastic.co/guide/en/logstas ... s-csv.html

My question is, how can I send those selected data to the CSV file at the same time?

I found some articles online related my requirement. But most of them are not real time creating CSV files. those examples says, We have to give a specific index file to generate the csv.

I want to write in to the csv file real time, whenever searching criteria matched, it has to be written in to the csv.

can we use elasticsearch input plugin ;

elasticsearch {
hosts => "localhost:9200"
index => "index-file-name"
query => '
{"query": {
(host="IP_1" & host="IP_2" & host="IP_3" & host="IP_4") && ("String_1:" && "String_2")}
}
}

If we can use it, then what would be the search query? Can we use same query as I have used to search data?

Once found the data using elasticsearch plugin, then how can I forward the logs to csv-ouput plugin?

Can you please give me a little guide.

Thank you
Luke.

[cdienger]

An output like like this would write to an csv file as it comes in without having to query the elasticsearch database:

Code: Select all

if [host] in ["IP_1","IP_2","IP_3"] and ([message] =~ "String_1" or [message] =~ "String_2") {
csv { 
fields => ["timestamp", "host", "message" ]
path => "/usr/local/nagioslogserver/output.csv"
}
}

[lukedevon]

Hi

Thank you. It works.

I have another question;

I want to execute some shell commands to cut only a specific strings in the message before the complete message sending to the csv file.
I just tested and the shell commands are working fine after the csv file has been created.

But I need to execute the shell commands before creating the csv file from the output plugin itself.

According to the example here;

if [host] in ["IP_1","IP_2","IP_3"] and ([message] =~ "String_1" or [message] =~ "String_2") {
csv {
fields => ["timestamp", "host", "message" ]
path => "/usr/local/nagioslogserver/output.csv"
}
}

I need to run the shell commands right after field section, where the string comes in the message.
fields => ["timestamp", "host", "message" ]

my shell command grep some string in the message and cut only the specific strings out of it and save it to the csv file.

May I know is this doable within the output plugin ?

Regards
Luke.

[cdienger]

You can use a grok match in a filter to pull this information out, store it in a field, and then save that field to the file. The filter:

Code: Select all

if [host] in ["IP_1","IP_2","IP_3"] and ([message] =~ "String_1" or [message] =~ "String_2") {
grok{
match => { "message" => "(?<string>.*)" }
}
}

string can be replaced with the fieldname you'd like to use. the ".*" is a regex to select all and this would need to be changed to pull the string you want.

You can then use this as the output:

Code: Select all

if ["" in string] {
csv {
fields => ["timestamp", "host", "string" ]
path => "/usr/local/nagioslogserver/output.csv"
}
}

[lukedevon]

Thank you. It was a great help and I was able to write in to the filtered data to a file.

[scottwilkerson]

Thank you. It was a great help and I was able to write in to the filtered data to a file.

Great!

Locking thread