Importing query data

[stecino]

This is an example query result that I have exported to json, json2.txt contins the query body

curl -XGET 'http://xx.xx.1.246/nagioslogserver/inde ... 0b1d990a1a' -d json2.txt > test.json

I want to be able to convert this json into csv format. But when I put it through converter it complains about validity of json file. I used json validator and it complains on every message

{
"_index":"logstash-2015.01.30",
"_type":"syslog",
"_id":"5dfs8ONnTbWYr9w8-gfhqg",
"_score":1,
"_source":{
"message":"connect from unknown[
10.xx.xx.xxx
]",

Assuming that json file generated is the same everytime, how can I get rid of this issue?

[tmcdonald]

Unless this was just a typo on the forums, it looks like you left out the closing "}" bracket.

[stecino]

Unless this was just a typo on the forums, it looks like you left out the closing "}" bracket.

It was just a snippet, I fount out the root cause. It's because messages field is multiline instead of a single line. How can i go from multiline to single line?

[scottwilkerson]

If you have logs with multiline input you would need to create a new input fo rthem adding the multiline codec
http://logstash.net/docs/1.4.2/codecs/multiline

this will definately require some knowledge of how the messages are written as you will have to form a pattern to match for the stream

[stecino]

Inspect generating the elasticsearch query, doesn't return all the results. It keeps returning 10 results

curl -XGET 'http://xx.xx.x.246/nagioslogserver/inde ... 0b1d990a1a'

Is there a way to bypass nagios, and hit ElasticSearch API directly?

[tmcdonald]

Code: Select all

curl -XGET 'http://127.0.0.1:9200/?pretty'

That will do a basic query that returns some info about the cluster. You can add the "_search" endpoint to the URL with a "&q=something" added on to search for "something":

Code: Select all

curl -XGET 'http://127.0.0.1:9200/_search?pretty&q=something'

Note that by default this needs to be done from the command line.

[stecino]

Code: Select all

curl -XGET 'http://127.0.0.1:9200/?pretty'

That will do a basic query that returns some info about the cluster. You can add the "_search" endpoint to the URL with a "&q=something" added on to search for "something":

Code: Select all

curl -XGET 'http://127.0.0.1:9200/_search?pretty&q=something'

Note that by default this needs to be done from the command line.

For a specific issue in hand, I want to grab from index logstash-2015.02.05, all records that have a type=LTM302_log

Would a query be something like this?

I am getting 0 hits

curl -XGET 'http://127.0.0.1:9200/_search?pretty&q= ... 2015.02.05'
{
"took" : 2982,
"timed_out" : false,
"_shards" : {
"total" : 161,
"successful" : 161,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}

I also tried type=syslog from current index, still getting 0 hits

[root@pbur2nls2 ~]# curl -XGET 'http://127.0.0.1:9200/_search?pretty&q= ... ype;syslog'
{
"took" : 1805,
"timed_out" : false,
"_shards" : {
"total" : 161,
"successful" : 161,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}

[tmcdonald]

The semicolons in your query shouldn't be there, and you have one too many "q"s. Try this:

Code: Select all

curl -XGET 'http://127.0.0.1:9200/logstash-2015.02.05/_search?q=type:LTM302_log&pretty'

[stecino]

The semicolons in your query shouldn't be there, and you have one too many "q"s. Try this:

Code: Select all

curl -XGET 'http://127.0.0.1:9200/logstash-2015.02.05/_search?q=type:LTM302_log&pretty'

what if I wanted to do multiple fileds, and also multiple index files, how can I do it, please?

Can you give me an example?

[stecino]

Another thing I tried, please note this is a output snippet.

curl -XGET 'http://127.0.0.1:9200/logstash-2015.02. ... log&pretty'
{
"took" : 38,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 2504,
"max_score" : 1.0,
"hits" : [ {
"_index" : "logstash-2015.02.06",
"_type" : "eb_map_log",
"_id" : "SfbgpwBkRry3p8dHEVtckg",
"_score" : 1.0,
"_source":{"message":"06 Feb 2015 02:49:00 DEBUG com.platform.feed.twitter.service.impl.TwitterFollowerDispenserImpl - Twitter follower lists 2643230248140801 claimed, will run on server 204","@version":"1","@timestamp":"2015-02-06T10:49:07.000Z","type":"eb_map_log","host":"10.xx.xx.xxx","priority":133,"timestamp":"Feb 6 02:49:07","logsource":"xxxapi02","program":"eb_map_log","severity":5,"facility":16,"facility_label":"local0","severity_label":"Notice","geoip":{"ip":"xx.xx.xx.xx","country_code2":"US","country_code3":"USA","country_name":"United States","continent_code":"NA","region_name":"CA","city_name":"Burbank","postal_code":"91501","latitude":34.1994,"longitude":-118.29140000000001,"dma_code":803,"area_code":818,"timezone":"America/Los_Angeles","real_region_name":"California","location":[-118.29140000000001,34.1994]},"tags":["geoip"]}
}, {

instead of 2504 hits as indicated, it only returns 20 results in the hits list. Why is this the case?