Page 1 of 2
Incorrect data from command line export?
Posted: Wed Dec 23, 2015 11:17 am
by jsharris
I am trying to find a way to export log information to file but when searching I found that exporting or reporting on the log data was not possible from within Nagios Log Server. I found at
https://support.nagios.com/forum/viewto ... 37&t=34370 that the query can be entered at the command line and the results, ugly as they are, could be written to a file.
However, when attempting to do so the data returned does not match the data in the GUI. Here is my query:
Code: Select all
curl -XGET 'http://servername/index.php/api/backend/logstash-2015.12.23,logstash-2015.12.22/_search?pretty&token=xxxxxxx' -d '{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"query_string": {
"query": "searchitem*"
}
}
]
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"from": 1450799593626,
"to": 1450885993626
}
}
}
]
}
}
}
},
"highlight": {
"fields": {
"*": {}
},
"fragment_size": 2147483647,
"pre_tags": [
"@start-highlight@"
],
"post_tags": [
"@end-highlight@"
]
},
"size": 3000,
"sort": [
{
"@timestamp": {
"order": "desc",
"ignore_unmapped": true
}
},
{
"@timestamp": {
"order": "desc",
"ignore_unmapped": true
}
}
]
}'
My dashboard currently returns 5 items using this query. The output file indicates a total of 32,920,667 hits with only 12 actually included in the output file with none of them matching the criteria of the query. How can I successfully access this information and make it available to those requesting a report on whatever we are logging?
Also, the above referenced ticket indicates that reporting functionality is one of the most requested features. Are there any updates on when this will be available as without any sort of reporting or export ability it renders the data gathered virtually useless; just sending upper management a link to the GUI when they ask "give me a list of who accessed x between y and z?" probably won't go over very well
As always, any and all assistance is appreciated.
Re: Incorrect data from command line export?
Posted: Wed Dec 23, 2015 4:22 pm
by jolson
I imagine you're getting this API call from the Nagios Log Server GUI directly. Could you show me which 'inspect' button you're pressing to generate the query? It looks like your query is targeting the last 24 hours of log time - does that sound like what you have present on your dashboard?
Could you let me know specifically what you're trying to query for? I'd like to generate a query that you could use on your system and see if the results differ from what the 'inspect' button provided you with.
Re: Incorrect data from command line export?
Posted: Mon Dec 28, 2015 8:41 am
by jsharris
Screenshot of inspect button location attached; it is in the All Events panel. In this instance it is a search for "SPDFS1 and sharris" over the last 24 hours.
Re: Incorrect data from command line export?
Posted: Mon Dec 28, 2015 4:50 pm
by jolson
The following query (ran from your NLS box) should get you what you're after:
Code: Select all
curl -XPOST "http://localhost:9200/_search" -d'
{
"query": {
"filtered": {
"filter": {
"range": {
"@timestamp": {
"from": "now-5m",
"to": "now"
}
}
},
"query": {
"query_string": {
"query": "syslog"
}
}
}
}
}'
This will look through the last 5 minutes of logs for any mention of the query 'syslog'. Please modify the 'query' and 'from' fields as you see fit - I have tested and verified that this query works properly on my box.
Re: Incorrect data from command line export?
Posted: Tue Dec 29, 2015 6:41 am
by jsharris
The results appear to include what I may be looking for but the they are returned as a wall of completely unformatted text with no breaks (single line). How can we get the minimal formatting which was provided in the original query? Also, based on a search of the term "timestamp" it looks like the query only returned 10 results while the GUI returned 16.
Re: Incorrect data from command line export?
Posted: Tue Dec 29, 2015 10:22 am
by tmcdonald
This should do it. Just forgot the
?pretty flag:
Code: Select all
curl -XPOST "http://localhost:9200/_search?pretty" -d'
{
"query": {
"filtered": {
"filter": {
"range": {
"@timestamp": {
"from": "now-5m",
"to": "now"
}
}
},
"query": {
"query_string": {
"query": "syslog"
}
}
}
}
}'
Re: Incorrect data from command line export?
Posted: Tue Dec 29, 2015 10:52 am
by jsharris
That worked to break up the entries (still way too much information though and no way to filter it) but we can probably find a way to make it work. However, the data returned still does not match between the two methods. GUI is now returning 4 items while the command line is coming back with 10. When I change the query to two days (tried using now-48h and now-2d), the GUI shows 20 and the command line still has 10.
Re: Incorrect data from command line export?
Posted: Tue Dec 29, 2015 11:59 am
by jolson
Are you certain that you're using the same queries between the command line and the GUI? If so, please check on the system time of NLS and ensure that it's correct:
On my test system, the results are identical.
no way to filter it
What kind of filtering mechanism are you looking for? Would something like
type=syslog suit your environment?
Re: Incorrect data from command line export?
Posted: Tue Dec 29, 2015 12:11 pm
by jsharris
Looks like the time, date, and timezone are set correctly. As near as I can tell the queries are the same; the time range is either one day or two, the query is "spdfs1 and sharris" for both.
As for the filter, I think I may have stated that incorrectly. We do not need the level of detail provided by default by the command line query results. So where the columns can be specified in the GUI panel, a similar selection option from the command line is what we would want (e.g. for query "x", provide the hostname, username, timestamp, directory, etc.). Is there an option in the query syntax that could provide this (like a "select x,y,z from table1" in SQL)?
Re: Incorrect data from command line export?
Posted: Tue Dec 29, 2015 12:25 pm
by jolson
There's certainly a construct to accomplish what you want, give this a try:
Code: Select all
curl -XPOST "http://localhost:9200/_search?pretty" -d'
{
"fields": ["user", "message", "type"],
"query": {
"filtered": {
"filter": {
"range": {
"@timestamp": {
"from": "now-5m",
"to": "now"
}
}
},
"query": {
"query_string": {
"query": "syslog"
}
}
}
}
}'
The fields you define in the 'fields' setting are the ones that will show up in your query results.
Would you please try a different query to see whether or not the time is still improper? I can't reproduce any strange behavior on my lab system, but I think it could be related to your query.