Incorrect data from command line export?

[jsharris]

I am trying to find a way to export log information to file but when searching I found that exporting or reporting on the log data was not possible from within Nagios Log Server. I found at https://support.nagios.com/forum/viewto ... 37&t=34370 that the query can be entered at the command line and the results, ugly as they are, could be written to a file.

However, when attempting to do so the data returned does not match the data in the GUI. Here is my query:

Code: Select all

curl -XGET 'http://servername/index.php/api/backend/logstash-2015.12.23,logstash-2015.12.22/_search?pretty&token=xxxxxxx' -d '{
  "query": {
	  "filtered": {
	  "query": {
	  "bool": {
	  "should": [
	  {
		  "query_string": {
		  "query": "searchitem*"
	  }
	  }
	  ]
	  }
	  },
	  "filter": {
	  "bool": {
	  "must": [
	  {
		  "range": {
		  "@timestamp": {
		  "from": 1450799593626,
		  "to": 1450885993626
	  }
	  }
	  }
	  ]
	  }
	  }
	  }
	  },
  "highlight": {
  "fields": {
  "*": {}
  },
  "fragment_size": 2147483647,
  "pre_tags": [
  "@start-highlight@"
  ],
  "post_tags": [
  "@end-highlight@"
  ]
  },
  "size": 3000,
  "sort": [
	  {
	  "@timestamp": {
	  "order": "desc",
	  "ignore_unmapped": true
  }
  },
  {
	  "@timestamp": {
	  "order": "desc",
	  "ignore_unmapped": true
  }
  }
  ]
}'
    

My dashboard currently returns 5 items using this query. The output file indicates a total of 32,920,667 hits with only 12 actually included in the output file with none of them matching the criteria of the query. How can I successfully access this information and make it available to those requesting a report on whatever we are logging?

Also, the above referenced ticket indicates that reporting functionality is one of the most requested features. Are there any updates on when this will be available as without any sort of reporting or export ability it renders the data gathered virtually useless; just sending upper management a link to the GUI when they ask "give me a list of who accessed x between y and z?" probably won't go over very well

As always, any and all assistance is appreciated.

[jolson]

I imagine you're getting this API call from the Nagios Log Server GUI directly. Could you show me which 'inspect' button you're pressing to generate the query? It looks like your query is targeting the last 24 hours of log time - does that sound like what you have present on your dashboard?

Could you let me know specifically what you're trying to query for? I'd like to generate a query that you could use on your system and see if the results differ from what the 'inspect' button provided you with.

[jsharris]

Screenshot of inspect button location attached; it is in the All Events panel. In this instance it is a search for "SPDFS1 and sharris" over the last 24 hours.

[jolson]

The following query (ran from your NLS box) should get you what you're after:

Code: Select all

curl -XPOST "http://localhost:9200/_search" -d'
{
    "query": {
        "filtered": {
        	"filter": {
        		"range": {
         			"@timestamp": {
          			"from": "now-5m",
                                "to": "now"
        					}
                        }
                    },
                "query": {
                	"query_string": {
                    "query": "syslog"
                }
            }
        }
    }
}'

This will look through the last 5 minutes of logs for any mention of the query 'syslog'. Please modify the 'query' and 'from' fields as you see fit - I have tested and verified that this query works properly on my box.

[jsharris]

The results appear to include what I may be looking for but the they are returned as a wall of completely unformatted text with no breaks (single line). How can we get the minimal formatting which was provided in the original query? Also, based on a search of the term "timestamp" it looks like the query only returned 10 results while the GUI returned 16.

[tmcdonald]

This should do it. Just forgot the ?pretty flag:

Code: Select all

curl -XPOST "http://localhost:9200/_search?pretty" -d'
{
    "query": {
        "filtered": {
        	"filter": {
        		"range": {
         			"@timestamp": {
          			"from": "now-5m",
                                "to": "now"
        					}
                        }
                    },
                "query": {
                	"query_string": {
                    "query": "syslog"
                }
            }
        }
    }
}'

[jsharris]

That worked to break up the entries (still way too much information though and no way to filter it) but we can probably find a way to make it work. However, the data returned still does not match between the two methods. GUI is now returning 4 items while the command line is coming back with 10. When I change the query to two days (tried using now-48h and now-2d), the GUI shows 20 and the command line still has 10.

[jolson]

Are you certain that you're using the same queries between the command line and the GUI? If so, please check on the system time of NLS and ensure that it's correct:

Code: Select all

date
grep timezone /etc/php.ini

On my test system, the results are identical.

no way to filter it

What kind of filtering mechanism are you looking for? Would something like type=syslog suit your environment?

[jsharris]

Looks like the time, date, and timezone are set correctly. As near as I can tell the queries are the same; the time range is either one day or two, the query is "spdfs1 and sharris" for both.

As for the filter, I think I may have stated that incorrectly. We do not need the level of detail provided by default by the command line query results. So where the columns can be specified in the GUI panel, a similar selection option from the command line is what we would want (e.g. for query "x", provide the hostname, username, timestamp, directory, etc.). Is there an option in the query syntax that could provide this (like a "select x,y,z from table1" in SQL)?

[jolson]

There's certainly a construct to accomplish what you want, give this a try:

Code: Select all

curl -XPOST "http://localhost:9200/_search?pretty" -d'
{
    "fields": ["user", "message", "type"],
    "query": {
        "filtered": {
           "filter": {
              "range": {
                  "@timestamp": {
                   "from": "now-5m",
                                "to": "now"
                       }
                        }
                    },
                "query": {
                   "query_string": {
                    "query": "syslog"
                }
            }
        }
    }
}'

The fields you define in the 'fields' setting are the ones that will show up in your query results.

Would you please try a different query to see whether or not the time is still improper? I can't reproduce any strange behavior on my lab system, but I think it could be related to your query.