Page 1 of 2
Need some help on accessing log data
Posted: Mon Dec 07, 2015 2:18 pm
by gregwhite
I need some quick help. The person that set up our Nagios log server is out for the next 3 months and I have been trying to do a crash course. The server has been set up to save syslogs for 6 months. However, when the network folks do a query it will only go back a few days to a week. I am seeing indexes that go back to August when we started collecting the syslog data. is there something I need to do to make that data available?
Thanks,
Greg
Re: Need some help on accessing log data
Posted: Mon Dec 07, 2015 3:54 pm
by rkennedy
Just to make sure, what is the 'Time Filter' running at when executing a query?
logserver-timefilter.PNG
Re: Need some help on accessing log data
Posted: Mon Dec 07, 2015 4:10 pm
by gregwhite
Ok, It was the time filter. I need to set that before I do the query. Thanks.
The other problem that I am seeing is when I log in, it takes several minutes to get to the home page and several more to go to the dashboard tab. I have set the time filter to go back to July 15th. and it has been loading for 10 minutes and is only back to October 30th. Are there configuration setting we need to change? We are taking in syslogs for 551 network devices.
Thanks,
Re: Need some help on accessing log data
Posted: Mon Dec 07, 2015 4:57 pm
by rkennedy
Glad to see that worked!
What kind of resources do you have allocated to this machine?
From the CLI, can you run top|head -n17 and post the output?
Re: Need some help on accessing log data
Posted: Mon Dec 07, 2015 5:37 pm
by gregwhite
I set the time filter to select data from 8/02/15 to present and it only goes back to 10/20/15 and stops.
Here is output of top|head –n17
I also increased the value of LS_HEAP_SIZE=”1024m”
in /etc/sysconfig/logstash file and restarted bot logstash and elasticsearch daemons.
Log server output.jpg
Re: Need some help on accessing log data
Posted: Mon Dec 07, 2015 6:07 pm
by gregwhite
When I do a query, it only lists results from the current day and time. It loads data back to 10/30 even though I set my time filter to August 03rd.
Sorry to be a pain.
Re: Need some help on accessing log data
Posted: Tue Dec 08, 2015 11:25 am
by jolson
It's possible that there is index rotation involved here - I'd like to see the output of the following command:
Code: Select all
curl 'localhost:9200/_cluster/health?level=indices&pretty'
Re: Need some help on accessing log data
Posted: Tue Dec 08, 2015 11:57 am
by gregwhite
Attached please find the results.
Re: Need some help on accessing log data
Posted: Tue Dec 08, 2015 12:06 pm
by gregwhite
Oops! Here you go.
output_curl_lkenmycroft.txt
Re: Need some help on accessing log data
Posted: Tue Dec 08, 2015 12:38 pm
by jolson
How much RAM is in your instance(s)? The HEAP_SIZE setting should be set to _at least_ half of the available memory in any given instance.
Your indices look good, but there's quite a lot of data being held open currently - it looks like you have data open from July until now. As you can imagine, this can be taxing on Nagios Log Server - my suggestion is to either increase your HEAP_SIZE (and your memory if necessary) or close down some of the older indices you don't need for the sake of speed. Thanks!