Need advice to organize indexes

[Sergei Averichev]

Hello,

Starting to deploy Nagios Log Server and have few questions on better way to manage/organize indexes. Now having NLS in Hyper-V with total volume of index ~22GB per day.

Right now I just store indexes for few days on a default location. The main "greedy" log contributor is Firewall/IPS appliance with ~30 000 000 Docs per day. The task is to store its logs for 3 months with occasional retro analysis. While the "hot" regularly checked indexes are usually about one week old. There are other "lazy" log sources (WiFI, Linuxes, Captive Portal, Switches, Routers, VPN) with ~ 250 000 Docs per day. These logs are not that often checked and need to be stored for 1 month at most.

And so my questions are...

As far as I understand there is no option right now to store indexes for "greedy" and "lazy" sources on separate volumes? I.e. Flash/SSD and NL-SAS?

What filesystem is preferred for volume to store indexes on?

Can I separate "lazy" logs from indexes and delete them after 1 month while keeping "greedy" logs?

How should I organize archiving and rotating logs for our needs. Should I...
1) Keep indexes open for 7 days?
2) Close indexes older than 7 days, while keeping 'em?
3) Move closed indexes older that 14 days to archive (snapshot it)?
4) When I'll need to analyse some archived date - I'll need restore index for that day from archive (snapshot?), open it and than I can query it.

Or in case I have enough highperf Flash/SAS storage...
1) I'll close my indexes older ther 7 days and just keep'em all for 3 month, automatically deleting ones older than that? While freely opening ones needed for query?


As for over all backup - we use Veeam as corporate solution.

[jdunitz]

You've got the right idea here.

In HyperV, the limiting factor will be the underlying storage, unless you're allocating physical devices directly to the VM.

In general, ext4 and xfs should provide the best performance. I'd go with ext4, unless you have a specific preference.

You're right--all the logs go into one big heap, currently. So, all the management you'll do is just choosing how many days of indices to keep open, and that's based on what you typically need to go back and refer to. If you're doing reports and extracts that go back a week, keep a week open, then keep the closed ones for as long as you're likely to need to go back in the event of unexpected or occasional research.

Closed indices don't take up much in terms of system resources, other than disk space. When you open them back up, they'll consume more memory until you close them again. You can reopen them manually, or with an API call via curl rather easily.

Hope this helps!

--Jeffrey

[Sergei Averichev]

Hope this helps!

--Jeffrey

Hello,

Sure this information was quite helpful!

[benjaminsmith]

Hi,

Sure this information was quite helpful!

Great! Let us know if you need anything else.