Log Server Maintenance Clarification?

[lazzarinof]

Good afternoon all,

I'm trying to ensure we have the optimal maintenance schedule for our log servers. I know this will vary greatly by organization, but I'm hoping clarification on the primary automated maintenance could help.

Optimize Indexes older than:
This will use a Lucene forceMerge on an index that will not accept or ingest any new data.
Set this to 0 to disable this functionality.
Close indexes older than:
Marks indexes older that this value as closed
Closed indexes do not take any system resources other than disk space, however they cannot be
searched unless re-opened
Set to 0 to disable.
Delete indexes older than:
Deletes indexes older than this value, freeing resources
This is permanent, the only way to restore a deleted index is from an archived snapshot
Set to 0 to disable

For these three tasks: I'll have to admit I'm currently completely in the dark on the use case for optimizing indexes. The environment I'm currently in has it set to 0. I'm trying to find a reason for forceMerge, but can't.
We currently close and delete indexes older than 22 days. Is there any reason to close prior to deleting the index (like, close indexes on the 21st day, delete on the 22nd?). If closing and deleting at the same time, is there any reason to close them at all?

Thank you for any possible clarification!
-Frank

[ssax]

Optimize:

See here for what a forcemerge does in the optimize process:

https://www.elastic.co/guide/en/elastic ... merge.html

This has a memory/cpu impact while optimizing, your system will vary based on system load, index size, number of nodes, number of queries/filters/alerts setup and how complex they are, etc.

Close:

Indices in the closed state have no memory impact but still consumes storage

Delete:

Deleted indices have no memory or storage impact

In order to properly utilize the settings you'll want to setup snapshots to backup your data:

https://assets.nagios.com/downloads/nag ... enance.pdf

I usually recommend:

*** The intended goal here is to keep your system as lean as possible within your business data policy requirements (if any) so that your system performs as expected as you grow/time progresses, rough estimates can be calculated based on total system memory, total system storage, and average daily index size

Optimize: 2 (the default)

Close: 7 - Because of memory footprint, your setting will differ based on total system memory, index size, number of nodes, total queries/filters/alerts running against system, etc. You can reopen if you need to query them.

Delete: 30 - Because of storage requirements, your setting will differ based on index size/total available disk space, you can restore specific indices from snapshot at any time to automatically import and reopen the indices for querying.

Snapshots repository setup off-server (NFS/CIFs mount).

*** Only enable Delete after snapshots have been validated as working.

[lazzarinof]

Awesome, thank you very much ssax!