Log repository archival searches

[rferebee]

Good morning Nagios team,

Would you ever consider creating a secondary application that would allow Log Server administrators to perform archival searches of non-live logs? For example, currently due to storage restrictions, we're only able to keep roughly 30 days of live logs to perform searches against. Anything older needs to manually be reloaded into the console. We retain 2 years of logs per our agency policy, so if we need to go back say 18 months it creates a lot of overhead work for the Log Server admin team and in most cases can be a slow process.

I know Exchange has a mail archiver that allows for rapid search only processes against e-mail databases. I was wondering if you have ever considered implementing something like that for Log Server. A "search only" console where all the resources are devoted entirely to performing searches against a log repository.

Maybe something like that already exists? I don't know, but it would be really cool to have if it doesn't.

[mcapra]

This is doable with rollups:
https://www.elastic.co/guide/en/elastic ... ollup.html
https://www.elastic.co/guide/en/elastic ... earch.html

Which could at least give you a "reasonably good guess" as to which indices you needed to open to get the full data you're looking for.

Though that (and ILM) is new as of Elasticsearch 6.x IIRC. Not sure which ES version NLS is running these days.

[rferebee]

This is doable with rollups:
https://www.elastic.co/guide/en/elastic ... ollup.html
https://www.elastic.co/guide/en/elastic ... earch.html

Which could at least give you a "reasonably good guess" as to which indices you needed to open to get the full data you're looking for.

Though that (and ILM) is new as of Elasticsearch 6.x IIRC. Not sure which ES version NLS is running these days.

I appreciate your reply, however, being a State entity we would be unable to implement any solution considered "experimental" as our data and retention policies are scrutinized very heavily. Also, I was hoping the solution whether it exists or not could be more user friendly, perhaps a dashboard (like Log Server) where anyone needing to run a query could do so.

[cdienger]

It doesn't exist currently with NLS so I will file a feature request. Thanks for sharing your input.

[rferebee]

Yes, a feature request would be greatly appreciated. I look forward to what you folks come up with.

Thank you!

[cdienger]

Submitted.