When does Elasticsearch stop writing to an index?

[li_alm]

Hello,

I need to write a script that sends queries directly to elasticsearch and I need to process the whole previous day.
Example:
today = 06.10.2017
yesterday = 05.10.2017
yesterday index = logstash-2017.10.05

At what (minimum) time is it fine to run the script (on 06.10.2017) so that I am sure that elasticsearch stops writing to logstash-2017.10.05 (and begins writing to logstash-2017-10-06)?

Thank you.
Regards,
Liviu

[mcapra]

In the context of Nagios Log Server, ElasticSearch isn't responsible for "writing" indices; It's just a database that holds stuff.

Logstash is what writes events to ElasticSearch. When Logstash stops writing to an index is when the day rolls over and a new index is created (again, all done by Logstash)

At what (minimum) time is it fine to run the script (on 06.10.2017) so that I am sure that elasticsearch stops writing to logstash-2017.10.05 (and begins writing to logstash-2017-10-06)?

So if you wanted to write to logstash-2017.10.06, a safe time to do that might be when logstash-2017.10.07 is created. Just have your script check for when the next day's index is created, then process the previous day.

[li_alm]

First of all, sorry for the confusion. Yes, ES is the database, and Logstash writes data to ES.

Q: "when the day rolls over" - so when the day changes on the machine nagios is running on, I can safely say logstash will no longer write to the "logstash-<previous day>" index?
So I can run my script at 01h:00 AM, on 06.10.2017, to check the entire logs on 05.10.2017=logstash-2017.10.05.

Thanks.
Liviu

[scottwilkerson]

It needs to be pointed out that there is no defined cut-off that make an index unwritable.

The index logs go in depends on what the timestamp is in the log.

For example, if you choose to import logs that contain data from October 1 you can do that and they will add to logstash-2017.10.01

[li_alm]

Wow, thanks.
So you're saying if today=06.10.2017 and ES receives from logstash a message timestamped=04.10.2017, this message will be written into logstash-2017.10.04?
So i'ts actually the timestamp that matters, not the time running on the machine nagios is installed.

Liviu

[scottwilkerson]

Wow, thanks.
So you're saying if today=06.10.2017 and ES receives from logstash a message timestamped=04.10.2017, this message will be written into logstash-2017.10.04?
So i'ts actually the timestamp that matters, not the time running on the machine nagios is installed.

Liviu

Correct, if the log line has a timestamp or date that gets parsed. If the log line doesn't, it will set the date to now and go in today's index.

[li_alm]

Yes, the timestamp in the message received by logstash is used as the timestamp in the nagios system.

match => { "message" => ...(?:%{TIMESTAMP_ISO8601:syslog_ts}|-)...
date {
match => [ "syslog_ts", "ISO8601" ]
}

[scottwilkerson]

that would do it