how to filter for latest events of an event chain?

[_asp_]

Hi,

during the processing of an order, the order goes through multiple status. The order is identified by an id.
For example we have status_pending, status_in_process_1, status_in_process_2, status_end_success, status_end_failure.

id, current_status and and other fields are parsed from the logs and inserted into logserver as fields.

When I filter for an order_id, then I would find multiple events, e.g.:

order_1, status_start
order_1, status_in_process_1
order_2, status_start
order_3, status_start
order_3, status_in_process_1
order_3, status_in_process_2
order_3, status_end_success
order_4, status_start
order_4, status_in_process_1
order_4, status_in_process_2
order_4, status_end_failure

Now I would like to create a pie chart, where I only see the latest status of an order and push them into a pie chart for example. Only the bold marked entries above should be taken into account.

How can I do that?

Thanks, Andreas

[hsmith]

This is a pretty interesting request. Does the order part of the script ever change? Or is it always something like order, order2, or does it keep going up... like order23123102, order23123103...

I'm trying to figure out if this is doable.

[_asp_]

since we are processing in parallel the order_id's are not in growing order.
so they can also occur in following order:

order_1, status_start
order_2, status_start
order_1, status_in_process_1
order_3, status_start
order_4, status_start
order_4, status_in_process_1
order_3, status_in_process_1
order_3, status_in_process_2
order_4, status_in_process_2
order_4, status_end_failure
order_3, status_end_success

only the bold are needed for analysis.

is it for example possible during logstash filter to query elasticsearch and remove tags from events which are already inside elasticsearch?
Then if i process order_id x I could search for all other entries with order_id x and tag "last_entry". I would remove the tag and add it to the event i am currently processing.

But an option in pure postprocessing would also be nice because we would be more flexible here

[hsmith]

I think it would be too resource intensive if you we're querying elasticsearch each time a log came in. A post-processing option would be good. You've given me a lot of ideas for good feature requests. You have some pretty unique requests, but I agree with them.