Refactor rollups meta (AKA Rollup V2)

[polyfractal]

This is a meta issue discussing a complete rewrite of the Elasticsearch rollup codebase with the aim to improve the following points:

• Drop rollup jobs in favour of integrating rollups with ILM. This means that rolling up an index will work similarly to shrinking an index. The rollup will be done when indexing is complete and the action will rollup the entire index at the same time.
• Make rollup functionality easier to setup and administer from an operational point of view. For example, allow limited editing of existing rollup configuration (add new metrics etc)
• Make rollup indices behave much more like regular indices, simplifying querying and management.
• Improve reliability. Existing rollup jobs are not atomic and sometimes fail midway, leaving rollup indices that are not complete. We should make the rollup computation atomic.
• Improve performance of rollup jobs. Some large-scale use cases can run into bottlenecks where the search phase of rollup is not fast enough (due to limited thread involvement across cluster).
• Implement support for pre-aggregated data structures to enable cardinality, percentiles [Rollup] Support for data-structure based metrics (Cardinality, Percentiles, etc) #33214

Below we outline a high level plan of changes that will help us achieve the above goals:

• Remove the _rollup_search endpoint in favour of implementing search on rollup indexes within the _search endpoint ([RollupV2] Implement search resolution #67783) @csoulios
• Create an aggregate_metric field type that will store pre-aggregated metrics (min, max, sum etc) for a specific rollup group. (Implement aggregate_metric field mapper #49830)
• Modify existing metric aggregations so that they operate on the aggregate_metric fields. The aggregate_metric field type will provide the correct metric to the requesting aggregator, while using the same field name. This field type will not be indexed but stored as a binary doc values. (Implement aggregations on aggregate metrics #53986)
• Modify aggregators to pull doc count rather than just incrementing. Allows the aggregator framework to treat a single "rollup" document as if it were multiple "raw" documents. (Add doc_count field mapper #58339)
• Create a rollup_meta cluster metadata to store information such as date_histogram interval and timezone etc. (Add RollupV2 cluster metadata behind feature-flag #64680)
• Create new Rollup endpoint to rollup indices based on the new model (Adds a new Rollup Action #64900)
• Create an ILM rollup action and remove the rollup job functionality. issue, PR @talevy
• Refactor out all the Rollup-V2 mentions in the code-base in favor of the Rollup-Action issue @talevy
• [Docs] Rollup Documentation (meta tracking issue: [DOCS] Rollup refactor docs #65515)
• Fix muted tests that are flaky (Various RollupActionSingleNodeTests failing with IndexNotFoundException #69799)
• Add ability for doing rollups of rollups. This means supporting _doc_count field and aggregate_metric_double fieldtypes (Add RollupAction support for AggregateDoubleMetric fields #70534)
• Rollup Action Improvements and Testing
  • Action should ensure that relevant index-metadata/settings is copied to rollup index (hidden index?, tier?, ILM Execution state?)
  • Resolving simplification strategies (Simplify ILM Policy solution for managing lifecycle of rollup indices #70334)
  • (Maybe, depending on ILM Simplification) Add ability for Rollup ILM Action to optionally-delete original index upon rollup. This means atomically removing the original index and adding the rollup index into the datastream backing indices.
  • Rollup Action must throw exception when rollup index already exists issue
  • DateHistogramGroupConfig is re-using the original config with a delay option, need to make a new Action-specific config without a delay
  • add more config verification before executing the action
  • Add more thorough tests for various groupings in the indexer
  • Add Rally benchmark for rolling up indices
  • investigate Jim's new broadcast action gh compare link

[elasticmachine]

Pinging @elastic/es-analytics-geo

[exekias]

ey @polyfractal I see this concept of "Grouping Tuple" really interesting for the metrics use case. I think doesn't only apply to rollups, but can be used in some queries too!

It actually sounds familiar to what I did in Beats here: elastic/beats#10293, where we want to have enough information to do downsampling on query time to make sure results actually make sense. I wonder if we could reuse this grouping tuple field here?

[pcsanwald]

Great outline! I think the additional thing that would be good to think about and add to the outline, is what the migration path looks like for users currently using roll ups (I’m assuming that the new field type would happen during a rollup ILM action and therefore wouldn’t require a full re-index or source index?) and also what breaking changes we are proposing (at least _rollup_search removal).

@csoulios @polyfractal tagging y'all in case this got lost in the shuffle :)

[giladgal]

Regarding the relation with the Rollup functionality that is already released: Rollup is currently an experimental feature in Elasticsearch. We plan to make Rollup GA when we release the refactored rollup which will make rolled up indices queryable through _search and _async_search and which will make rollup an action in ILM. We are not making Rollup GA before that because we know this change will bring a change in the API. We will make an effort to provide an upgrade path for Rollup indices created with the current Rollup processes to the new refactored Rollup system, which will make these indices queryable through the _search and _async_search API endpoints.

[mbudge]

Will rollups handle new fields arriving in the data?

For example winlogbeat adds new fields when one appears in the windows event log.

Would this new field get rolled up automatically or would we have to manually modify the rollup job?

[ppf2]

The ability for rollup API to generate time-based indices that can be managed/retired over time is a common ask from the field. +1 on ILM integration.

[heoehmke]

Will the new rollup feature also support boolean fields in terms aggregations (as proposed in #49537)? The current rollups only support keyword and numeric fields and it doesn't seem too complicated to make using booleans possible there.

[csoulios]

The team has decided that there is a great fit between rollups and the new metrics database we are implementing (#74660).

The fact that dimensions and metrics are first class citizens allows us to simplify the rollup configuration. Also, leveraging the index structure in TSDB results in greatly improved performance.

The team views rollups as a fundamental operation of a metrics database and we focus all our efforts to develop such a feature. Hence, I am closing this issue and work on rollups/downsampling will be tracked inside #74660.