[Transform] separate pivot and extract function interface

[hendrikmuhs]

separate pivot from the indexer and introduce an abstraction layer, pivot becomes a function. This opens
the possibility to add more functions to transform.

cleaned up version of #55287 after removing experimental code

piggy backed fixes:

• when running geo tile group_by it could fail due to query clause limit (unreleased)
• new style page size using settings was not validating limit of 10k (7.8)

Review hints

• the most important abstraction is the Function interface
• change collection - the essential part for continuous transform - has been abstracted as Function.ChangeCollector, all pivot related parts are now in CompositeBucketsChangeCollector

ToDo's

Must

• abstract progress gatherer

Found problems not introduced by this PR

• update should validate ingest pipeline
• GeoTileFieldCollector can not run with pagesize > 1024
• validateConfig/Query should use ValidationException

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)

[benwtrent]

I will have to read through it a second time.

The separation of "function" from "data" gives us flexibility. But, it seems we might be sacrificing polymorphism. I am unsure the best way to reconcile this right now as it is designed.

[davidkyle]

👍 I like the redesign it makes the code cleaner

[benwtrent]

I don't know what we are gaining by not having SingleGroupSource have a createFieldCollector method. We would avoid this switch statement.

Are we just trying to keep from adding classes to core?

[hendrikmuhs]

To me it seems unavoidable, config and implementation should be separate, like PivotConfig and Pivot. so far it was simple enough to put in into ...GroupSource (which was totally ok). With the re-factoring it gets more complicated and there are already plans, that will add more code (maybe one day we end up with 1 class per GroupSource).

Yes keeping code away from core is a reason. If I would add it the implementation to ...GroupSource, I would not only need to move the code there, but also the collector interface.

I think this change is similar to what is done in other places. The boiler-plate (switch) isn't great, but I do not see a need for something registry-style (like aggregations), at least at the moment.

[davidkyle]

LGTM

👍 nice change