idea: Declarative K8s API as the API for docker env.

[bwplotka]

Just an idea: But it would be amazing to contain some service like e2e.Runnable or instrumented e2e.Runnable in a declarative, mutable state. Ideally, something that speaks a common language like K8s APIs. Then have docker engine supporting an important subset of K8s API for local use. There would be a few benefits to this:

• We would be able to compose adjustments of e.g flags for different tests together better like Jsonnet allows (also adds huge cognitive load potentially!). The current approach has similar issues to https://github.com/bwplotka/mimic initial deployment at Improbable - the input for adjusting services is getting out of control (check ruler or querier helpers for e.g e2e: Refactored service helpers to be consistent with new API. thanos-io/thanos#5348)
• We could REUSE some Infrastructure as Go code (e.g. https://github.com/bwplotka/mimic) for both productions, staging, testing etc K8s clusters AS WELL AS local simplified e2e docker environments!

[hunjixin]

good idea , i am finding something like this.