Support for partial override (variable substitutes) for CRD attributes

[senyan]

What would you like to be added:
Support for partial override for CRD attribute
Why is this needed:
for example if I have a federated deployment with an multi-cloud environment, we would like to use the docker registry of that local cloud with different url, it would be nice to be able to do
image: <registry>/app1:v1

where <registry> is a variable and being override while deploy to different federated clusters. So when I deploy one version of the container to one cluster, it is smart enough to substitute the while fan out the deployment to federated clusters.

/kind feature

[marun]

Overrides already allow per-cluster variation, just at the field level. For the use case you describe the entire image name would have to be provided for each cluster. Do you think the complexity of defining variables and a substitution mechanism are justified for such a use case, or is this use case a trivial example of something more involved?

[senyan]

I realized the fed-v2 already allow change the whole value with Json path which is useful. But I would not feel this is a trivial example, especially for people continuously roll out applications. For a fast pace team, I want to deploy and roll out dozens times a day. image version is the most frequent changed variables of the deployment. This use case is probably the biggest use case for service developers (roll out new service version).

If every time I have to make a call to change the deploymentoverrides config then change the deployment config, that would be really a bummer. Even getting harder with kubectl. Plus this is not something hard to implement， for a POC, it is easy enough to do something with a regex pattern like:

    clusterOverrides:
    - path: image
      pattern: ``<regex>``
      value: ``<some value>``

Certainly, for multi-cloud environments, there could be more involved use cases such as change annotations per release etc.

[senyan]

@marun Please let me know if this a good thing to add in your mind. I can code the initial POC. With more thought over the weekends, I think this is a bigger multi-cluster (or cloud) workflow type of thing. Good to talk about.

[marun]

@senyan I don't think variable substitution in overrides is something that fedv2 should support directly. There are many longstanding threads on this in Kubernetes, and I think for good reason efforts to incorporate it have been rejected time and again. There are many solutions outside of core Kube that do support variable substitution, and I encourage you to investigate them instead (e.g. helm, kustomize, envsubst, etc).

Separetely, are you aware that varying the image for a deployment with overrides is not supported today (let alone with variable substitution)?

[senyan]

@marun
I wouldn't fight hard for this particular solution (substitute by variable/regex), as I feel it is hacky as well, but wanna raise the issue that some capability of adding transformation logic is needed while the federation control plane propagate the kube object to federated clusters.

The exists of ClusterOverrides seems having the intent of solving this problem. But it has been insufficient. The solution could be some plugin capability built into the federation-controller-manager container like execute some binary like cni, webhooks etc.

So, all CRD types includes deployment can be customized per cluster. And more advanced multi-cluster workflow can be supported. Wonder if you or people from kubernetes-multicluster-sig would further explore this.

[marun]

The current API of fedv2 is intended to be low-level and not necessarily user-facing - it's intended to drive controllers. The integration point for higher-level functionality is intended to be the API rather than adding an explicit plugin mechanism to the controller manager. Higher-level functionality could be implemented as webhooks on the federation api types, or as new API types that could be acted upon by new controllers to generate the federated API types. As I alluded to in my previous comment, there are also a ton of tools outside the API (including ci/cd pipelines like Spinnaker) that could be used to customize how the federation resources were composed.

[senyan]

To be very direct, The current API of fedv2 is intended to be low-level and not necessarily user-facing this is a bit drop expectation on my stand point. I understand the CRD in API will drive controllers, and I am expecting the controller will do a transformation based on my specification or business logic.

I know there are other cicd tools could do the job, but that defeats the purpose of fedv2 in my opinion. The attraction part of federation for us is that tools works for single cluster could transparently work for multiple clusters at least for common objects like Deployment. Certainly, this could conflict with
your current vision about fedv2. I would just see if anything will change in the near future.

[marun]

Tools that work for a single cluster do not transparently work across multiple clusters, at least beyond extremely limited use cases. Coordinating configuration across multiple clusters is simply a hard problem.

In fedv2, each Kubernetes type (e.g. Deployment) has a corresponding federated type (e.g. FederatedDeployment). The translation between these will be relatively trivial, but the API is intentionally different. That wasn't true for fedv1, but the lack of differentiation of the fedv1 APIs proved a considerable barrier to supportability and productization.

The current state of fedv2 is tightly focused on defining and propagating configuration across multiple cluster, experimenting with scheduling of propagation for workload types, and supporting multicluster services and ingress via dns. That doesn't preclude adding additional layers or integrating with other solutions, and I encourage you to continuing engaging with the community to better understand where our thinking is, how we got there, and where we hope to go next.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.