Setting buildKit: inCluster: overrides localRegistry config

[jmeickle-theaiinstitute]

What happened?

When an inCluster setting is provided, the localRegistry config is totally ignored.

This is related to, but not the same issue, as: #2700

What did you expect to happen instead?

I want to customize how the buildkit builder pod is created (due to needing a nodeSelector on my cluster), but also be able to push images to an in-cluster local registry (due to cred requirements)

How can we reproduce the bug? (as minimally and precisely as possible)

This snippet will use the Buildkit Kubernetes driver to launch a builder pod (including custom node selector). It will pass in the secret successfully and build the image. However, using the driver means it will not start a local registry, and it will try to push to a non-existent foo registry (on Dockerhub):

localRegistry:
  enabled: true
images:
  foo:
    image: foo
    buildKit:
      inCluster:
        nodeSelector: custom=true
      args: ["--secret", "id=MYSECRET"]

This snippet will use the local registry pod's Buildkit builder (which doesn't have config options to specify a node selector). It will not pass in the secret successfully, due to #2700 overwriting the buildKit: args:, and will fail to build the image requiring it.

localRegistry:
  enabled: true
images:
  foo:
    image: foo
    buildKit:
      args: ["--secret", "id=MYSECRET"]

This snippet (not specifying localRegistry at all) will do the same as above too (due to defaulting to using the local registry when unset).

images:
  foo:
    image: foo
    buildKit:
      args: ["--secret", "id=MYSECRET"]

Local Environment:

• DevSpace Version: 6.3.12
• Operating System: linux
• ARCH of the OS: AMD64
  Kubernetes Cluster:
• Cloud Provider: google
• Kubernetes Version:
  Client Version: v1.29.4
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.29.4-gke.1670000

Anything else we need to know?

It would be preferable to provide a way to patch any registry/builder pods before they're created, such as via kustomize. There are too many possible options in a buildkit and/or registry manifest to make adding args for all of them feasible. But at the same time, I don't want to have to write a from-scratch tool to provision appropriate buildkit builders (and there are some devspace internals for overriding where to push, which wouldn't be easy/possible to modify that way)