Images.buildKit.Args get ignored while using localRegistry

[ArcticXWolf]

What happened?
I want to add specific buildKit options (--secret ...) to the image build process, but also deploy to the local registry.
However, enabling the localRegistry weirdly overwrites the complete buildKit stanza in the devspace.yaml and no args are being applied.

What did you expect to happen instead?
That devspace applies images.[imagename].buildKit.args even when enabling localRegistry.

How can we reproduce the bug? (as minimally and precisely as possible)
Create the following three files in a new folder:

Dockerfile

FROM ubuntu

RUN --mount=type=secret,id=mysecret,dst=/secretfile cat /secretfile

secretfile

THIS IS A SECRET

devspace.yaml

version: v2beta1
name: test

vars:
  IMAGE: testimage
  DOCKERFILE: "./Dockerfile"

images:
  testimage:
    image: ${IMAGE}
    dockerfile: "${DOCKERFILE}"
    rebuildStrategy: ignoreContextChanges
    buildKit:
      args:
        - "--progress=plain"
        - "--secret"
        - "id=mysecret,src=./secretfile"

deployments: {}
dev: {}
localRegistry:
  enabled: true

Output on devspace build:

❯ devspace build --debug
12:01:38 info Using namespace 'default'
12:01:38 info Using kube context 'testcluster'
12:01:38 debug Use config:
version: v2beta1
name: test
images:
    testimage:
        name: testimage
        image: testimage
        dockerfile: ./Dockerfile
        rebuildStrategy: ignoreContextChanges
        buildKit:
            args:
                - --progress=plain
                - --no-cache
                - --secret
                - id=mysecret,src=./secretfile
localRegistry:
    enabled: true


12:01:38 debug Run pipeline:
name: build
run: |-
    run_dependencies --all --pipeline build
    build_images --all


12:01:38 run_dependencies --all --pipeline build
12:01:38 Marked project excluded: test
12:01:38 build_images --all
12:01:38 Ensuring image pull secret for registry: hub.docker.com...
12:01:38 Couldn't retrieve username for registry  from docker store
12:01:38 Couldn't retrieve password for registry  from docker store
12:01:39 local-registry: Starting Local Image Registry
12:01:39 local-registry: Namespace default is the default Devspace namespace
12:01:39 local-registry: Wait for local registry node port to be assigned...
12:01:39 local-registry: Check for running local registry
12:01:39 local-registry: Wait for running local registry pod...
12:01:40 build:testimage Rebuild image testimage because tag is missing
12:01:40 build:testimage Building image 'testimage:UIBCLmB' with engine 'localregistry'
12:01:40 build:testimage Sending build context to Docker daemon  4.096kB
12:01:40 build:testimage #1 [internal] load remote build context
12:01:40 build:testimage #1 DONE 0.0s
12:01:40 build:testimage 
12:01:40 build:testimage #2 copy /context /
12:01:40 build:testimage #2 DONE 0.0s
12:01:40 build:testimage 
12:01:40 build:testimage #3 [internal] load metadata for docker.io/library/ubuntu:latest
12:01:41 build:testimage #3 DONE 0.9s
12:01:41 build:testimage 
12:01:41 build:testimage #4 [stage-0 1/2] FROM docker.io/library/ubuntu@sha256:ec050c32e4a6085b423d36ecd025c0d3ff00c38ab93a3d71a460ff1c44fa6d77
12:01:41 build:testimage #4 resolve docker.io/library/ubuntu@sha256:ec050c32e4a6085b423d36ecd025c0d3ff00c38ab93a3d71a460ff1c44fa6d77 done
12:01:41 build:testimage #4 CACHED
12:01:41 build:testimage 
12:01:41 build:testimage #5 [stage-0 2/2] RUN --mount=type=secret,id=mysecret,dst=/secretfile cat /secretfile
12:01:41 build:testimage #0 0.053 cat: /secretfile: No such file or directory
12:01:41 build:testimage #5 ERROR: process "/bin/sh -c cat /secretfile" did not complete successfully: exit code: 1
12:01:41 build_images: build images: error building image localhost:30211/testimage:UIBCLmB: failed to solve: process "/bin/sh -c cat /secretfile" did not complete successfully: exit code: 1
12:01:41 fatal exit status 

Local Environment:

• DevSpace Version: 6.3.2
• Operating System: linux
• ARCH of the OS: AMD64 | ARM64 | i386
  Kubernetes Cluster:
• Cloud Provider: other
• Kubernetes Version: 1.24.2 (irrelevant since this issue is not interacting with kubernetes)

Anything else we need to know?
I have already debugged the issue in devspace code. The reason is that when you enable the localRegistry, then unintuitively a different docker builder is being used (localregistry vs buildkit). The localregistry builder also uses buildkit (for online builds) or docker (for local builds), but does not reuse the code from the real buildkit or docker builder.

Thus the localregistry builder does not have any access to the images.[imagename].buildKit.args config parameters and cannot apply those to the build. This is also the same when setting localRegistry.localbuild=true.

The problem is: We need a local build with buildkit (which works when disabling localRegistry) AND need to push the image to the local registry. But currently you cannot use both together.

My proposal (but since I do not know the devspace code well you might have different opinions/reasons) would be to isolate the build and push parts of the devspace build pipeline, so any builder can be used with localRegistry. This also makes localRegistry more DRY, since you do not implement the full docker build pipeline there AND in the docker/buildKit builder.

Also: is there a workaround to already use this now? (creating a new pipeline or else)

[ArcticXWolf]

Just to clarify our usecase:
We need to build an image which uses python and one of the pip packages is inside a private pipy-index. So we need to include the credentials for this during the pip install step of the build. However copying or setting the credentials via envs will leak them in the image layer history. Thus we want to use the intended mechanism of buildkit secrets.

So we need devspace to:

• Use the local buildkit builder (otherwise the builder cannot access local files)
• Use the buildkit args that we set in devspace.yaml
• Push the image to the local registry

All three together are currently not possible, because the upload to local registry is bound to its own set of builders (docker or remote buildkit).

[lizardruss]

Hello! Thanks for submitting an issue. This is something we will work on enabling.

[jmeickle-theaiinstitute]

also ran into this and spent a lot of time against it before I realized what was happening