Regression since v0.19.0 using --cache=true with shell parameter expansion in RUN instruction of Dockerfile

[tanguydelignieresaccenture]

Actual behavior
Starting with debug-v0.19.0 (reproduced with debug-v1.0.0), building using --cache=true, a Dockerfile having RUN instruction with ENV or ARG variable and shell parameter expansion is not working anymore with the following error:
error building image: error building stage: failed to optimize instructions: failed to process "RUN : \"${BINARY_NAME:?Build argument needs to be set and non-empty.}\"": unsupported modifier (?) in substitution

$ docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200825-001/build1:/workspace gcr.io/kaniko-project/executor:debug-v0.19.0 /kaniko/executor --cache=true --registry-mirror index.docker.io --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200925-001_build1:debug-v0.19.0
INFO[0002] Resolved base name alpine:3.9 to alpine:3.9
INFO[0002] Resolved base name alpine:3.9 to alpine:3.9
INFO[0002] Retrieving image manifest alpine:3.9
INFO[0004] Retrieving image manifest alpine:3.9
INFO[0006] Built cross stage deps: map[]
INFO[0006] Retrieving image manifest alpine:3.9
INFO[0008] Retrieving image manifest alpine:3.9
error building image: error building stage: failed to optimize instructions: failed to process "RUN : \"${BINARY_NAME:?Build argument needs to be set and non-empty.}\"": unsupported modifier (?) in substitution

$

It might be related to #1085

With debug-v0.18.0, building using --cache=true, a Dockerfile having RUN instruction with ENV or ARG variable and shell parameter expansion is working:

(Do not pay attention to the --registry-mirror flag, it's only for dockerhub hostname resolution -- see #1209, nor the cache layer push error at the end, they do not prevent test the case or reproduce the issue)

$ docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200825-001/build1:/workspace gcr.io/kaniko-project/executor:debug-v0.18.0 /kaniko/executor --cache=true --registry-mirror index.docker.io --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200925-001_build1:debug-v0.18.0
INFO[0002] Resolved base name alpine:3.9 to alpine:3.9
INFO[0002] Resolved base name alpine:3.9 to alpine:3.9
INFO[0002] Retrieving image manifest alpine:3.9
INFO[0005] Retrieving image manifest alpine:3.9
INFO[0008] Built cross stage deps: map[]
INFO[0008] Retrieving image manifest alpine:3.9
INFO[0010] Retrieving image manifest alpine:3.9
INFO[0011] Checking for cached layer index.docker.io/tanguydelignieres/kaniko_bugs_20200925-001_build1/cache:f76715517b6fb892b2ff84dd41aa7d430b9c14dd18db39815391c649fac302e0...
INFO[0014] No cached layer found for cmd RUN : "${BINARY_NAME:?Build argument needs to be set and non-empty.}"
INFO[0014] Unpacking rootfs as cmd RUN : "${BINARY_NAME:?Build argument needs to be set and non-empty.}" requires it.
INFO[0018] Taking snapshot of full filesystem...
INFO[0018] Resolving paths
INFO[0018] ENV BINARY_NAME=abinaryname
INFO[0018] No files changed in this command, skipping snapshotting.
INFO[0018] RUN : "${BINARY_NAME:?Build argument needs to be set and non-empty.}"
INFO[0018] cmd: /bin/sh
INFO[0018] args: [-c : "${BINARY_NAME:?Build argument needs to be set and non-empty.}"]
INFO[0018] Taking snapshot of full filesystem...
INFO[0018] Resolving paths
INFO[0018] No files were changed, appending empty layer to config. No layer added to image.
INFO[0018] Pushing layer index.docker.io/tanguydelignieres/kaniko_bugs_20200925-001_build1/cache:f76715517b6fb892b2ff84dd41aa7d430b9c14dd18db39815391c649fac302e0 to cache now
WARN[0024] error uploading layer to cache: failed to push to destination index.docker.io/tanguydelignieres/kaniko_bugs_20200925-001_build1/cache:f76715517b6fb892b2ff84dd41aa7d430b9c14dd18db39815391c649fac302e0: unsupported status code 401

$

Expected behavior
With >= debug-v0.19.0 building using --cache=true, a Dockerfile having RUN instruction with ENV or ARG variable and shell parameter expansion should still work as in <= debug-v0.18.0.

To Reproduce

1. Use a Dockerfile with a RUN instruction with ENV or ARG variable and shell parameter expansion
2. Build using kaniko executor >= debug-v0.19.0, with --cache=true

Additional Information

• Dockerfile

FROM alpine:3.9

ENV BINARY_NAME=abinaryname

# Check for mandatory build arguments
RUN : "${BINARY_NAME:?Build argument needs to be set and non-empty.}"

• Build Context
  No specific Build Context
• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor:debug-v0.18.0 sha256:fee68fd4aacd3b1efea1d57d04426baf14e8961db41c4aa95ba47f7fa8e45fa5
gcr.io/kaniko-project/executor:debug-v0.19.0 sha256:0d0e34396f47ec6d5fd75aebb9772147a78d96ed2bbb16ec892bd178efdc8307

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[schmitch]

actually we run into this aswell, we use a multilayer image and our build image needs two args (NPMRC credentials and NUGET credentials)

[tanguydelignieresaccenture]

FYI, reproduced on debug-v1.3.0

[yellowhat]

Hi, I am a getting the same issue. Is this still relevant?

[ukd1]

Ya, also getting this. Is the project dead?