Kaniko returns chown: unknown user/group during FluentD container build

[samuelebistoletti]

Actual behavior
Kaniko returns "chown: unknown user/group fluent:fluent" during FluentD container build with the following command in Dockerfile:

RUN mkdir -p /fluentd/etc/certs \
  && mkdir -p /var/log/fluentd \
  && chown -R fluent:fluent /var/log/fluentd

Expected behavior
Container build is expected to complete successfully without errors, like in Docker environment.

To Reproduce
Steps to reproduce the behavior:

Just try to build the container with the following command or with the GitLab pipeline attached in the "Additional information" section. Variable values are specified in the GitLab pipeline.

/kaniko/executor
      --context $KANIKO_BUILD_CONTEXT
      --dockerfile $DOCKERFILE_PATH
      --destination $IMAGE_TAG $KANIKO_ARGS
      --build-arg CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX=$CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX
      --build-arg FLUENTD_TAG=$FLUENTD_TAG
      --build-arg FLUENTD_CONFIG_ENV=$FLUENTD_CONFIG_ENV

Additional Information

User and group exists in FluentD original image. Also if I try to run the build pipeline with a Docker in Docker executable no error is raised.

Dockerfile

ARG CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX
ARG FLUENTD_TAG

FROM $CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX/fluent/fluentd:$FLUENTD_TAG

ARG FLUENTD_CONFIG_ENV=prod

ENV TZ=Europe/Rome

USER root

RUN apk add --no-cache --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && apk add --no-cache --update tzdata \
 && sudo gem install fluent-plugin-opensearch \
 && sudo gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

RUN mkdir -p /fluentd/etc/certs \
  && mkdir -p /var/log/fluentd \
  && chown -R fluent:fluent /var/log/fluentd

COPY fluentd/entrypoint.sh /bin/

USER fluent

Context

Dockerfile is built in a GitLab Pipeline with the following script:

default:
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]

variables:
  GIT_STRATEGY: clone
  CI_REPOSITORY_URL: ***
  FLUENTD_BASE_IMAGE_TAG: "v1.14-1"

stages:
  - ci-lint
  - build
  - deploy

build-fluentd-image:
  stage: build
  variables:
    # Additional options for Kaniko executor.
    # For more details see https://github.com/GoogleContainerTools/kaniko/blob/master/README.md#additional-flags
    KANIKO_ARGS: ""
    KANIKO_BUILD_CONTEXT: $CI_PROJECT_DIR/fluentd
  script:
    # If the user provide IMAGE_TAG then use it, else build the image tag using the default logic.
    # Default logic
    # Compose docker tag name
    # Git Branch/Tag to Docker Image Tag Mapping
    #   * Default Branch: main -> latest
    #   * Branch: feature/my-feature -> branch-feature-my-feature
    #   * Tag: v1.0.0/beta2 -> v1.0.0-beta2
    - |
      if [ -z ${IMAGE_TAG+x} ]; then
        if [ "$CI_COMMIT_REF_NAME" = $CI_DEFAULT_BRANCH ]; then
          VERSION="latest"
          FLUENTD_CONFIG_ENV=prod
        elif [ -n "$CI_COMMIT_TAG" ];then
          NOSLASH=$(echo "$CI_COMMIT_TAG" | tr -s / - )
          SANITIZED="${NOSLASH//[^a-zA-Z0-9\-\.]/}"
          VERSION="$SANITIZED"
          FLUENTD_CONFIG_ENV=prod
        else \
          NOSLASH=$(echo "$CI_COMMIT_REF_NAME" | tr -s / - )
          SANITIZED="${NOSLASH//[^a-zA-Z0-9\-]/}"
          VERSION="branch-$SANITIZED"
          FLUENTD_CONFIG_ENV=dev
        fi
        export IMAGE_TAG=$CI_REGISTRY_IMAGE/fluentd:$VERSION
        export FLUENTD_CONFIG_ENV
      fi
    - echo $IMAGE_TAG
    - mkdir -p /kaniko/.docker
    # Write credentials to access Gitlab Container Registry within the runner/ci
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"},\"$CI_DEPENDENCY_PROXY_SERVER\":{\"auth\":\"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    # Build and push the container. To disable push add --no-push
    - DOCKERFILE_PATH=${DOCKERFILE_PATH:-"$KANIKO_BUILD_CONTEXT/Dockerfile"}
    - >-
      /kaniko/executor
      --context $KANIKO_BUILD_CONTEXT
      --dockerfile $DOCKERFILE_PATH
      --destination $IMAGE_TAG $KANIKO_ARGS
      --build-arg CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX=$CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX
      --build-arg FLUENTD_TAG=$FLUENTD_TAG
      --build-arg FLUENTD_CONFIG_ENV=$FLUENTD_CONFIG_ENV
  # Run this job in a branch/tag where a Dockerfile exists
  rules:
    - exists:
        - Dockerfile
    # custom Dockerfile path
    - if: $DOCKERFILE_PATH
    # custom build context without an explicit Dockerfile path
    - if: $KANIKO_BUILD_CONTEXT != $CI_PROJECT_DIR
  tags:
    - docker

Kaniko Image (fully qualified with digest)

Using docker image sha256:e1e8f684a8d2023df1c21f06f2a66f7ec7ef965e6d092c7efe00ca6d592b2e6b for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:3bc3f3a05f803cac29164ce12617a7be64931748c944f6c419565f500b65e8db ...

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[hown3d]

This is the same issue as #1456.

I looked into fluent/fluentd:latest image and it doesn't create the user fluent:

Output from /etc/passwd and /etc/group

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/usr/lib/news:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
operator:x:11:0:operator:/root:/bin/sh
man:x:13:15:man:/usr/man:/sbin/nologin
postmaster:x:14:12:postmaster:/var/spool/mail:/sbin/nologin
cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
ftp:x:21:21::/var/lib/ftp:/sbin/nologin
sshd:x:22:22:sshd:/dev/null:/sbin/nologin
at:x:25:25:at:/var/spool/cron/atjobs:/sbin/nologin
squid:x:31:31:Squid:/var/cache/squid:/sbin/nologin
xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin
games:x:35:35:games:/usr/games:/sbin/nologin
postgres:x:70:70::/var/lib/postgresql:/bin/sh
cyrus:x:85:12::/usr/cyrus:/sbin/nologin
vpopmail:x:89:89::/var/vpopmail:/sbin/nologin
ntp:x:123:123:NTP:/var/empty:/sbin/nologin
smmsp:x:209:209:smmsp:/var/spool/mqueue:/sbin/nologin
guest:x:405:100:guest:/dev/null:/sbin/nologin
nobody:x:65534:65534:nobody:/:/sbin/nologin

$ cat /etc/group
root:x:0:root
bin:x:1:root,bin,daemon
daemon:x:2:root,bin,daemon
sys:x:3:root,bin,adm
adm:x:4:root,adm,daemon
tty:x:5:
disk:x:6:root,adm
lp:x:7:lp
mem:x:8:
kmem:x:9:
wheel:x:10:root
floppy:x:11:root
mail:x:12:mail
news:x:13:news
uucp:x:14:uucp
man:x:15:man
cron:x:16:cron
console:x:17:
audio:x:18:
cdrom:x:19:
dialout:x:20:root
ftp:x:21:
sshd:x:22:
input:x:23:
at:x:25:at
tape:x:26:root
video:x:27:root
netdev:x:28:
readproc:x:30:
squid:x:31:squid
xfs:x:33:xfs
kvm:x:34:kvm
games:x:35:
shadow:x:42:
postgres:x:70:
cdrw:x:80:
usb:x:85:
vpopmail:x:89:
users:x:100:games
ntp:x:123:
nofiles:x:200:
smmsp:x:209:smmsp
locate:x:245:
abuild:x:300:
utmp:x:406:
ping:x:999:
nogroup:x:65533:
nobody:x:65534:

This should be fixed once #2106 is merged.

[samuelebistoletti]

Hi @hown3d !

fluent/fluentd:latest image is very old, this tag was last updated 4 years ago, don't know the reason:
https://hub.docker.com/layers/fluentd/fluent/fluentd/latest/images/sha256-7eece00d1bc784ac1e9722b2580911cd3ead5afd740dad6594be945b3b1dd884?context=explore

I'm using version v1.14-1 (Docker tag is v1.14-1) and user fluent is ok inside the container:

/ $ cat /etc/passwd
root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/mail:/sbin/nologin
news:x:9:13:news:/usr/lib/news:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
man:x:13:15:man:/usr/man:/sbin/nologin
postmaster:x:14:12:postmaster:/var/mail:/sbin/nologin
cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
ftp:x:21:21::/var/lib/ftp:/sbin/nologin
sshd:x:22:22:sshd:/dev/null:/sbin/nologin
at:x:25:25:at:/var/spool/cron/atjobs:/sbin/nologin
squid:x:31:31:Squid:/var/cache/squid:/sbin/nologin
xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin
games:x:35:35:games:/usr/games:/sbin/nologin
cyrus:x:85:12::/usr/cyrus:/sbin/nologin
vpopmail:x:89:89::/var/vpopmail:/sbin/nologin
ntp:x:123:123:NTP:/var/empty:/sbin/nologin
smmsp:x:209:209:smmsp:/var/spool/mqueue:/sbin/nologin
guest:x:405:100:guest:/dev/null:/sbin/nologin
nobody:x:65534:65534:nobody:/:/sbin/nologin
fluent:x:100:101:Linux User,,,:/home/fluent:/sbin/nologin

Here is the base image from which I'm starting the customization:
https://github.com/fluent/fluentd-docker-image/blob/master/v1.14/alpine/Dockerfile

[samuelebistoletti]

Issue solved, there was a problem with the GitLab pipeline that isn't passing the correct value to the --build-arg that sets the tag on FluentD image inside dockerfile. So by default fluentd was using the tag "latest" that was last updated 4 years ago and has not the user fluent inside. Passing the correct build arg with tag v1.14-1 for Fluentd solved the issue.