An updated multi-arch image is pulled with wrong architecture

[TrueXakeP]

Describe the bug
As I described in #1287 (comment) Watchtower fetches the wrong architecture of multi-arch images than the container is configured.

To Reproduce
docker-compose.yml:

version: "3.4"
services:
  httpd-base:
    image: httpd:2.4
    platform: linux/i386
    restart: "no"
    entrypoint:
      - sh
      - -c
      - exit
    profile:
      - pull-only
    labels:
      com.centurylinklabs.watchtower.scope: pull

docker pull httpd:2.4.53
docker image tag httpd:2.4.53 httpd:2.4
docker compose --profile pull-only up --no-start --no-deps --force-recreate httpd-base
# [+] Running 1/0
# ⠿ Container 42ffb5434485_42ffb5434485_1c-in-docker-httpd-base-1  Recreated

docker image inspect httpd:2.4 -f "{{.Architecture}}"
# 386

Container was recreated after previous experiments.

Run the update:

docker run --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /etc/localtime:/etc/localtime:ro \
-v /etc/timezone:/etc/timezone:ro \
containrrr/watchtower \
--scope pull \
--include-stopped \
--debug \
--run-once \
1c-in-docker-httpd-base-1

And this is an error:

docker image inspect httpd:2.4 -f "{{.Architecture}}"
# amd64

docker image inspect httpd:2.4.53 -f "{{.Architecture}}"
# 386

docker compose up --no-start --no-deps httpd-base
# [+] Running 0/0
#  ⠋ Container 1c-in-docker-httpd-base-1  Recreate                                                                                                                                                                                           0.0s
# Error response from daemon: image with reference httpd:2.4 was found but does not match the specified platform: wanted linux/386, actual: linux/amd64

Then my containers and images look like this:

docker ps -a

CONTAINER ID   IMAGE                      COMMAND                  CREATED         STATUS                        PORTS                                                                      NAMES
af4daecab294   httpd:2.4                  "sh -c exit"             6 minutes ago   Created                                                                                                  1c-in-docker-httpd-base-1
49a6db9bef82   containrrr/watchtower      "/watchtower --scope…"   2 hours ago     Exited (1) 26 minutes ago                                                                                1c-in-docker-watchtower-int-1
2f9cf991b06d   containrrr/watchtower      "/watchtower --scope…"   2 hours ago     Exited (1) 26 minutes ago                                                                                1c-in-docker-watchtower-1
fc78613e637b   nginx:1.20-alpine          "sh -c exit"             2 hours ago     Created                                                                                                  1c-in-docker-nginx-base-1
7c6a4473c0ef   alpine:latest              "sh -c exit"             2 hours ago     Created                                                                                                  1c-in-docker-openvpn-base-1
e416b619ce4f   alpine/socat               "sh -c exit"             2 hours ago     Created                                                                                                  1c-in-docker-dns-proxy-base-1
26b465323339   ubuntu:focal               "sh -c exit"             2 hours ago     Created                                                                                                  1c-in-docker-postgres-base-1
f53b84830da6   alpine/socat               "socat UDP-LISTEN:53…"   2 hours ago     Up 2 hours                                                                                               1c-in-docker-dns-proxy-udp-1
63b57c25f3dc   alpine/socat               "socat TCP-LISTEN:53…"   2 hours ago     Up 2 hours                                                                                               1c-in-docker-dns-proxy-tcp-1
4a1993b915c7   docker                     "docker-entrypoint.s…"   2 hours ago     Up 2 hours                                                                                               1c-in-docker-cleanup-builder-1
61465f925ad1   docker                     "docker-entrypoint.s…"   2 hours ago     Exited (143) 26 minutes ago                                                                              1c-in-docker-builder-1
e72fc2f1fb1f   crazymax/fail2ban:latest   "/entrypoint.sh fail…"   2 hours ago     Up 2 hours (healthy)                                                                                     fail2ban
79343488bf1b   kylemanna/openvpn          "ovpn_run"               2 hours ago     Up 2 hours                    0.0.0.0:1555->1194/udp, :::1555->1194/udp                                  openvpn
0a3194a32f4d   apt-mirror                 "/bin/sh -c 'trap ex…"   2 hours ago     Up 2 hours                                                                                               1c-in-docker-pg-mirror-1
36d999aab0fa   1c-server:8.3.15-1700      "/entrypoint.sh /bin…"   47 hours ago    Up 47 hours                   80/tcp, 1540-1541/tcp, 1545/tcp, 1560-1591/tcp                             1c-in-docker-1c-ras-1
250172c3bd07   1c-server:8.3.15-1700      "/entrypoint.sh rage…"   47 hours ago    Up 47 hours                   80/tcp, 1540-1541/tcp, 1545/tcp, 1560-1591/tcp                             1c-server
21557b7c0152   nginx:1.20-alpine          "/docker-entrypoint.…"   6 days ago      Up 4 days                     0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   1c-in-docker-nginx-1
c306058410ef   certbot/certbot            "/bin/sh -c 'trap ex…"   8 days ago      Up 4 days                     80/tcp, 443/tcp                                                            1c-in-docker-certbot-1
6fa70d5f5ea4   postgrespro:13             "/entrypoint.sh post…"   8 days ago      Up 4 days                     5432/tcp                                                                   1c-postgrespro

docker image ls -a

REPOSITORY              TAG           IMAGE ID       CREATED         SIZE
1c-server               8.3.15-1700   9378a4c273b4   6 days ago      2.05GB
httpd                   2.4           b260a49eebf9   7 days ago      145MB
postgrespro             13            2d3076a0a6d7   8 days ago      452MB
apt-mirror              latest        50ab5bfd075d   8 days ago      162MB
certbot/certbot         latest        dce0bfb74107   13 days ago     115MB
docker                  latest        0208b15ab970   2 weeks ago     294MB
ubuntu                  focal         20fffa419e3a   2 weeks ago     72.8MB
httpd                   2.4.53        6207b07928e9   3 weeks ago     141MB
alpine                  latest        e66264b98777   4 weeks ago     5.53MB
nginx                   1.20-alpine   6cf0c840a5a4   4 weeks ago     23.2MB
alpine/socat            latest        aadd62c3c1fe   2 months ago    7.04MB
containrrr/watchtower   latest        333de6ea525a   5 months ago    16.9MB
crazymax/fail2ban       latest        6efb19d309d4   5 months ago    75.5MB
kylemanna/openvpn       latest        3d5a3627a7f6   18 months ago   15.3MB

docker image inspect b260a49eebf9 -f "{{.Architecture}}"
# amd64

Expected behavior
The architecture of the httpd:2.4 (which is now 2.4.54) and httpd:2.4.53 images should match after the upgrade, and docker compose up --no-start --force-recreate httpd-base should correctly recreate the service.

Screenshots

Environment

uname -a
# Linux ubuntu-2gb-hel1-2 5.4.0-120-generic #136-Ubuntu SMP Fri Jun 10 13:40:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux`

lsb_release -a
# No LSB modules are available.
# Distributor ID: Ubuntu
# Description:    Ubuntu 20.04.4 LTS
# Release:        20.04
# Codename:       focal

docker -v
# Docker version 20.10.16, build aa7e414

docker compose version
# Docker Compose version v2.5.0

pwd
# /home/txp/1c-in-docker
# so docker compose project is '1c-in-docker'

Logs from running watchtower with the --debug option

time="2022-06-21T21:44:43+03:00" level=debug msg=pull
time="2022-06-21T21:44:43+03:00" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2022-06-21T21:44:44+03:00" level=debug msg="Making sure everything is sane before starting"
time="2022-06-21T21:44:44+03:00" level=info msg="Watchtower 1.4.0"
time="2022-06-21T21:44:44+03:00" level=info msg="Using no notifications"
time="2022-06-21T21:44:44+03:00" level=info msg="Only checking containers with name \"1c-in-docker-httpd-base-1\", in scope \"pull\""
time="2022-06-21T21:44:44+03:00" level=info msg="Running a one time update."
time="2022-06-21T21:44:44+03:00" level=debug msg="Checking containers for updated images"
time="2022-06-21T21:44:44+03:00" level=debug msg="Retrieving running, stopped and exited containers"
time="2022-06-21T21:44:44+03:00" level=debug msg="Trying to load authentication credentials." container=/1c-in-docker-httpd-base-1 image="httpd:2.4"
time="2022-06-21T21:44:44+03:00" level=debug msg="No credentials for httpd:2.4 found" config_file=/config.json
time="2022-06-21T21:44:44+03:00" level=debug msg="Got image name: httpd:2.4"
time="2022-06-21T21:44:44+03:00" level=debug msg="Checking if pull is needed" container=/1c-in-docker-httpd-base-1 image="httpd:2.4"
time="2022-06-21T21:44:44+03:00" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
time="2022-06-21T21:44:44+03:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2022-06-21T21:44:44+03:00" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2022-06-21T21:44:44+03:00" level=debug msg="Setting scope for auth token" image=httpd scope="repository:library/httpd:pull"
time="2022-06-21T21:44:44+03:00" level=debug msg="No credentials found."
time="2022-06-21T21:44:45+03:00" level=debug msg="Parsing image ref" host=index.docker.io image=httpd normalized="docker.io/library/httpd:2.4" tag=2.4
time="2022-06-21T21:44:45+03:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/library/httpd/manifests/2.4"
time="2022-06-21T21:44:45+03:00" level=debug msg="Found a remote digest to compare with" remote="sha256:f899e432292e4ee92772d35e43b2e3dcf30042b1c6385d33f00a9300c69ee729"
time="2022-06-21T21:44:45+03:00" level=debug msg=Comparing local="sha256:c479bec894c5a7f8878b28e52d03cc95b1e784612ecd01ac7c7394fc5fa2e6e2" remote="sha256:f899e432292e4ee92772d35e43b2e3dcf30042b1c6385d33f00a9300c69ee729"
time="2022-06-21T21:44:45+03:00" level=debug msg="Digests did not match, doing a pull."
time="2022-06-21T21:44:45+03:00" level=debug msg="Pulling image" container=/1c-in-docker-httpd-base-1 image="httpd:2.4"
time="2022-06-21T21:44:54+03:00" level=info msg="Found new httpd:2.4 image (b260a49eebf9)"
time="2022-06-21T21:44:54+03:00" level=info msg="Session done" Failed=0 Scanned=1 Updated=1 notify=no
time="2022-06-21T21:44:54+03:00" level=info msg="Waiting for the notification goroutine to finish" notify=no

Additional context
Platform of a container is always "linux"

docker container inspect f53b84830da6 -f "{{.Platform}}"
# linux

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[stoinov]

I am experiencing similar issue - due to being stuck with older version of docker (19.03.15), manifest are not properly parsed Typical output:

no matching manifest for linux/arm/v8 in the manifest list entries

I have to manually set --platform arm64 on my deployed containers.

Watchtower does not seem to honor that and just relies on the docker mechanism to handle platforms, so it fails to pull any new images as the pull does not return anything.

Can you make it use the ```--platform`` parameter if possible? Or have a general ENV for setting this globally?

[piksel]

Can you make it use the ```--platform`` parameter if possible? Or have a general ENV for setting this globally?

That seems a bit less useful, as you might only have a single container that you need to use another platform for. But if the container was created with an explicitly defined platform, watchtower should be able to retain that platform. The only issue I see, is correctly identifying that it's using a non-default platform.
I'm currently thinking it could be done by:

1. Determine the docker host platform, should be possible with some API call
2. Check the current image for multi-arch to determine if the default platform was available, if not, it was explicitly set.

I don't know if 2. is possible though, and without it we would pin a container to the previous platform, even though a native one becomes available. But perhaps that's not that much of an issue, especially if we put that behaviour behind a flag, like --retain-platform.

[stoinov]

My suggestion - check the container architecture from an inspect and use it as platform. This can be a global flag or per container tag.

[simskij]

My suggestion - check the container architecture from an inspect and use it as platform. This can be a global flag or per container tag.

None of that should be needed. The image should have a platform attached to it, which should be the one it pulls next time. I strongly feel there is something else at play here as we rely on the docker engine and dockerhub to do the heavy lifting.

[piksel]

@simskij well, the image does have a platform, but we are not adding that to the pull API call.
It should just be a matter of adding this to dockerClient.PullImage:

	if persistPlatform {
		opts.Platform = container.ContainerInfo().Platform
	}