Watchtower fails to finalise update to itself in a custom docker image

[dennislapchenko]

Describe the bug

There are some issues about watchtower failing to update itself, but none address specifically this.
Even logically, I did expect that watchtower won't be able to restart itself while keeping up with the update flow.
Existence of PR's to fix this makes me think it should work OOTB but i am missing something..

Steps to reproduce

• docker-compose.yaml:

version: '3'

services:
  watchtower:
    image: 1337.dkr.ecr.eu-central-1.amazonaws.com/watchtower-ecr:latest
    restart: always
    environment:
      WATCHTOWER_CLEANUP: true
      WATCHTOWER_LOG_LEVEL: debug
      WATCHTOWER_NO_STARTUP_MESSAGE: true
      WATCHTOWER_ROLLING_RESTART: true
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

• pushing an update to latest tag results in the sequence described in the log below.

Expected behavior

Watchtower is able to perform a rolling update on itself

Screenshots

No response

Environment

• Platform: amazon linux 2
• Architecture: x86_64
• Docker Version: Docker version 20.10.25, build b82b9f3

Your logs

app-watchtower-1  | time="2023-11-09T10:49:07Z" level=debug msg="Retrieving running containers"
app-watchtower-1  | time="2023-11-09T10:49:07Z" level=debug msg="Got image name: 1337.dkr.ecr.eu-central-1.amazonaws.com/watchtower-ecr:latest"
app-watchtower-1  | time="2023-11-09T10:49:07Z" level=debug msg="Digests did not match, doing a pull."
app-watchtower-1  | time="2023-11-09T10:49:07Z" level=debug msg="Pulling image" container=/app-watchtower-1 image="1337.dkr.ecr.eu-central-1.amazonaws.com/watchtower-ecr:latest"
app-watchtower-1  | time="2023-11-09T10:49:07Z" level=info msg="Found new 1337.dkr.ecr.eu-central-1.amazonaws.com/watchtower-ecr:latest image (0699444d0c70)"
app-watchtower-1  | time="2023-11-09T10:49:07Z" level=debug msg="No pre-update command supplied. Skipping" container=/app-watchtower-1
app-watchtower-1  | time="2023-11-09T10:49:07Z" level=info msg="Stopping /app-watchtower-1 (a597b39e974e) with SIGTERM"
app-watchtower-1  | time="2023-11-09T10:49:07Z" level=info msg="Waiting for running update to be finished..."
app-watchtower-1  | time="2023-11-09T10:49:17Z" level=debug msg="Removing container a597b39e974e"
app-watchtower-1 exited with code 137

### Additional context

_No response_

[piksel]

You are running a custom version of watchtower and it seems to be lacking the required labels to identify itself. How did you create the images?

[dennislapchenko]

@piksel

FROM elephantventures/watchtower-ecr:latest

COPY --from=containrrr/watchtower:1.6.0 /watchtower /watchtower

ENTRYPOINT ["/watchtower"]

it was done like this, because I could not manage to correctly setup ECR helper, even though followed all the instructions. But using this guys old image - worked, so i just add the fresh watchtower binary there. Makes sense that it could be missing labels, need to see what I have to add and try again

[piksel]

You can find the label in our Dockerfile:

watchtower/dockerfiles/Dockerfile

Line 8 in 9180e95

LABEL "com.centurylinklabs.watchtower"="true"

You should know that running it this way is not supported and you are basically on your own. If you want help with the ECR sidecar, please open a discussion about it.