Watchtower with cleanup doesn't free image space (run system prune) #1332
Labels
Comments
Christoph-AK
added
Priority: Medium
Status: Available
Type: Bug
Christoph-AK
changed the title
|
Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏 |
|
This was also an issue for us last week on the same set of specs. AWS Ubuntu with --cleanup enabled did not actually prune the disk, causing an upgrade to fail. Still digging to make sure the upgrade failed due to only disk space, but there were a bunch of previous upgrades that had not been pruned by watchtower. |
|
Seems like this was an issue before here: #662 might be related. |
|
Could someone show the logs of a run? If you are running with |
Not sure if this helps, but here's my logs and my compose file. The images do get uploaded, the containers do restart. But old images aren't removed. formatted watchtower log |
|
So far I only got the behavior on one production system. As it is important that that one runs well, I included the gc container into my compose. gc:
image: drone/gc
restart: unless-stopped
container_name: gc
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- GC_INTERVAL=1hWith that all images get cleaned as expected. Unfortunately with that I can't really provide any useful logs on this. The continuos log of watchtower: https://gist.github.com/Christoph-AK/052523cf83c093a3cba2dca63684c7b3 Thanks for looking into this! |
|
@pewter77 so, the containers Update: Perhaps the reason for your storage filling up is because of all those failed updates? For every check it would pull the new image, but since it cannot update the container it won't remove the old one. |
|
@Christoph-AK Could you check what images are present in Also, I added some more logging to #1466 so that all images that are removed (including parent images) are actually logged (using |
|
_watchtower_logs (1).txt I should mention that I do have connectivity issues because of my location, I have to use obfuscating proxies and such to pass over country wide internet firewalls. Though they are fairly consistent, sometimes they timeout. But like I said these containers seem to be updated, just the leftover images are still here so I'm not sure this is the issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
After running watchtower with cleanup for a while, my /var/lib/docker/overlay2 directory filled up and wasn't cleaned.
I needed to run
docker system prunefor the space to be freed.To Reproduce
Run watchtower with cleanup for a while with a frequently updating service, like grafana-oss.
Expected behavior
The space of images gets correctly cleaned up after updates.
Environment
Ubuntu on AWS, Docker version 20.10.14, build a224086
The text was updated successfully, but these errors were encountered: