Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

True zero-downtime deployment by request draining #21

Open
wowu opened this issue Dec 16, 2023 · 4 comments · May be fixed by #36
Open

True zero-downtime deployment by request draining #21

wowu opened this issue Dec 16, 2023 · 4 comments · May be fixed by #36

Comments

@wowu
Copy link
Owner

wowu commented Dec 16, 2023

Currently there's no way of telling Traefik that old container is going to be stopped, so it might route requests to a container that is shutting down. I'm creating this issue to track the progress of figuring out what is the best way of implementing this.

The problem was mentioned in this StackOverflow question: https://stackoverflow.com/questions/75918681/how-to-avoid-downtime-when-using-docker-rollout-with-traefik

Current idea

The easiest way seems to fail healthchecks before the container is going to be stopped, so Traefik is not routing new requests to the unhealthy container(s). This can be achieved by adding ! test -f /drain to container healthcheck, that is "fail if there exists a file named drain in /", and docker-rollout can create this file before stopping the old container.

I'm not sure if this behavior should be hardcoded in the tool, as there might be better ways of implementing request draining for proxies other than Traefik / nginx. Implementing hook support would allow docker-rollout users to implement true zero downtime deployment in two steps:

  1. Add && ! test -f /drain to current container healtcheck in compose file
  2. Add a hook like --before-stop "docker exec $1 touch /drain && sleep 10" to create the file manually
@rogerdz
Copy link

rogerdz commented Jan 10, 2024

https://doc.traefik.io/traefik/middlewares/http/retry/
maybe this can help ?

@immortaly007
Copy link

immortaly007 commented Oct 4, 2024

I like the current idea, but wouldn't the "sleep" timout need to be at least the interval of the health check times the "retry" amount for the health check? Otherwise the container might not enter the "unhealthy" state before the shutdown sequence starts.

Maybe another option (specifically for traefik): in the --before-stop hook, add the label traefik.enable=false. (note that I have not tested how quickly traefik would pick up this change).

@wowu
Copy link
Owner Author

wowu commented Oct 7, 2024

Thats a useful insight, thanks! We should describe the requirements for sleep in docs if we go for the hooks solution, or wait a required time in docker-rollout itself.

Docker container labels are immutable (at least for now: moby/moby#21721), so we cannot use them to deregister the container from Traefik.

@wowu wowu linked a pull request Nov 3, 2024 that will close this issue
@wowu
Copy link
Owner Author

wowu commented Nov 3, 2024

I created a sample implementation in #36, it would be amazing if someone could test it with their setup to confirm it works correctly 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants