#504 Initial support for Docker 1.12 Swarm Mode

[diegofernandes]

This new provide just work with one network and traefik.port label.

I include a provide swarm it`s quite similar with docker provider but this swarm provide watch for services data.

[samber]

Thanks @diegofernandes for this contribution. Swarm mode is something we definitely want in Traefik !

(Related to #504)

[samber]

https://get.docker.com/builds/Linux/x86_64/docker-${DOCKER_VERSION} replies 404 when applied with 1.12 and circle-ci fail on that error :-(

Could you:

• add a .tgz after `docker-${DOCKER_VERSION}``
• set DOCKER_VERSION to 1.12.0

[vdemeester]

We should add a new docker version instead of replacing 1.10 one.
I think I handle the tar.gz thing in the docker file (but I might be wrong and could be in anothe project but…)

[diegofernandes]

Yeap, for sure.
I will do that.

[samber]

Both Swarm and Docker providers code and respective templates are very similar. Did you check if it was possible to make Swarm inheriting from Docker to avoid code duplication ?

I am aware that types are different (dockertypes.ContainerJSON vs swarm.Service), but we may adapt Docker provider to have a common type (custom struct ?).

[diegofernandes]

My first thought was to do in the same provide, but ContainerJSON and Service are quite different, we could do in the same provide with a Docker Provider Object or the Traefik configurations Objects.

[vdemeester]

@diegofernandes I do agree. Maybe we could put some code in common between the two (probably do a refactor and have a docker package too) ; but I think we should focus first on design decision before diving too much into code.

• Should we route to the virtual ip or container ips ? I tend to want the first (as it is now) — depending on how network is set up on, it could be a little messy.
• Do we want to allow some customization (virtual ip, all ips, ...) ?
• Do we want to support all ResolutionMode (or warn) ?

We should also add a TODO comment to not forgot to switch to an event based watch when docker swarm mode support events too.

[diegofernandes]

@vdemeester I think for now we should only accept ResolutionModeVIP, the ResolutionModeDNSRR i am not sure about the docker implementation, its sound not totaly implemented.
About the VIP, as you say using the VIP simplify the code and the swarm mode to the route for us.
I agree with a lot of networks the thing gonna be a little messy, but if we use de containers IPs the balance rule stay on traefik not on docker.
I think for now we could support VIP and add all IPs later.

[emilevauge]

@diegofernandes thanks a lot for your contribution.
Could you add a swarm mode section in the documentation?

[errm]

In kubernetes we route to the pod (container) ips. However I think we have better guarantees about how the network should behave so sticking with the vip is fine for first release.

[timchunght]

DUPLICATED to here because it is more appropriate as a general PR comment. Edited

@diegofernandes I recently built an image using your master branch and tried the following in my docker 1.12 swarm cluster.

docker service create --replicas 3 --publish 80:80 --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock --name traefik-swarm timchunght/traefik:swarm --web --docker --docker.domain=docker.localhost --logLevel=DEBUG

docker service create --replicas 3 --name whoami --label "traefik.frontend.rule=Host:whoami.docker.localhost" emilevauge/whoami 

curl -H Host:whoami.docker.localhost http://ip_of_any_of_the_nodes

but it doesn't seem to work yet because I am not getting the output from the whoami service. The curl command returned 404 page not found, which means that the traefik service seems to be running perfectly.

I tried the same traefik image on a single node docker server using the docker-compose config at https://docs.traefik.io/#docker and it worked so I know that the image was correctly built. Let me know and I will be happy to test the new solutions. Thanks

[diegofernandes]

@timchunght instead of docker provider use swarm provider(this will change).

docker service create --replicas 3 --publish 80:80 --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock --name traefik-swarm timchunght/traefik:swarm --web --swarm --swarm.domain=docker.localhost --logLevel=DEBUG

docker service create --replicas 3 --name whoami --label "traefik.port=80" --label "traefik.frontend.rule=Host:whoami.docker.localhost" emilevauge/whoami 

[emilevauge]

To clarify, is this PR ready to be reviewed and merged ?

[timchunght]

@diegofernandes I tested the two commands you posted and

curl -H Host:whoami.docker.localhost http://ip_of_any_of_the_nodes

is still 404 page not found. Thanks

[diegofernandes]

@emilevauge I'm doing a refactory on my code to merge Docker Provider and Swarm Provider. So It is not ready.

[diegofernandes]

@timchunght How many nodes do you have in the cluster?

[diegofernandes]

@vdemeester,@samber, @errm , @emilevauge , @timchunght I think this is ready for review.

[SvenAbels]

Is there a time plan on when the next release of Traefik with this swarm mode support will be published on docker hub?

[LTheobald]

I've also been trying to get this to work without much luck but it's probably my lack of Docker experience. I'm trying to find it via docker run first to see if it can even start up before I try it as a service. I'm trying things along the lines of:

docker run -p 80:80 --name traefik traefik-docker --web --docker.swarmmode --loglevel=DEBUG

traefik-docker is Traefik master with @diegofernandes's branch merged into it. It's then packaged with a simple Dockerfile:

FROM debian:jessie
COPY traefik_linux_amd64 /usr/local/bin
EXPOSE 80
ENTRYPOINT ["/usr/local/bin/traefik_linux_amd64"]

But on startup I get various messages along the line of:

time="2016-09-09T12:36:57Z" level=info msg="Starting provider *main.WebProvider {\"Address\":\":8080\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":false}"
time="2016-09-09T12:36:57Z" level=info msg="Starting server on :80"
time="2016-09-09T12:36:57Z" level=debug msg="Docker connection established with docker  (API )"
time="2016-09-09T12:36:57Z" level=error msg="Failed to list services for docker swarm mode, error Cannot connect to the Docker daemon. Is the docker daemon running on this host?"
time="2016-09-09T12:36:57Z" level=error msg="Docker connection error Cannot connect to the Docker daemon. Is the docker daemon running on this host?, retrying in 295.685407ms"
time="2016-09-09T12:36:57Z" level=debug msg="Docker connection established with docker  (API )"
time="2016-09-09T12:36:57Z" level=error msg="Failed to list services for docker swarm mode, error Cannot connect to the Docker daemon. Is the docker daemon running on this host?"
time="2016-09-09T12:36:57Z" level=error msg="Docker connection error Cannot connect to the Docker daemon. Is the docker daemon running on this host?, retrying in 997.803326ms"

Has anyone got this running yet & could give me some guidance on what I'm missing. If I can get this nailed I'll happily get writing some documentation for #656.

[mvdstam]

@LTheobald You probably didn't mount the docker daemon:

docker run -p 80:80 -v /var/run/docker.sock:/var/run/docker.sock --name traefik traefik-docker --web --docker.swarmmode --loglevel=DEBUG

[LTheobald]

@mvdstam Thanks a lot - You're spot on. Right, let's see if I can get this going then.

[LTheobald]

Just to confirm I've got @diegofernandes's branch working. I do get a 500 error when I attempt to access the web backend though & nothing is logged that I can see.

[mvdstam]

Is this ready to be merged and released? Can't wait to try this out with Docker Swarm 1.12... 😄

[LTheobald]

From my testing, I'm getting very mixed results. I've not touched Træfik before though & my Docker knowledge is still novice.

Running the following commands:

docker network create --driver overlay swarmnet
docker service create --network swarmnet --constraint 'node.role == manager' --replicas 1 --publish 80:80 --publish 8080:8080 --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock --name traefik traefik-docker --web --docker.swarmmode --docker.domain=traefik --loglevel=DEBUG --debug
docker service create --network swarmnet --replicas 3 --publish 3000:80 --name whoami1 --label "traefik.port=80" --label "traefik.frontend.rule=Host:whoami1.traefik" emilevauge/whoami
docker service create --network swarmnet --replicas 3 --publish 4000:80 --name whoami2 --label "traefik.port=80" --label "traefik.frontend.rule=Host:whoami2.traefik" emilevauge/whoami
curl -H Host:whoami1.traefik http://dev-docker-swarm-manager
curl -H Host:whoami2.traefik http://dev-docker-swarm-manager

For some reason I only get a 200 response for whoami2. whoami1 gives me a 504 Bad Gateway. Adding a whoami3, I get a 504 on that too so it's not a last thing wins situation. The curls that result in the 504 give me a log message of Error forwarding to http://10.0.0.12:80, err: dial tcp 10.0.0.12:80: getsockopt: no route to host

I'd love to hear if anyone else has had any luck with this so I know it's not just me!

And sorry if this seems like I'm complaining/moaning etc. - I'm really not & really appreciate what @diegofernandes has done.
`

[diegofernandes]

@LTheobald thanks for testing. in that case, you shouldn't use --publish, the Træfik access the backend containers through the inner network

[emilevauge]

LGTM @diegofernandes !
Still needs a rebase before merging ;)

[emilevauge]

Hi, traefik v1.1 developments will be frozen this week, on Thursday 29th. If you want your PR to be part of this release, please update your PR as soon as possible :)

[krishofmans]

+1 for getting this in the 1.1, I know for a fact that a lot of people are waiting for a solution like this to be swarm compatible.

[diegofernandes]

@emilevauge I will try to do a rebase, but all that I had done It was merging :(

[emilevauge]

@diegofernandes OK, once rebased, I will merge it right away :) While rebasing, it would also be great to squash some of your commits in order to keep a clean history. Thanks!

[vdemeester]

@diegofernandes hum the first line of the commit should be a little more descriptive than Sample file 👼 Something like Initial support for docker 1.12.

[emilevauge]

Thanks @diegofernandes for this great PR :)
Traefik 1.1 will have docker swarm mode support 🎉

[diegofernandes]

Thanks @emilevauge for merge this PR. I'm very happy to be part of this great project.

[pascalandy]

Traefik 1.1 will have docker swarm mode support 🎉

Can't wait for this.

[ralphtheninja]

Very awesome! But what about docs and examples?

[vdemeester]

@ralphtheninja will be done before the release 😉

[ralphtheninja]

@vdemeester Let me know if you need some testing!

[pascalandy]

@vdemeester Let me know if you need some testing!

+1

[pascalandy]

Its working great when Traefik is setup using Docker.

@gajus Are you saying this really stable with 1.12.1 (1.12+) ? Hope so :)

[dac388]

Would love to see the documentation updated for this.

[chiflux]

I'm trying to run traefik runs as swarm service and on a different host as the whoami service. Is this supposed to work?

For me it works only, if WHOAMI_SERVICEhas the same value as PROXY_HOST. 
This is not unexpected, since the local /var/run/docker.sock is mounted (i.e. only local Docker events are observed).

Since this makes little sense, I assume that this cannot be the final solution and I missed something essential. I guess that the docker events should be monitored instead the local docker socket. Right?!

Can somebody tell me, how I have to run the traefik container, so it actually receives swarm and not local events?

Thanks for any help!

NETWORK=ci-network
DOMAIN=mydomain.com
WHOAMI_SERVICE=whoami
PROXY_SERVICE=proxy

PROXY_HOST=sds06
WHOAMI_HOST=sds04


docker network create --driver overlay --subnet 10.0.9.0/24 --opt encrypted $NETWORK

docker service rm ${PROXY_SERVICE}
sleep 1
docker service create --network $NETWORK  --constraint "node.hostname==${PROXY_HOST}"  --replicas 1 --publish 80:80 --publish 8080:8080 --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock --name ${PROXY_SERVICE} containous/traefik:pr-602 --web --docker.swarmmode --docker.domain=${DOMAIN}

docker service rm ${WHOAMI_SERVICE}
sleep 5
docker service create --network ${NETWORK} --name ${WHOAMI_SERVICE} --label "traefik.frontend.rule=Host:${WHOAMI_SERVICE}.${DOMAIN}" --label "traefik.port=80"  --constraint "node.hostname==${WHOAMI_HOST}"  emilevauge/whoami

sleep 5
docker service ps ${PROXY_SERVICE}
docker service ps ${WHOAMI_SERVICE}

[vdemeester]

@christoph-hipp there is no docker events for swarm mode yet 😓 (meaning, there is no event emitted on the /events endpoint for service creation and such…). When you enable swarm mode on traefik using --docker.swarmmode it's gonna poll the service API instead of listening to events.

Note that Træfik must talk to a manager socket for now (no other way to get swarm mode api for now, but this might evolve).

For me it works only, if WHOAMI_SERVICE has the same value as PROXY_HOST

I think you mean WOAMI_HOST right ? Anyway, as long as the service is on the same network as traefik (for now) and obtain a virtual ip (which should always happen), the traefik container (task) and the whoami container (task) don't have to be on the same node at all.

A little bit more info would be cool too, like the docker version, how did you create the cluster — where does it run (virtualbox, …), with which flag (mainly --advertise-addr and --listen-addr).
Without træfik even, does two containers on the same overlay network reach each other ?

@christoph-hipp I also think creating an issue with your comment would be better than commenting on the PR :stuck_out_tongue_closed_eyes: (easier to follow, easier to fix maybe, …).

[chiflux]

Yes WHOAMI_HOST of course. Thank you for spotting this.

@vdemeester
I think, I'm using a pretty standard environment: Native Ubuntu 16.04 with Docker Engine 1.12.2 on 2 different computers (i.e. no VM!). The cluster consists of 2 manager nodes created with Docker Engine 1.12.
I could also open a issue, but I thought, it would be better to post it in the pull request of the feature, since it is not yet released. Right?

Question: Do I start traefik correct? If yes: Are there any other prerequisites to consider. If no: What is wrong?

Finally: If this feature is currently really working for my case, it would be really cool, if somebody posts a consistent and simple sample. An HowTo for this rather simple and common use-case is more useful than writing comments and/or opening new issues.

@vdemeester I assume from what you are writing that you have a working sample. Would be cool, if you could share it!

[vdemeester]

@christoph-hipp there is https://github.com/containous/traefik/blob/v1.1/docs/user-guide/swarm-mode.md (I didn't realize it was still not published as 1.1 is not yet released — we're at rc2 I think). I have another one that does almost the same thing on digital ocean (I'll share it a little later, not on my laptop right now 😅 ).

I didn't notice the first time, you should use traefik:v1.1.0-rc1 image instead (it's probably more up-to-date). Other than that, the way you start traefik seems correct, that's why I asked about container/task talking to each others 😝

[pascalandy]

On the docker hub I see v1.1.0-rc3. Is is incorporating code from v1.1.0-rc1 ?

Cheers!

[djalal]

@pascalandy : for bleeding-edge Docker images for Traefik, see containous/traefik on Docker Hub