Proposal: Mirroring Support for Private Registries

[jlhawn]

Docker-DCO-1.1-Signed-off-by: Josh Hawn [email protected] (github: jlhawn)

[stevenschlansker]

I can mirror my private registries with this patch! This is amazing :)

[bordicon]

So amazing :D

[tarnfeld]

Nice one. 👍

[jlhawn]

There's probably a bunch of refactoring that would need to be done before this could get merged though.

[SvenDowideit]

and documentation! user facing wise, this is huge!! ;)

[stevenschlansker]

Documentation is in jlhawn#1

[stevenschlansker]

@SvenDowideit we have some documentation now, anything else you would like to see on this PR? Would love to get it in for 1.4.0 if possible, happy to work on any further improvements needed.

[bjaglin]

This is highly anticipated indeed, especially since https is now more or less enforced (which is a good thing) so using a local squid3 is not an easy option anymore. I wonder why the support for mirrors did not include private registries from the start? Is there a gotcha I am missing here?

[bjaglin]

Nevermind my last sentence above - I just had a look at the diff and realized it was still a proposal. @jlhawn is help wanted on this one or is this already ongoing?

[jessfraz]

Added to next so we can review after 1.4.0 release tomorrow.

[dmp42]

Making mirroring a first class citizen is definitely a target for 1.5
cc @stevvooe

Specifically adressing the engine performance issues (docker-archive/docker-registry#785), usability, namespace issues, and caching as part of the registry rewrite. In its current state, adding this feature to v1 is problematic.

[SvenDowideit]

This looks like the lines are more than 80 chars, but i'm not counting yet :)

[SvenDowideit]

LGTM - though now I'm curious if we need some way to configure images that should not be mirrored :/

[stevenschlansker]

@SvenDowideit shouldn't that be done by the registry? If it refuses to mirror an image it can simply not return it, and then the docker client will continue trying mirrors in order.

It's much easier to manage on the registry side as well - no need to roll out updates to the docker configuration on every client machine just to refuse to mirror an image.

I think having a simple and straightforward path to mirroring available now is much more important than making it very configurable - presumably the "v2 registry" feature will have all the configuration knobs. I just need some way to distribute images and can't wait indefinitely! The poor guys over in the EU already hate docker because of this, between the network latency and the fact that 'docker pull' is really really slow...

[SvenDowideit]

correct - it would be expected to be registry functionality :) and certainly not for this PR

the guys in the EU have a much better situation than those of us in AU - our internet noodle is almost back to modem standard.

[bobrik]

If we going to merge that PR without support from registry side, local registries will be unnecessarily mirrored. I have local registry for X TB in s3-like storage and mirror docker hub to the same s3-like storage, but with different path.

After upgrade without registry support I'll need to have 2X TB in s3-like storage + more traffic + more io + who knows what else. That doesn't make a lot of sense.

[stevvooe]

@bobrik We are working on registry support to make this more efficient. The plan is to have pull-through mirroring, such that a local registry, configured as the mirror in the client, can check local storage and fallback to a remote registry, such as docker hub, if the content is unavailable locally. This should massively cut incoming bandwidth at the cost of some extra local storage.

If this doesn't work for you, please let me know why (detail about your setup may help to clarify your position).

[stevvooe]

@bobrik Wanted to clarify that my comments above are related to the V2 registry.

[bobrik]

@stevvooe my setup is:

1. Local registry
2. Local mirror

My idea is that local mirror should not do any mirroring for local registry, since it would only increase bandwidth usage. Mirroring remote registries and docker hub is perfectly fine.

Is there an issue where mirroring is discussed for v2 registry?

[SvenDowideit]

Docs LGTM, once the lines are made 80 char or less

@fredlf @jamtur01

[stevvooe]

@bobrik The discussion is in docker-archive/docker-registry#658, albeit its a little thin, currently. I'll put together a plan there in the next few weeks. For your setup, the plan for V2 is to only require a single registry instance for this use case. Push/pull would be allowed locally, with fallback to a remote registry along with caching of the remote resources.

I don't want to promise you too much now, but adding your use case to the comments will be helpful when the design gets fully fleshed out.

[stevenschlansker]

@SvenDowideit the documentation in that file already has multiple lines that stretch to over 100 chars, and some even reaching 200 chars. But just for you I'll fix it anyway :P

@jlhawn mind merging jlhawn#2 ?

[tarnfeld]

@tarnfeld that person was hitting their mirror as well, and everything was configured like it should - still pulling from their mirror was not any faster than from the hub itself, for the reason I mentioned (lies in docker itself, see below).

Sorry perhaps I didn't explain my point well enough. The testing I did resulted in the mirror never being hit and I verified the code path with verbose logging, the mirroring code never gets hit in the client hence why this fix is being proposed. The client always went to the primary registry directly regardless of the mirror configuration, because the code does not allow private registries to go via mirrors.

It's also worth bearing in mind that not all registries are being powered by the docker-registry project.

[shin-]

There is a lot of confusion here regarding what we call mirrors in the docker daemon, and what we call mirrors in the docker-registry code, which makes the conversation all the more difficult to follow.

[SvenDowideit]

as an anecdote :)

before the --registry-mirror code was released working, hub downloads of the golang image used up about 80% of my bandwidth quota for the preceding month or 2 (whenever we started using it for boot2docker - now, it is not quite irrelevant.

each fresh test of a boot2docker vm would re-download the docker project build images, which would take in the order of an hour, whereas now, its close enough to instant that i don't have time to make another coffee.

so I imagine for a distributed organisation using private registries for their internal workflows, the problem is just as big a deal.

[bobrik]

Imagine that you deploy 800mb app (2 layers) 20-40 times a day to 150 machines daily. You deploy it from private registry and you also have mirror set up because docker hub is giving you images at 14kbps.

Right now workflow is:

1. Push image to local registry
2. Pull image from local registry on every node directly

With this change:

1. Push image to local registry
2. Pull image from local mirror, figure out that it's not there (images deployed once, why would it be there?), start mirroring in 150 parallel forks.
3. If that worked, pull image from mirror that is not a bit faster than local registry because it shares the same distributed storage backend.

Why do that? I don't see a reason to break existing behavior, since we cannot disable mirroring for local registries.

If you really need to mirror private registries — go ahead and set up caching nginx in front of your private registry, it is just highly cacheable http service after all. You have a solution. I don't see which solution do I have except "avoid docker upgrades before v2 registry is out".

[tarnfeld]

@bobrik That is a fair point, however when you're deploying a 3GB image once a month, and you have lots of new machines coming in and out of the cluster, it's a pain. Mirroring helps solve that problem because you can aggregate the downloads, when you're frequently changing images you have a slightly different problem because you simply have to ship the bytes at some point in time, and that's always going to be as slow as the first pull, and the more contention you have it will be slower yet.

Pull image from local mirror, figure out that it's not there (images deployed once, why would it be there?), start mirroring in 150 parallel forks.

I've put a diagram together to hopefully clear up any confusion of what we're trying to achieve.

Note: Please ignore the practicalities of the fact we have two locations (or regions) but only one single link :-)

Without docker mirroring and a single registry

With docker mirroring and a single registry

It's important to note that we're not trying to save on bandwidth as such (in terms of total bytes transferred) though that is a side bonus of doing this, but trying to reduce network contention. As you can see 76 seconds vs 17 seconds is a pretty significant speed up.

If you really need to mirror private registries — go ahead and set up caching nginx in front of your private registry, it is just highly cacheable http service after all. You have a solution. I don't see which solution do I have except "avoid docker upgrades before v2 registry is out".

This change does not preclude simply using a caching proxy, if you use a caching nginx proxy you still need to point the docker client (daemon) at the proxy instead of the primary registry, which is not possible currently in docker for images NOT hosted on index.docker.io. As far as I am aware this change is not at all tied to the docker/docker-registry project.

The main point of allowing mirrors to be configured at the daemon level is to abstract and remove the concept from the user. Mirrors may be configured in a cluster to improve performance based on network layout or security constraints, which is not something a user that's launching a docker container needs to be concerned with. It's also very nice to be able to use the same string (e.g registry.my-net.internal/foo/bar:1.0.0) to describe your image regardless of which site it's launching in, and therefore which mirror to use.

[bobrik]

Your "private mirror" could be replaced with nginx that only does caching (with locking) for layer files. Just point private registry domain to that nginx in every location. I fully understand you goals and desire to do that with registry itself.

My concern is that now we are breaking my existing setup without giving me an ability to fix it. I have 1gb link to my mirror and 1gb link to my registry.

[host] - 1gb -> [local registry] changes to [host] - 1gb -> [mirror] - 1gb -> [local registry].

Just let me have the old way without an extra hop. I'd love to see mirroring for private registries, but with the way to disable it for local ones.

[tarnfeld]

@bobrik This is an optional configuration. A mirror is not required to be able to use a private registry, nor will it be with this change.

I fully understand you goals and desire to do that with registry itself.

This change has nothing to do with the actual registry, this is a configuration option in the docker client which allows you to tell it to point to the [mirror]. Without this, your example [host] - 1gb -> [mirror] - 1gb -> [local registry] is impossible.

Is your concern that you would like to use your registry as a mirror for index.docker.io BUT not for your own private images? If so, that's OK, this won't cause a problem for you.

[bobrik]

Is your concern that you would like to use your registry as a mirror for index.docker.io BUT not for your own private images?

Exactly. I want to use mirroring registry for index.docker.io and all private registries, except my own registry that cannot become faster with mirroring.

[tarnfeld]

I see, that makes sense. Intriguing that you're using multiple private registries but that's a legit issue. @stevenschlansker any thoughts?

[stevenschlansker]

@bobrik @tarnfeld If your mirror has a mirror of public registry content but not your private registry, then when it fails to serve the image Docker will fall back to the 'real' private registry that you want it to read from. So as long as your 'mirror of index.docker.io' registry does not mirror your actual content, the only regression here is that it must check once that the image is not there before doing the same pull as it always did.
Do you agree?

[bobrik]

I agree. How do I do that?

[fredlf]

Where "in order" is left to right? Alphabetical?

[fredlf]

A copy-edit and a minor question. Once those nits are picked, docs LGTM.

[stevenschlansker]

It looks like the merge of #8456 causes this entire patch to end up as a merge conflict.

It looks like that PR changes how mirrors are handled, without mentioning that fact at all in the description. I did a brief read-through of #8456 but can't see how to resolve this patch against that one. And I'm not terribly thrilled about rebasing this patch again. Maybe someone who knows Docker better than I do will care enough to rebase.

[crosbymichael]

@stevenschlansker

I'm a little confused right now, I don't see any code in this PR right now. Did you remove those commits or what's up?

I'm sure someone else or one of the maintainers can rebase the PR for you.

[jlhawn]

@crosbymichael Yes, I removed the code commit for now because #8456 made it very different and now explicitly sets mirroring for the official "indexInfo" config. We'll have to rethink how to specify mirrors for private registries - probably a new CLI option on the daemon.

[jamtur01]

Docs LGTM

[icecrime]

Closing this as it is now tracked on docker/distribution (I guess distribution/distribution#19 is a good entrypoint).