Unable to push to internal registry using Docker 1.8 due to "Error parsing HTTP response: invalid character 'M' looking for beginning of value: "Method not allowed\n"" #4415

nak3 · 2015-08-28T17:13:17Z

Env:

$ openshift version
openshift v1.0.5-73-g29f82f4
kubernetes v1.1.0-alpha.0-1605-g44c91b1

$ docker version
Client:
 Version:      1.8.1.fc21
 API version:  1.20
 Package Version: docker-io-1.8.1-2.git32b8b25.fc21.x86_64
 Go version:   go1.4.2
 Git commit:   32b8b25/1.8.1
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.1.fc21
 API version:  1.20
 Package Version: 
 Go version:   go1.4.2
 Git commit:   32b8b25/1.8.1
 Built:        
 OS/Arch:      linux/amd64

NOTE: I tested on Fedora21 with docker 1.8. But I could reproduce CentOS7 with docker 1.8

Issue:

With docker v1.8, S2I build failed because it can't push to docker reigstry with this error

"Error parsing HTTP response: invalid character 'M' looking for beginning of value: "Method not allowed\n""

Please see entire log

Reproduce steps:

step-1. Running docker 1.8
step-2. Running openshift (in my case stndalone mode)
step-3. Run S2I
- eg) oc new-app https://github.com/nak3/helloworld-v3.git -l app=hello
- I tested oc new-app https://github.com/openshift/sti-python.git --context-dir=3.3/test/standalone-test-app -l app=python --strategy=source, but same result.
- I tested Custom storategy but it is same result.
step-4. Check build log (pushing to the docker registry failed due to the invalid character error)

The text was updated successfully, but these errors were encountered:

mfojtik · 2015-09-03T09:35:59Z

This comment might be related:

moby/moby#15636 (comment)

mfojtik · 2015-09-03T09:36:40Z

@ncdc @rajatchopra ^^

mfojtik · 2015-09-03T09:37:53Z

Also might be related to this: #4378 if Docker registry use Route.

ncdc · 2015-09-03T12:48:05Z

The issue is that our copy of the registry doesn't support PATCH when uploading layers, but Docker 1.8 has moved to using that HTTP verb. We'll have to upgrade our registry, which we can't do until we refactor pruning (#3333).

cc @pweil- @smarterclayton

ncdc · 2015-09-03T12:50:44Z

Our copy of the registry's layerUploadDispatcher has this:

handler := http.Handler(handlers.MethodHandler{
        "POST": http.HandlerFunc(luh.StartLayerUpload),
        "GET":  http.HandlerFunc(luh.GetUploadStatus),
        "HEAD": http.HandlerFunc(luh.GetUploadStatus),
        // TODO(stevvooe): Must implement patch support.
        // "PATCH":    http.HandlerFunc(luh.PutLayerChunk),
        "PUT":    http.HandlerFunc(luh.PutLayerUploadComplete),
        "DELETE": http.HandlerFunc(luh.CancelLayerUpload),
    })

whereas upstream master has this:

handler := http.Handler(handlers.MethodHandler{
        "POST":   http.HandlerFunc(buh.StartBlobUpload),
        "GET":    http.HandlerFunc(buh.GetUploadStatus),
        "HEAD":   http.HandlerFunc(buh.GetUploadStatus),
        "PATCH":  http.HandlerFunc(buh.PatchBlobData),
        "PUT":    http.HandlerFunc(buh.PutBlobUploadComplete),
        "DELETE": http.HandlerFunc(buh.CancelBlobUpload),
    })

ncdc · 2015-09-03T13:49:04Z

FYI, it should be possible to bump our docker/distribution godep to the 2.0.1 tag. It's not a straightforward patch, but we should be able to get it in w/o too much difficulty.

nak3 · 2015-09-06T04:25:48Z

I am still hitting this error with latest origin build on F21/docker 1.8.

$ openshift version
openshift v1.0.5-262-gd22c5d2
kubernetes v1.1.0-alpha.0-1605-g44c91b1

$ docker version |grep Package
 Package Version: docker-io-1.8.1-2.git32b8b25.fc21.x86_64

$ git show -s --format=%H 
d22c5d2ae0c23fa6c1744947af88122867c31146

liggitt · 2015-09-06T04:27:27Z

Make sure you are building images and running with --latest-images=true. The fix is in the registry image, which you might still be pulling an old version of from dockerhub

nak3 · 2015-09-06T04:54:30Z

I added --latest-images=true to openshift start, but it is still same.
I updated origin-haproxy-router as well.

$ docker images |grep origin-haproxy-router
docker.io/openshift/origin-haproxy-router-base                  latest                                                             be628d930881        8 hours ago          259.4 MB
docker.io/openshift/origin-haproxy-router                       latest                                                             01f2fac1294d        35 hours ago         351.2 MB
openshift/origin-haproxy-router                                 c9008aa                                                            08fb31403552        8 days ago           348.7 MB
openshift/origin-haproxy-router                                 latest                                                             08fb31403552        8 days ago           348.7 MB
docker.io/openshift/origin-haproxy-router                       v1.0.5                                                             ce92a3a03ed7        12 days ago          348.4 MB
docker.io/openshift/origin-haproxy-router                       v1.0.4                                                             d2685ed64138        5 weeks ago          343.7 MB

liggitt · 2015-09-06T04:55:46Z

Sorry, that was a typo... the registry image is the one containing the fix

ncdc · 2015-09-06T09:43:55Z

You have to specify --latest-images when invoking oadm registry too.

On Sunday, September 6, 2015, Jordan Liggitt [email protected]
wrote:

Sorry, that was a typo... the registry image is the one containing the fix

—
Reply to this email directly or view it on GitHub
#4415 (comment).

nak3 · 2015-09-06T18:18:03Z

I am still getting same error.
I specified --latest-images openshift start and oadm registry both. And I think I used latest docker registry image.

sudo -E _output/local/go/bin/openshift start --latest-images --loglevel=4 --hostname=${ORIGIN_HOST} --volume-dir=${VOLUME_DIR} --etcd-dir=${ETCD_DIR} > ${ORIGIN_LOG} 2>&1 &

oadm registry  --latest-images --create --credentials=openshift.local.config/master/openshift-registry.kubeconfig

$ docker images |grep origin-docker-registry
docker.io/openshift/origin-docker-registry                      latest                                                             0d44ff4fabcb        2 days ago          288.6 MB
openshift/origin-docker-registry                                c9008aa                                                            d2494940681a        8 days ago          288.5 MB
openshift/origin-docker-registry                                latest                                                             d2494940681a        8 days ago          288.5 MB
docker.io/openshift/origin-docker-registry                      v1.0.5                                                             d58e391bf4f0        12 days ago         288.5 MB
docker.io/openshift/origin-docker-registry                      v1.0.4                                                             a5c39c44a38c        5 weeks ago         279.2 MB
docker.io/openshift/origin-docker-registry                      v1.0.0                                                             b2dc31c34ed4        10 weeks ago        326.3 MB

I0906 18:15:55.835938       1 cfg.go:66] Using serviceaccount user for Docker authentication for image 172.30.226.178:5000/default/sti-ruby:latest
I0906 18:15:55.835987       1 sti.go:149] Using provided push secret for pushing 172.30.226.178:5000/default/sti-ruby:latest image
I0906 18:15:55.836012       1 sti.go:151] Pushing 172.30.226.178:5000/default/sti-ruby:latest image ...
F0906 18:15:56.463331       1 builder.go:47] Build error: Failed to push image: Error parsing HTTP response: invalid character 'M' looking for beginning of value: "Method not allowed\n"

0xmichalis · 2015-09-07T09:55:55Z

@nak3 remove all docker-registry images from your system and build again

nak3 · 2015-09-07T15:27:46Z

After clean up all of my local images and download newest images, it worked. At this time, I downloaded this ID's registry image.

$ docker images |grep registry
docker.io/openshift/origin-docker-registry   latest              af376e7e7f33        18 hours ago        288.6 MB

nak3 mentioned this issue Aug 28, 2015

origin-custom-docker-builder doesn't work if docker version is different #4390

Closed

bparees added kind/bug Categorizes issue or PR as related to a bug. component/imageregistry labels Sep 2, 2015

mfojtik mentioned this issue Sep 3, 2015

Unable to push to docker registry with Docker 1.8 #4495

Closed

pweil- assigned 0xmichalis Sep 3, 2015

0xmichalis mentioned this issue Sep 4, 2015

Bump registry #4519

Merged

openshift-bot closed this as completed in #4519 Sep 4, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to push to internal registry using Docker 1.8 due to "Error parsing HTTP response: invalid character 'M' looking for beginning of value: "Method not allowed\n"" #4415

Unable to push to internal registry using Docker 1.8 due to "Error parsing HTTP response: invalid character 'M' looking for beginning of value: "Method not allowed\n"" #4415

nak3 commented Aug 28, 2015

mfojtik commented Sep 3, 2015

mfojtik commented Sep 3, 2015

mfojtik commented Sep 3, 2015

ncdc commented Sep 3, 2015

ncdc commented Sep 3, 2015

ncdc commented Sep 3, 2015

nak3 commented Sep 6, 2015

liggitt commented Sep 6, 2015

nak3 commented Sep 6, 2015

liggitt commented Sep 6, 2015

ncdc commented Sep 6, 2015

nak3 commented Sep 6, 2015

0xmichalis commented Sep 7, 2015

nak3 commented Sep 7, 2015

Unable to push to internal registry using Docker 1.8 due to "Error parsing HTTP response: invalid character 'M' looking for beginning of value: "Method not allowed\n"" #4415

Unable to push to internal registry using Docker 1.8 due to "Error parsing HTTP response: invalid character 'M' looking for beginning of value: "Method not allowed\n"" #4415

Comments

nak3 commented Aug 28, 2015

Env:

Issue:

Reproduce steps:

mfojtik commented Sep 3, 2015

mfojtik commented Sep 3, 2015

mfojtik commented Sep 3, 2015

ncdc commented Sep 3, 2015

ncdc commented Sep 3, 2015

ncdc commented Sep 3, 2015

nak3 commented Sep 6, 2015

liggitt commented Sep 6, 2015

nak3 commented Sep 6, 2015

liggitt commented Sep 6, 2015

ncdc commented Sep 6, 2015

nak3 commented Sep 6, 2015

0xmichalis commented Sep 7, 2015

nak3 commented Sep 7, 2015