Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flaky 'Remote end closed connection' in docker build #75

Open
ryanramage opened this issue May 18, 2016 · 14 comments
Open

flaky 'Remote end closed connection' in docker build #75

ryanramage opened this issue May 18, 2016 · 14 comments

Comments

@ryanramage
Copy link

I am trying to apt-get install to build a docker image. The problem is the httpredir is inconsistant in failures. Once it will run fine, next time I will get 4 packages failed and then next run 2 packages. It all seems to revolve around 'Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]'

Step 14 : RUN apt-get -yq --force-yes install libcairo2-dev
 ---> Running in b69c38c7f35c
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  libcairo-gobject2 libcairo-script-interpreter2 libcairo2 libexpat1
  libexpat1-dev libfontconfig1-dev libfreetype6-dev libice-dev libice6
  liblzo2-2 libpixman-1-0 libpixman-1-dev libpthread-stubs0-dev libsm-dev
  libsm6 libx11-dev libx11-doc libxau-dev libxcb-render0 libxcb-render0-dev
  libxcb-shm0 libxcb-shm0-dev libxcb1-dev libxdmcp-dev libxext-dev
  libxrender-dev libxrender1 x11-common x11proto-core-dev x11proto-input-dev
  x11proto-kb-dev x11proto-render-dev x11proto-xext-dev xorg-sgml-doctools
  xtrans-dev
Suggested packages:
  libcairo2-doc libice-doc libsm-doc libxcb-doc libxext-doc
The following NEW packages will be installed:
  libcairo-gobject2 libcairo-script-interpreter2 libcairo2 libcairo2-dev
  libexpat1-dev libfontconfig1-dev libfreetype6-dev libice-dev libice6
  liblzo2-2 libpixman-1-0 libpixman-1-dev libpthread-stubs0-dev libsm-dev
  libsm6 libx11-dev libx11-doc libxau-dev libxcb-render0 libxcb-render0-dev
  libxcb-shm0 libxcb-shm0-dev libxcb1-dev libxdmcp-dev libxext-dev
  libxrender-dev libxrender1 x11-common x11proto-core-dev x11proto-input-dev
  x11proto-kb-dev x11proto-render-dev x11proto-xext-dev xorg-sgml-doctools
  xtrans-dev
The following packages will be upgraded:
  libexpat1
1 upgraded, 35 newly installed, 0 to remove and 51 not upgraded.
Need to get 10.3 MB of archives.
After this operation, 34.5 MB of additional disk space will be used.
Get:1 http://security.debian.org/ jessie/updates/main libexpat1 amd64 2.1.0-6+deb8u2 [79.4 kB]
Get:2 http://security.debian.org/ jessie/updates/main libexpat1-dev amd64 2.1.0-6+deb8u2 [126 kB]
Get:3 http://httpredir.debian.org/debian/ jessie/main libpixman-1-0 amd64 0.32.6-3 [507 kB]
Get:4 http://httpredir.debian.org/debian/ jessie/main libxcb-render0 amd64 1.10-3+b1 [17.5 kB]
Get:5 http://httpredir.debian.org/debian/ jessie/main libxcb-shm0 amd64 1.10-3+b1 [11.5 kB]
Get:6 http://httpredir.debian.org/debian/ jessie/main libxrender1 amd64 1:0.9.8-1+b1 [31.4 kB]
Get:7 http://httpredir.debian.org/debian/ jessie/main libcairo2 amd64 1.14.0-2.1+deb8u1 [746 kB]
Get:8 http://httpredir.debian.org/debian/ jessie/main libcairo-gobject2 amd64 1.14.0-2.1+deb8u1 [304 kB]
Err http://httpredir.debian.org/debian/ jessie/main x11-common all 1:7.7+7
  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]
Get:9 http://httpredir.debian.org/debian/ jessie/main liblzo2-2 amd64 2.08-1.2 [54.6 kB]
Get:10 http://httpredir.debian.org/debian/ jessie/main libcairo-script-interpreter2 amd64 1.14.0-2.1+deb8u1 [341 kB]
Get:11 http://httpredir.debian.org/debian/ jessie/main libice6 amd64 2:1.0.9-1+b1 [58.8 kB]
Get:12 http://httpredir.debian.org/debian/ jessie/main libsm6 amd64 2:1.2.2-1+b1 [33.6 kB]
Get:13 http://httpredir.debian.org/debian/ jessie/main libfreetype6-dev amd64 2.5.2-3+deb8u1 [638 kB]
Get:14 http://httpredir.debian.org/debian/ jessie/main libfontconfig1-dev amd64 2.11.0-6.3 [896 kB]
Err http://httpredir.debian.org/debian/ jessie/main libxdmcp-dev amd64 1:1.1.1-1+b1
  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]
Get:15 http://httpredir.debian.org/debian/ jessie/main xorg-sgml-doctools all 1:1.11-1 [21.9 kB]
Err http://httpredir.debian.org/debian/ jessie/main x11proto-kb-dev all 1.0.6-2
  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]
Get:16 http://httpredir.debian.org/debian/ jessie/main x11proto-core-dev all 7.0.26-1 [729 kB]
Get:17 http://httpredir.debian.org/debian/ jessie/main libxau-dev amd64 1:1.0.8-1 [23.6 kB]
Get:18 http://httpredir.debian.org/debian/ jessie/main x11proto-input-dev all 2.3.1-1 [157 kB]
Err http://httpredir.debian.org/debian/ jessie/main libx11-dev amd64 2:1.6.2-3
  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]
Get:19 http://httpredir.debian.org/debian/ jessie/main xtrans-dev all 1.3.4-1 [99.3 kB]
Get:20 http://httpredir.debian.org/debian/ jessie/main libpthread-stubs0-dev amd64 0.3-4 [3866 B]
Get:21 http://httpredir.debian.org/debian/ jessie/main libxcb1-dev amd64 1.10-3+b1 [83.8 kB]
Get:22 http://httpredir.debian.org/debian/ jessie/main x11proto-render-dev all 2:0.11.1-2 [20.8 kB]
Get:23 http://httpredir.debian.org/debian/ jessie/main libxrender-dev amd64 1:0.9.8-1+b1 [38.9 kB]
Get:24 http://httpredir.debian.org/debian/ jessie/main x11proto-xext-dev all 7.3.0-1 [212 kB]
Get:25 http://httpredir.debian.org/debian/ jessie/main libxext-dev amd64 2:1.3.3-1 [107 kB]
Get:26 http://httpredir.debian.org/debian/ jessie/main libice-dev amd64 2:1.0.9-1+b1 [66.3 kB]
Get:27 http://httpredir.debian.org/debian/ jessie/main libsm-dev amd64 2:1.2.2-1+b1 [35.8 kB]
Get:28 http://httpredir.debian.org/debian/ jessie/main libpixman-1-dev amd64 0.32.6-3 [523 kB]
Get:29 http://httpredir.debian.org/debian/ jessie/main libxcb-render0-dev amd64 1.10-3+b1 [22.8 kB]
Get:30 http://httpredir.debian.org/debian/ jessie/main libxcb-shm0-dev amd64 1.10-3+b1 [12.8 kB]
Get:31 http://httpredir.debian.org/debian/ jessie/main libcairo2-dev amd64 1.14.0-2.1+deb8u1 [884 kB]
Get:32 http://httpredir.debian.org/debian/ jessie/main libx11-doc all 2:1.6.2-3 [2026 kB]
Fetched 8913 kB in 10s (824 kB/s)
E: Failed to fetch http://httpredir.debian.org/debian/pool/main/x/xorg/x11-common_7.7+7_all.deb  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]

E: Failed to fetch http://httpredir.debian.org/debian/pool/main/libx/libxdmcp/libxdmcp-dev_1.1.1-1+b1_amd64.deb  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]

E: Failed to fetch http://httpredir.debian.org/debian/pool/main/x/x11proto-kb/x11proto-kb-dev_1.0.6-2_all.deb  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]

E: Failed to fetch http://httpredir.debian.org/debian/pool/main/libx/libx11/libx11-dev_1.6.2-3_amd64.deb  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c apt-get -yq --force-yes install libcairo2-dev' returned a non-zero code: 100

Next Run

Step 14 : RUN apt-get -yq --force-yes install libcairo2-dev
 ---> Running in 9ab1544d97bd
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  libcairo-gobject2 libcairo-script-interpreter2 libcairo2 libexpat1
  libexpat1-dev libfontconfig1-dev libfreetype6-dev libice-dev libice6
  liblzo2-2 libpixman-1-0 libpixman-1-dev libpthread-stubs0-dev libsm-dev
  libsm6 libx11-dev libx11-doc libxau-dev libxcb-render0 libxcb-render0-dev
  libxcb-shm0 libxcb-shm0-dev libxcb1-dev libxdmcp-dev libxext-dev
  libxrender-dev libxrender1 x11-common x11proto-core-dev x11proto-input-dev
  x11proto-kb-dev x11proto-render-dev x11proto-xext-dev xorg-sgml-doctools
  xtrans-dev
Suggested packages:
  libcairo2-doc libice-doc libsm-doc libxcb-doc libxext-doc
The following NEW packages will be installed:
  libcairo-gobject2 libcairo-script-interpreter2 libcairo2 libcairo2-dev
  libexpat1-dev libfontconfig1-dev libfreetype6-dev libice-dev libice6
  liblzo2-2 libpixman-1-0 libpixman-1-dev libpthread-stubs0-dev libsm-dev
  libsm6 libx11-dev libx11-doc libxau-dev libxcb-render0 libxcb-render0-dev
  libxcb-shm0 libxcb-shm0-dev libxcb1-dev libxdmcp-dev libxext-dev
  libxrender-dev libxrender1 x11-common x11proto-core-dev x11proto-input-dev
  x11proto-kb-dev x11proto-render-dev x11proto-xext-dev xorg-sgml-doctools
  xtrans-dev
The following packages will be upgraded:
  libexpat1
1 upgraded, 35 newly installed, 0 to remove and 51 not upgraded.
Need to get 10.3 MB of archives.
After this operation, 34.5 MB of additional disk space will be used.
Get:1 http://security.debian.org/ jessie/updates/main libexpat1 amd64 2.1.0-6+deb8u2 [79.4 kB]
Get:2 http://security.debian.org/ jessie/updates/main libexpat1-dev amd64 2.1.0-6+deb8u2 [126 kB]
Get:3 http://httpredir.debian.org/debian/ jessie/main libpixman-1-0 amd64 0.32.6-3 [507 kB]
Get:4 http://httpredir.debian.org/debian/ jessie/main libxcb-render0 amd64 1.10-3+b1 [17.5 kB]
Get:5 http://httpredir.debian.org/debian/ jessie/main libxcb-shm0 amd64 1.10-3+b1 [11.5 kB]
Get:6 http://httpredir.debian.org/debian/ jessie/main libxrender1 amd64 1:0.9.8-1+b1 [31.4 kB]
Get:7 http://httpredir.debian.org/debian/ jessie/main libcairo2 amd64 1.14.0-2.1+deb8u1 [746 kB]
Get:8 http://httpredir.debian.org/debian/ jessie/main libcairo-gobject2 amd64 1.14.0-2.1+deb8u1 [304 kB]
Get:9 http://httpredir.debian.org/debian/ jessie/main liblzo2-2 amd64 2.08-1.2 [54.6 kB]
Get:10 http://httpredir.debian.org/debian/ jessie/main libcairo-script-interpreter2 amd64 1.14.0-2.1+deb8u1 [341 kB]
Get:11 http://httpredir.debian.org/debian/ jessie/main x11-common all 1:7.7+7 [287 kB]
Get:12 http://httpredir.debian.org/debian/ jessie/main libice6 amd64 2:1.0.9-1+b1 [58.8 kB]
Get:13 http://httpredir.debian.org/debian/ jessie/main libsm6 amd64 2:1.2.2-1+b1 [33.6 kB]
Get:14 http://httpredir.debian.org/debian/ jessie/main libfreetype6-dev amd64 2.5.2-3+deb8u1 [638 kB]
Err http://httpredir.debian.org/debian/ jessie/main libx11-dev amd64 2:1.6.2-3
  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]
Get:15 http://httpredir.debian.org/debian/ jessie/main libfontconfig1-dev amd64 2.11.0-6.3 [896 kB]
Get:16 http://httpredir.debian.org/debian/ jessie/main xorg-sgml-doctools all 1:1.11-1 [21.9 kB]
Get:17 http://httpredir.debian.org/debian/ jessie/main x11proto-core-dev all 7.0.26-1 [729 kB]
Err http://httpredir.debian.org/debian/ jessie/main libcairo2-dev amd64 1.14.0-2.1+deb8u1
  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]
Get:18 http://httpredir.debian.org/debian/ jessie/main libxau-dev amd64 1:1.0.8-1 [23.6 kB]
Get:19 http://httpredir.debian.org/debian/ jessie/main libxdmcp-dev amd64 1:1.1.1-1+b1 [40.2 kB]
Get:20 http://httpredir.debian.org/debian/ jessie/main x11proto-input-dev all 2.3.1-1 [157 kB]
Get:21 http://httpredir.debian.org/debian/ jessie/main x11proto-kb-dev all 1.0.6-2 [269 kB]
Get:22 http://httpredir.debian.org/debian/ jessie/main xtrans-dev all 1.3.4-1 [99.3 kB]
Get:23 http://httpredir.debian.org/debian/ jessie/main libpthread-stubs0-dev amd64 0.3-4 [3866 B]
Get:24 http://httpredir.debian.org/debian/ jessie/main libxcb1-dev amd64 1.10-3+b1 [83.8 kB]
Get:25 http://httpredir.debian.org/debian/ jessie/main x11proto-render-dev all 2:0.11.1-2 [20.8 kB]
Get:26 http://httpredir.debian.org/debian/ jessie/main libxrender-dev amd64 1:0.9.8-1+b1 [38.9 kB]
Get:27 http://httpredir.debian.org/debian/ jessie/main x11proto-xext-dev all 7.3.0-1 [212 kB]
Get:28 http://httpredir.debian.org/debian/ jessie/main libxext-dev amd64 2:1.3.3-1 [107 kB]
Get:29 http://httpredir.debian.org/debian/ jessie/main libice-dev amd64 2:1.0.9-1+b1 [66.3 kB]
Get:30 http://httpredir.debian.org/debian/ jessie/main libsm-dev amd64 2:1.2.2-1+b1 [35.8 kB]
Get:31 http://httpredir.debian.org/debian/ jessie/main libpixman-1-dev amd64 0.32.6-3 [523 kB]
Get:32 http://httpredir.debian.org/debian/ jessie/main libxcb-render0-dev amd64 1.10-3+b1 [22.8 kB]
Get:33 http://httpredir.debian.org/debian/ jessie/main libxcb-shm0-dev amd64 1.10-3+b1 [12.8 kB]
Get:34 http://httpredir.debian.org/debian/ jessie/main libx11-doc all 2:1.6.2-3 [2026 kB]
Fetched 8625 kB in 10s (827 kB/s)
E: Failed to fetch http://httpredir.debian.org/debian/pool/main/libx/libx11/libx11-dev_1.6.2-3_amd64.deb  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]

E: Failed to fetch http://httpredir.debian.org/debian/pool/main/c/cairo/libcairo2-dev_1.14.0-2.1+deb8u1_amd64.deb  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c apt-get -yq --force-yes install libcairo2-dev' returned a non-zero code: 100
@ryanramage
Copy link
Author

This issue seems to be related to issue #74

@emirotin
Copy link

Hit this one today

E: Failed to fetch http://httpredir.debian.org/debian/pool/main/libx/libxau/libxau-dev_1.0.8-1_armhf.deb  Error reading from server. Remote end closed connection [IP: 128.31.0.66 80]

@harsha-y
Copy link

Both these mirrors seem to be consistently throwing errors today -
[IP: 5.153.231.35 80]
[IP: 128.31.0.66 80]

@dnorwood
Copy link

Hi Harsha,

What are the errors you are receiving?

Best regards,

Donald Norwood
-Debian Mirrors Team

On 06/13/2016 03:51 PM, Harsha Yalamanchili wrote:

Both these mirrors seem to be consistently throwing errors today -
[IP: 5.153.231.35 80]
[IP: 128.31.0.66 80]


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#75 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAePXItM4ERv70tOsk4UeZ86OBWyR0OZks5qLbSygaJpZM4IhZFu.

@sublimino
Copy link

sublimino commented Jun 22, 2016

Same errors for me when building Docker containers.

Dockerfile inherits from latest debian:jessie image (sadly, but necessarily, it installs a lot of packages. Failures are non-deterministic).

Errors:

...
Get:52 http://httpredir.debian.org/debian/ jessie/main fontconfig-config all 2.11.0-6.3 [273 kB]
Err http://httpredir.debian.org/debian/ jessie/main libqt4-dbus amd64 4:4.8.6+git64-g5dc8b2b+dfsg-3+deb8u1
  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]
Get:53 http://httpredir.debian.org/debian/ jessie/main libfontconfig1 amd64 2.11.0-6.3 [329 kB]
...
Get:351 http://httpredir.debian.org/debian/ jessie/main sqlite3 amd64 3.8.7.1-1+deb8u1 [101 kB]
Fetched 159 MB in 1min 1s (2605 kB/s)
E: Failed to fetch http://httpredir.debian.org/debian/pool/main/m/mesa/libglapi-mesa_10.3.2-1+deb8u1_amd64.deb  Error reading from server. R
emote end closed connection [IP: 5.153.231.35 80]

E: Failed to fetch http://httpredir.debian.org/debian/pool/main/q/qt4-x11/libqt4-dbus_4.8.6+git64-g5dc8b2b+dfsg-3+deb8u1_amd64.deb  Error re
ading from server. Remote end closed connection [IP: 5.153.231.35 80]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

I've previously had errors building this same Dockerfile with other IPs:

E: Failed to fetch http://httpredir.debian.org/debian/pool/main/q/qt4-x11/qdbus_4.8.6+git64-g5dc8b2b+dfsg-3+deb8u1_amd64.deb  Error reading 
from server. Remote end closed connection [IP: 128.31.0.66 80]
...
E: Failed to fetch http://httpredir.debian.org/debian/pool/main/t/texlive-base/texlive-base_2014.20141024-2_all.deb  Error reading from serv
er. Remote end closed connection [IP: 5.153.231.35 80]
...
E: Failed to fetch http://httpredir.debian.org/debian/pool/main/g/gsfonts/gsfonts_8.11+urwcyr1.0.7~pre44-4.2_all.deb  Error reading from ser
ver. Remote end closed connection [IP: 128.31.0.66 80]

Switching to a localised mirror in the Dockerfile corrects this error (I used the domain http://httpredir.debian.org/debian/ redirected me to)

FROM debian:jessie

ENV HOME /home/mnemosyne

RUN (\
    export DEBIAN_FRONTEND=noninteractive; \
    sed --in-place 's/httpredir.debian.org/mirror.sov.uk.goscomb.net/' /etc/apt/sources.list && \
    apt-get update && \
    apt-get install -y --no-install-recommends \
      <pkg list>

@MichalKononenko
Copy link

Same issue over here. The 5.153.231.35 mirror is acting up.

@jonahoffline
Copy link

Experiencing the same issues with mirror 5.153.231.35

@joenap
Copy link

joenap commented Sep 12, 2016

It seems that #72 #74 are related to this, which I believe is caused by the underlying mirrors not having the specific packages. My team has experienced this for months, but we usually rebuild until we get a good mirror.

I propose retrying individual packages from other "random" mirrors (or whatever the method that determines the mirror in the first place), until the missing packages are successfully found and downloaded. It can fail after some number of attempts, or a timeout.

That may be tricky depending on how and where the mirror list is available in the code, and how it interfaces with apt, I'm sure. If I find time I may poke around at it. Otherwise, I would love to hear from the authors about the feasibility.

Cheers

@inakrin
Copy link

inakrin commented Sep 13, 2016

All the day today issues with Remote end closed connection [IP: 5.153.231.35 80]
Able to build containers after 3-9 runs.

@Demuxx
Copy link

Demuxx commented Sep 20, 2016

Same issue with [IP: 128.31.0.66 80] and [IP: 5.153.231.35 80] for xz-utils amd64 5.1.1alpha+20120614-2+b3 and libssl-dev amd64 1.0.1t-1+deb8u3 and libevent-openssl-2.0-5 amd64 2.0.21-stable-2.

@kralos
Copy link

kralos commented Oct 3, 2016

I'm getting this same issue on my own project (nothing to do with this repo). From the many builds I've investigated I'm beginning to get the feeling it's related to concurrent connections from the same build machine. If I have a test run by itself it seems to always work, but if I push more than 1 branch at a time they seem to break.

@mattpickle
Copy link

Same issue with [IP: 128.31.0.66 80]

@kralos
Copy link

kralos commented Oct 11, 2016

Update: I switched my apt mirror to one near my ci server and it hasn't failed since. I think this is just an issue with the httpredir.debian.org mirror. I suggest if you aren't publishing your docker image to do the same.

@anisse
Copy link

anisse commented Jan 29, 2017

Seeing this for 2001:41c8:1000:21::21:35 as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests