Docker upload CI is broken

[cooperlees]

Haven't debugged yet but seems a build level bug inside the container. Need to see if it can be reproduced locally.

error: failed to solve: process "/bin/sh -c pip install --no-cache-dir --upgrade pip setuptools wheel     && apt update && apt install -y git     && cd /src     && pip install --no-cache-dir .[colorama,d]     && rm -rf /src     && apt remove -y git     && apt autoremove -y     && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 1
Error: buildx failed with: error: failed to solve: process "/bin/sh -c pip install --no-cache-dir --upgrade pip setuptools wheel     && apt update && apt install -y git     && cd /src     && pip install --no-cache-dir .[colorama,d]     && rm -rf /src     && apt remove -y git     && apt autoremove -y     && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 1

Example fail: https://github.com/psf/black/actions/runs/1391464711

[vbarbaresi]

The detailed error is actually Failed to build regex
Because it tries to build from source on linux/arm64 and fails with:

error: command 'gcc' failed: No such file or directory

It seems that regex did have some prebuilt wheels for arm64 in previous versions, but the latest doesn't have any aarch64 builds yet.

regex's CI might be broken: they have a travis-ci but I don't find any public interface where we could see the build status.

I see 3 possible solutions:

• Wait for them to upload prebuilt wheels
• Fallback on a pinned version (2021.10.8) that has prebuilt wheels
• Install build tools in docker (gcc) and potentially other dependencies like python3-dev

[cooperlees]

Thanks for debugging further!

Lets try the following:

• Install development packages into a builder image and have the ability to build our deps
  • Would like to keep the images as small as possible
• If that's to hard, maybe we pin for now and somehow be alerted to when we get latest version wheels again

[vbarbaresi]

I opened this issue on regex project: https://bitbucket.org/mrabarnett/mrab-regex/issues/434/aarch64-build-is-failing
I didn't find the Travis build logs last time, but now I finally found them: https://app.travis-ci.com/bitbucket/mrabarnett/mrab-regex/builds
Now that I see the logs I guess that it's probably a fluke in the aarch64 build. It doesn't seem fundamentally broken

I saw that you pinned regex and that it solved the issue for now.
I tried to build after installing build-essential and that was enough to solve the issue:

https://github.com/vbarbaresi/black/compare/7bf233a..bb86cd#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557

https://github.com/vbarbaresi/black/runs/4061992219?check_suite_focus=true
Though it took ~12 min to build instead of 4 min, so I don't think it's a good solution.
It would be good as a workaround in case some dependency is missing wheels again

[cooperlees]

Thanks for trying this. I think long term we should definitely support building from source but will probably need a multi stage build so we can keep the image size around the current ~50MB ...

If you/anyone have cycles I'd totally take a PR making us able to build from source via a multi stage build to keep our image size low.

[vbarbaresi]

I took a stab at it #2582

[cooperlees]

This is all fixed and was used to release 20.10b0 - Thanks for your help!