Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install Helix Python packages into a venv in a separate docker stage #1142

Merged
merged 11 commits into from
Jul 24, 2024
Merged

Install Helix Python packages into a venv in a separate docker stage #1142

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
63867ea
Build the crypto wheel on alpine in a venv in a separate docker stage.
jkoritzinsky Jul 17, 2024
1285b28
Update alpine 3.20 to build the venv in a separate stage
jkoritzinsky Jul 19, 2024
a6b7fe2
Remove all usages of --break-system-packages by using a venv
jkoritzinsky Jul 19, 2024
7b1fbc8
Fixup Alpine images
jkoritzinsky Jul 22, 2024
459d819
Feedback
jkoritzinsky Jul 22, 2024
85fba4d
Install newer cargo versions on the venv stages where necessary
jkoritzinsky Jul 23, 2024
ef80a7c
Install more missing dependencies in venv stages
jkoritzinsky Jul 23, 2024
7d063b2
Add libffi-dev to debian image
jkoritzinsky Jul 24, 2024
99cde31
Merge branch 'main' of github.com:dotnet/dotnet-buildtools-prereqs-do…
jkoritzinsky Jul 24, 2024
60c26cc
Add missing packages based on CI failures
jkoritzinsky Jul 24, 2024
b572fc0
Install libssl-dev
jkoritzinsky Jul 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 20 additions & 19 deletions src/alpine/3.17/helix/amd64/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM amd64/alpine:3.17 as msquic
FROM amd64/alpine:3.17 AS msquic

# build MsQuic as we don't have packages
RUN apk add --upgrade --no-cache \
Expand Down Expand Up @@ -30,6 +30,19 @@ RUN cmake -B build/linux/x64_Release_openssl3 \
cmake --build build/linux/x64_Release_openssl3 --config Release && \
cmake --install build/linux/x64_Release_openssl3 --prefix /msquic

FROM amd64/alpine:3.17 AS venv

RUN apk add --upgrade --no-cache \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev

RUN python3 -m venv /venv && \
source /venv/bin/activate && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl

FROM amd64/alpine:3.17

Expand All @@ -53,23 +66,6 @@ RUN apk add --upgrade --no-cache \
sudo \
tzdata

# Install Helix Dependencies
RUN apk add --upgrade --no-cache \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev && \
ln -sf /usr/bin/python3 /usr/bin/python && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl && \
apk del \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev

# Copy msquic from the msquic image into our image that will run on Helix
COPY --from=msquic /msquic /usr

Expand All @@ -85,4 +81,9 @@ RUN /usr/sbin/adduser -D -g '' -G adm -s /bin/bash -u 1000 helixbot && \

USER helixbot

RUN python3 -m venv /home/helixbot/.vsts-env
# Install Helix Dependencies

ENV VIRTUAL_ENV=/home/helixbot/.vsts-env
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
COPY --from=venv /venv $VIRTUAL_ENV
37 changes: 19 additions & 18 deletions src/alpine/3.17/helix/arm32v7/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,19 @@ RUN cmake -B build/linux/arm_Release_openssl3 \
cmake --build build/linux/arm_Release_openssl3 --config Release && \
cmake --install build/linux/arm_Release_openssl3 --prefix /msquic

FROM arm32v7/alpine:3.17 as venv

RUN apk add --upgrade --no-cache \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev

RUN python3 -m venv /venv && \
source /venv/bin/activate && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl

FROM arm32v7/alpine:3.17

Expand All @@ -54,23 +67,6 @@ RUN apk add --upgrade --no-cache \
sudo \
tzdata

# Install Helix Dependencies
RUN apk add --upgrade --no-cache \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev && \
ln -sf /usr/bin/python3 /usr/bin/python && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl && \
apk del \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev

# Copy msquic from the msquic image into our image that will run on Helix
COPY --from=msquic /msquic /usr

Expand All @@ -86,4 +82,9 @@ RUN /usr/sbin/adduser -D -g '' -G adm -s /bin/bash -u 1000 helixbot && \

USER helixbot

RUN python3 -m venv /home/helixbot/.vsts-env
# Install Helix Dependencies

ENV VIRTUAL_ENV=/home/helixbot/.vsts-env
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
COPY --from=venv /venv $VIRTUAL_ENV
37 changes: 19 additions & 18 deletions src/alpine/3.17/helix/arm64v8/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,19 @@ RUN cmake -B build/linux/arm64_Release_openssl3 \
cmake --build build/linux/arm64_Release_openssl3 --config Release && \
cmake --install build/linux/arm64_Release_openssl3 --prefix /msquic

FROM arm64v8/alpine:3.17 as venv

RUN apk add --upgrade --no-cache \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev

RUN python3 -m venv /venv && \
source /venv/bin/activate && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl

FROM arm64v8/alpine:3.17

Expand All @@ -54,23 +67,6 @@ RUN apk add --upgrade --no-cache \
sudo \
tzdata

# Install Helix Dependencies
RUN apk add --upgrade --no-cache \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev && \
ln -sf /usr/bin/python3 /usr/bin/python && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl && \
apk del \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev

# Copy msquic from the msquic image into our image that will run on Helix
COPY --from=msquic /msquic /usr

Expand All @@ -86,4 +82,9 @@ RUN /usr/sbin/adduser -D -g '' -G adm -s /bin/bash -u 1000 helixbot && \

USER helixbot

RUN python3 -m venv /home/helixbot/.vsts-env
# Install Helix Dependencies

ENV VIRTUAL_ENV=/home/helixbot/.vsts-env
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
COPY --from=venv /venv $VIRTUAL_ENV
38 changes: 20 additions & 18 deletions src/alpine/3.18/helix/amd64/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,20 @@ RUN cmake -B build/linux/x64_Release_openssl3 \
cmake --build build/linux/x64_Release_openssl3 --config Release && \
cmake --install build/linux/x64_Release_openssl3 --prefix /msquic

FROM amd64/alpine:3.18 AS venv

RUN apk add --upgrade --no-cache \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev

RUN python3 -m venv /venv && \
source /venv/bin/activate && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl


FROM amd64/alpine:3.18

Expand All @@ -53,23 +67,6 @@ RUN apk add --upgrade --no-cache \
sudo \
tzdata

# Install Helix Dependencies
RUN apk add --upgrade --no-cache \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev && \
ln -sf /usr/bin/python3 /usr/bin/python && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl && \
apk del \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev

# Copy msquic from the msquic image into our image that will run on Helix
COPY --from=msquic /msquic /usr

Expand All @@ -85,4 +82,9 @@ RUN /usr/sbin/adduser -D -g '' -G adm -s /bin/bash -u 1000 helixbot && \

USER helixbot

RUN python3 -m venv /home/helixbot/.vsts-env
# Install Helix Dependencies

ENV VIRTUAL_ENV=/home/helixbot/.vsts-env
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
COPY --from=venv /venv $VIRTUAL_ENV
37 changes: 19 additions & 18 deletions src/alpine/3.18/helix/arm32v7/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,19 @@ RUN cmake -B build/linux/arm_Release_openssl3 \
cmake --build build/linux/arm_Release_openssl3 --config Release && \
cmake --install build/linux/arm_Release_openssl3 --prefix /msquic

FROM arm32v7/alpine:3.18 as venv

RUN apk add --upgrade --no-cache \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev

RUN python3 -m venv /venv && \
source /venv/bin/activate && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl

FROM arm32v7/alpine:3.18

Expand All @@ -54,23 +67,6 @@ RUN apk add --upgrade --no-cache \
sudo \
tzdata

# Install Helix Dependencies
RUN apk add --upgrade --no-cache \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev && \
ln -sf /usr/bin/python3 /usr/bin/python && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl && \
apk del \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev

# Copy msquic from the msquic image into our image that will run on Helix
COPY --from=msquic /msquic /usr

Expand All @@ -86,4 +82,9 @@ RUN /usr/sbin/adduser -D -g '' -G adm -s /bin/bash -u 1000 helixbot && \

USER helixbot

RUN python3 -m venv /home/helixbot/.vsts-env
# Install Helix Dependencies

ENV VIRTUAL_ENV=/home/helixbot/.vsts-env
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
COPY --from=venv /venv $VIRTUAL_ENV
38 changes: 20 additions & 18 deletions src/alpine/3.18/helix/arm64v8/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@ RUN cmake -B build/linux/arm64_Release_openssl3 \
cmake --build build/linux/arm64_Release_openssl3 --config Release && \
cmake --install build/linux/arm64_Release_openssl3 --prefix /msquic

FROM arm64v8/alpine:3.18 as venv

RUN apk add --upgrade --no-cache \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev

RUN python3 -m venv /venv && \
source /venv/bin/activate && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl


FROM arm64v8/alpine:3.18

Expand All @@ -54,23 +68,6 @@ RUN apk add --upgrade --no-cache \
sudo \
tzdata

# Install Helix Dependencies
RUN apk add --upgrade --no-cache \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev && \
ln -sf /usr/bin/python3 /usr/bin/python && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl && \
apk del \
cargo \
libffi-dev \
linux-headers \
python3-dev \
openssl-dev

# Copy msquic from the msquic image into our image that will run on Helix
COPY --from=msquic /msquic /usr

Expand All @@ -86,4 +83,9 @@ RUN /usr/sbin/adduser -D -g '' -G adm -s /bin/bash -u 1000 helixbot && \

USER helixbot

RUN python3 -m venv /home/helixbot/.vsts-env
# Install Helix Dependencies

ENV VIRTUAL_ENV=/home/helixbot/.vsts-env
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
COPY --from=venv /venv $VIRTUAL_ENV
37 changes: 20 additions & 17 deletions src/alpine/3.20/helix/amd64/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM amd64/alpine:3.20 as msquic
FROM amd64/alpine:3.20 AS msquic

# build MsQuic as we don't have packages
RUN apk add --upgrade --no-cache \
Expand Down Expand Up @@ -30,6 +30,19 @@ RUN cmake -B build/linux/x64_Release_openssl3 \
cmake --build build/linux/x64_Release_openssl3 --config Release && \
cmake --install build/linux/x64_Release_openssl3 --prefix /msquic

FROM amd64/alpine:3.20 AS venv

RUN apk add --upgrade --no-cache \
python3-dev \
build-base \
libffi-dev \
gcc \
linux-headers

RUN python3 -m venv /venv && \
source /venv/bin/activate && \
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl

FROM amd64/alpine:3.20

Expand Down Expand Up @@ -67,21 +80,11 @@ RUN /usr/sbin/adduser -D -g '' -G adm -s /bin/bash -u 1000 helixbot && \
chmod 755 /root && \
echo "helixbot ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/helixbot

USER helixbot

# Install Helix Dependencies
RUN apk add --upgrade --no-cache \
build-base \
gcc \
linux-headers && \
# Create virtual environment
python3 -m venv /home/helixbot/.vsts-env && \
source /home/helixbot/.vsts-env/bin/activate && \
# Install Helix scripts
pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
pip install ./helix_scripts-*-py3-none-any.whl && \
# Delete unnecessary dependencies
apk del \
build-base \
gcc \
linux-headers

USER helixbot
ENV VIRTUAL_ENV=/home/helixbot/.vsts-env
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
COPY --from=venv /venv $VIRTUAL_ENV
Loading
Loading