[libc++][Android] Add libcxx-builder-android Docker image #69273
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a Dockerfile for a new Docker image, libcxx-builder-android, that extends libcxx-builder with support for testing Android. The image includes these things: * An Android Clang compiler and sysroot. * The Android platform-tools (e.g. adb), so that an Android buildbot can run programs on an Android device. At container startup, copy these platform tools to an "android-platform-tools" Docker volume to share them with an emulator container. This copying ensures that the emulator and libcxx-builder containers avoid mismatched adb versions. * Docker, so that an Android buildbot can manage a sibling Docker container that runs the Android emulator. Add an Android-specific run-buildbot-container script for local development. Currently using this script requires building libcxx-build-android and an emulator image locally. Fixes: llvm#69270 Differential Revision: https://reviews.llvm.org/D155271
llvmbot
added
the
libc++
|
@llvm/pr-subscribers-libcxx Author: Ryan Prichard (rprichard) ChangesAdd a Dockerfile for a new Docker image, libcxx-builder-android, that extends libcxx-builder with support for testing Android. The image includes these things:
Add an Android-specific run-buildbot-container script for local development. Currently using this script requires building libcxx-build-android and an emulator image locally. Fixes: #69270 Full diff: https://github.com/llvm/llvm-project/pull/69273.diff 3 Files Affected:
diff --git a/libcxx/utils/ci/vendor/android/Dockerfile b/libcxx/utils/ci/vendor/android/Dockerfile
new file mode 100644
index 000000000000000..0acfff8e031dc81
--- /dev/null
+++ b/libcxx/utils/ci/vendor/android/Dockerfile
@@ -0,0 +1,83 @@
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+#
+# This Dockerfile extends ldionne/libcxx-builder with Android support, including
+# Android Clang and sysroot, Android platform-tools, and the Docker client.
+#
+# $ docker build -t libcxx-builder-android libcxx/utils/ci/vendor/android
+#
+
+FROM ldionne/libcxx-builder
+
+# Switch back to the root user to install things into /opt and /usr.
+USER root
+WORKDIR /
+
+# Install the Android platform tools (e.g. adb) into /opt/android/sdk.
+RUN apt-get update && apt-get install -y unzip
+RUN mkdir -p /opt/android/sdk && cd /opt/android/sdk && \
+ curl -LO https://dl.google.com/android/repository/platform-tools-latest-linux.zip && \
+ unzip platform-tools-latest-linux.zip && \
+ rm platform-tools-latest-linux.zip
+
+# Install the current Android compiler. Specify the prebuilts commit to retrieve
+# this compiler version even after it's removed from HEAD.
+ENV ANDROID_CLANG_VERSION=r498229b
+ENV ANDROID_CLANG_PREBUILTS_COMMIT=5186d132c99aa75dc25207c392e3ea5b93d0107e
+RUN git clone --filter=blob:none --sparse \
+ https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86 \
+ /opt/android/clang && \
+ git -C /opt/android/clang checkout ${ANDROID_CLANG_PREBUILTS_COMMIT} && \
+ git -C /opt/android/clang sparse-checkout add clang-${ANDROID_CLANG_VERSION} && \
+ rm -fr /opt/android/clang/.git && \
+ ln -sf /opt/android/clang/clang-${ANDROID_CLANG_VERSION} /opt/android/clang/clang-current && \
+ # The "git sparse-checkout" and "ln" commands succeed even if nothing was
+ # checked out, so use this "ls" command to fix that.
+ ls /opt/android/clang/clang-current/bin/clang
+
+# Install an Android sysroot. New AOSP sysroots are available at
+# https://ci.android.com/builds/branches/aosp-main/grid, the "ndk" target. The
+# NDK also makes its sysroot prebuilt available at
+# https://android.googlesource.com/platform/prebuilts/ndk/+/refs/heads/dev/platform/sysroot.
+ENV ANDROID_SYSROOT_BID=10957860
+RUN cd /opt/android && \
+ curl -L -o ndk_platform.tar.bz2 \
+ https://androidbuildinternal.googleapis.com/android/internal/build/v3/builds/${ANDROID_SYSROOT_BID}/ndk/attempts/latest/artifacts/ndk_platform.tar.bz2/url && \
+ tar xf ndk_platform.tar.bz2 && \
+ rm ndk_platform.tar.bz2
+
+# Install Docker. Mark the binary setuid so it can be run without prefixing it
+# with sudo. Adding the container user to the docker group doesn't work because
+# /var/run/docker.sock is owned by the host's docker GID, not the container's
+# docker GID.
+RUN apt-get update && apt-get install -y gpg && \
+ install -m 0755 -d /etc/apt/keyrings && \
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \
+ chmod a+r /etc/apt/keyrings/docker.gpg && \
+ echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
+ "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
+ tee /etc/apt/sources.list.d/docker.list > /dev/null && \
+ apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin && \
+ chmod u+s /usr/bin/docker
+
+COPY ./container-setup.sh /opt/android/container-setup.sh
+
+USER libcxx-builder
+WORKDIR /home/libcxx-builder
+
+# Add Android platform-tools to the PATH.
+ENV PATH="/opt/android/sdk/platform-tools:${PATH}"
+
+# Reset the buildkite-agent.cfg file. The tags are provided by an environment
+# variable instead.
+RUN cp /home/libcxx-builder/.buildkite-agent/buildkite-agent.dist.cfg \
+ /home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg
+
+# Modify the Buildkite agent cmdline to do Android setup stuff first.
+CMD /opt/android/container-setup.sh && buildkite-agent start
diff --git a/libcxx/utils/ci/vendor/android/container-setup.sh b/libcxx/utils/ci/vendor/android/container-setup.sh
new file mode 100755
index 000000000000000..56bc232fefa1e6b
--- /dev/null
+++ b/libcxx/utils/ci/vendor/android/container-setup.sh
@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+set -e
+
+# Different versions of adb can sometimes be incompatible (i.e. "adb server
+# version (nn) doesn't match this client (mm); killing..."). Ensure that the adb
+# in the main builder image matches that in the emulator by sharing the
+# platform-tools from the main image.
+if [ -d /mnt/android-platform-tools ]; then
+ sudo rm -fr /mnt/android-platform-tools/platform-tools
+ sudo cp -r /opt/android/sdk/platform-tools /mnt/android-platform-tools
+fi
diff --git a/libcxx/utils/ci/vendor/android/run-buildbot-container b/libcxx/utils/ci/vendor/android/run-buildbot-container
new file mode 100755
index 000000000000000..4ab83194c05d5c4
--- /dev/null
+++ b/libcxx/utils/ci/vendor/android/run-buildbot-container
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+# Similar to libcxx/utils/ci/run-buildbot-container, but adds additional options
+# needed for running Android tests.
+
+set -e
+
+MONOREPO_ROOT="$(git rev-parse --show-toplevel)"
+if [[ ! -d "${MONOREPO_ROOT}/libcxx/utils/ci/vendor/android" ]]; then
+ echo "Was unable to find the root of the LLVM monorepo; are you running from within the monorepo?"
+ exit 1
+fi
+
+DOCKER_OPTIONS=(-it)
+DOCKER_OPTIONS+=(--volume "${MONOREPO_ROOT}:/llvm")
+DOCKER_OPTIONS+=(--workdir "/llvm")
+DOCKER_OPTIONS+=(--cap-add=SYS_PTRACE)
+
+# Mount this volume to allow the main image to share its copy of the Android
+# platform tools with the emulator image, ensuring that the adb versions match.
+# This argument will create a new volume if it doesn't already exist.
+DOCKER_OPTIONS+=(--volume android-platform-tools:/mnt/android-platform-tools)
+
+# Pass through the Docker socket so that the buildbot can start a sibling
+# container running an Android emulator.
+if [ -S /var/run/docker.sock ]; then
+ DOCKER_OPTIONS+=(--volume /var/run/docker.sock:/var/run/docker.sock)
+fi
+
+docker run "${DOCKER_OPTIONS[@]}" libcxx-builder-android \
+ bash -c 'git config --global --add safe.directory /llvm; (/opt/android/container-setup.sh && exec bash)'
|
EricWF
approved these changes
Oct 19, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a Dockerfile for a new Docker image, libcxx-builder-android, that extends libcxx-builder with support for testing Android.
The image includes these things:
An Android Clang compiler and sysroot.
The Android platform-tools (e.g. adb), so that an Android buildbot can run programs on an Android device. At container startup, copy these platform tools to an "android-platform-tools" Docker volume to share them with an emulator container. This copying ensures that the emulator and libcxx-builder containers avoid mismatched adb versions.
Docker, so that an Android buildbot can manage a sibling Docker container that runs the Android emulator.
Add an Android-specific run-buildbot-container script for local development. Currently using this script requires building libcxx-build-android and an emulator image locally.
Fixes: #69270
Differential Revision: https://reviews.llvm.org/D155271