Make Linux build in a container for compatibility with older distros

Background
----------

The Linux build of Arduino IDE is dynamically linked against the libstdc++ and glibc shared libraries. This results in
it having a dependency on the version of the libraries that happens to be present in the environment it is built in.

Although newer versions of the shared libraries are compatible with executables linked against an older version, the
reverse is not true. This means that building Arduino IDE on a Linux machine with a recent distro version installed
causes the IDE to error on startup for users who have a distro with older versions of the dependencies. For example:

```
Error: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /home/per/Downloads/arduino-ide_nightly-20231006_Linux_64bit/resources/app/lib/backend/native/nsfw.node)
```

or:

```
Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/per/Downloads/arduino-ide_2.0.5-snapshot-90b1f67_Linux_64bit/resources/app/node_modules/nsfw/build/Release/nsfw.node)
```

We were originally able to achieve our targeted range of Linux distro compatibility by running the Linux build in the
ubuntu-18.04 GitHub Actions hosted runner machine. Unfortunately GitHub stopped offering that runner machine. This meant
we were forced to update to using the ubuntu-20.04 runner machine instead, which caused the loss of compatibility of
the automatically generated builds with previously supported distro versions (e.g., Ubuntu 18.04). Since that time, the
release builds have been produced manually, which is inefficient and prone to human error.

Update
------

The identified solution to restoring fully automated builds with the target range of Linux distro compatibility is to
run the build in a Docker container that provides the suitable environment. This means a combination of an older distro
version with the modern versions of the development tool dependencies of the build.

Such a combination was achieved by creating a bespoke image based on the ubuntu:18.04 image. The Dockerfile is hosted in
the Arduino IDE repository in order to allow it to be maintained in parallel with the code and infrastructure.

Image Publishing
----------------

A "Push Container Images" GitHub Actions continuous delivery workflow is added to push updated images to the GitHub
Container registry when a commit that modifies relevant files is pushed to the main branch.

This means the image does not have formally versioned tags and the IDE build uses the container that results from the
configuration at the tip of the main branch at the time of the build. I think that is a reasonable approach in this use
case where the image is targeted to a single application rather than intended to be used by multiple projects.

Container Validation
--------------------

The build workflow is configured to trigger on completion of that push workflow in order to provide validation of the IDE build using the
updated container as well as the resulting tester builds of the IDE.

A "Check Containers" GitHub Actions continuous integration workflow is added to provide basic validation for changes to
the Dockerfile. It will automatically build the image and run the container on any push or pull request that modifies
relevant files.

Container Workflow Design
-------------------------

With the goal of reusability, the image data is contained in a job matrix in the workflow to allow them to accommodate
any number of arbitrary images.

Build Workflow Configuration
----------------------------

A container property is added to the build job matrix data. If the container.image property is set to null, GitHub
Actions will run the job directly in the runner environment instead of in a container. This allows us to produce the
builds using either a container or a bare runner machine as is appropriate for each target.

Unfortunately the latest v4 version of the actions/checkout action used to checkout the repository into the job
environment has a dependency on a higher version of glibc than is provided by the Linux container. For this reason, the
workflow is configured to use actions/checkout@v3 for the Linux build job. We will likely receive pull requests from
Dependabot offering to update this outdated action dependency for the v4 and at each subsequent major version release of
the action (which are not terribly frequent). We must decline the bump of the action in that specific step, but accept
the bumps of all other usages of the action in the workflows. Dependabot remembers when you decline a bump so this
should not be too bothersome.

.github/workflows/assets/linux.Dockerfile

@@ -0,0 +1,112 @@
+# The Arduino IDE Linux build workflow job runs in this container.
+# syntax=docker/dockerfile:1
+
+FROM ubuntu:18.04
+
+# See: https://unofficial-builds.nodejs.org/download/release/
+ARG node_version="18.17.1"
+
+RUN \
+  apt-get \
+    --yes \
+    update
+
+# This is required to get add-apt-repository
+RUN \
+  apt-get \
+    --yes \
+    install \
+      "software-properties-common=0.96.24.32.22"
+
+# Install Git
+# The PPA is required to get a modern version of Git. The version in the Ubuntu 18.04 package repository is 2.17.1,
+# while action/checkout@v3 requires 2.18 or higher.
+RUN \
+  add-apt-repository \
+    --yes \
+    "ppa:git-core/ppa" && \
+  apt-get \
+    --yes \
+    update && \
+  \
+  apt-get \
+    --yes \
+    install \
+      "git=1:2.42.0-0ppa1~ubuntu18.04.1" && \
+  \
+  apt-get \
+    --yes \
+    purge \
+      "software-properties-common"
+
+# The repository path must be added to safe.directory, otherwise any Git operations on it would fail with a
+# "dubious ownership" error. actions/checkout configures this, but it is not applied to containers.
+RUN \
+  git config \
+    --add \
+    --global \
+    "safe.directory" "/__w/arduino-ide/arduino-ide"
+ENV \
+  GIT_CONFIG_GLOBAL="/root/.gitconfig"
+
+# Install Python
+# The Python installed by actions/setup-python has dependency on a higher version of glibc than available in the
+# ubuntu:18.04 container.
+RUN \
+  apt-get \
+    --yes \
+    install \
+      "python3.8-minimal=3.8.0-3ubuntu1~18.04.2" && \
+  \
+  ln \
+    --symbolic \
+    --force \
+    "$(which python3.8)" \
+    "/usr/bin/python3"
+
+# Install Theia's package dependencies
+# These are pre-installed in the GitHub Actions hosted runner machines.
+RUN \
+  apt-get \
+    --yes \
+    install \
+      "libsecret-1-dev=0.18.6-1" \
+      "libx11-dev=2:1.6.4-3ubuntu0.4" \
+      "libxkbfile-dev=1:1.0.9-2"
+
+# Install Node.js
+# It is necessary to use the "unofficial" linux-x64-glibc-217 build because the official Node.js 18.x is dynamically
+# linked against glibc 2.28, while Ubuntu 18.04 has glibc 2.27.
+ARG node_installation_path="/tmp/node-installation"
+ARG artifact_name="node-v${node_version}-linux-x64-glibc-217"
+RUN \
+  mkdir "$node_installation_path" && \
+  cd "$node_installation_path" && \
+  \
+  apt-get \
+    --yes \
+    install \
+      "wget=1.19.4-1ubuntu2.2" && \
+  \
+  archive_name="${artifact_name}.tar.xz" && \
+  wget \
+    "https://unofficial-builds.nodejs.org/download/release/v${node_version}/${archive_name}" && \
+  \
+  apt-get \
+    --yes \
+    purge \
+      "wget" && \
+  \
+  tar \
+    --file="$archive_name" \
+    --extract && \
+  rm "$archive_name"
+ENV PATH="${PATH}:${node_installation_path}/${artifact_name}/bin"
+
+# Install Yarn
+# Yarn is pre-installed in the GitHub Actions hosted runner machines.
+RUN \
+  npm \
+    install \
+    --global \
+      "[email protected]"

.github/workflows/build.yml

@@ -35,6 +35,13 @@ on:
       - '*.md'
   schedule:
     - cron: '0 3 * * *' # run every day at 3AM (https://docs.github.com/en/actions/reference/events-that-trigger-workflows#scheduled-events-schedule)
+  workflow_run:
+    workflows:
+      - Push Container Images
+    branches:
+      - main
+    types: 
+      - completed
 
 env:
   # See vars.GO_VERSION field of https://github.com/arduino/arduino-cli/blob/master/DistTasks.yml
@@ -49,6 +56,12 @@ env:
         # Human identifier for the job.
         name: Windows
         runs-on: windows-2019
+        # The value is a string representing a JSON document.
+        # Setting image value to null causes the job to run directly in the runner machine instead of in a container.
+        container: |
+          {
+            \"image\": null
+          }
         # Name of the secret that contains the certificate.
         certificate-secret: WINDOWS_SIGNING_CERTIFICATE_PFX
         # Name of the secret that contains the certificate password.
@@ -68,7 +81,11 @@ env:
           name: Windows_X86-64_zip
     - config:
         name: Linux
-        runs-on: ubuntu-20.04
+        runs-on: ubuntu-latest
+        container: |
+          {
+            \"image\": \"ghcr.io/arduino/arduino-ide/linux:main\"
+          }
         mergeable-channel-file: 'false'
       artifacts:
         - path: '*Linux_64bit.zip'
@@ -78,6 +95,10 @@ env:
     - config:
         name: macOS x86
         runs-on: macos-latest
+        container: |
+          {
+            \"image\": null
+          }
         # APPLE_SIGNING_CERTIFICATE_P12 secret was produced by following the procedure from:
         # https://www.kencochrane.com/2020/08/01/build-and-sign-golang-binaries-for-macos-with-github-actions/#exporting-the-developer-certificate
         certificate-secret: APPLE_SIGNING_CERTIFICATE_P12
@@ -93,6 +114,10 @@ env:
     - config:
         name: macOS ARM
         runs-on: macos-latest-xlarge
+        container: |
+          {
+            \"image\": null
+          }
         certificate-secret: APPLE_SIGNING_CERTIFICATE_P12
         certificate-password-secret: KEYCHAIN_PASSWORD
         certificate-extension: p12
@@ -255,20 +280,34 @@ jobs:
       matrix:
         config: ${{ fromJson(needs.select-targets.outputs.build-matrix) }}
     runs-on: ${{ matrix.config.runs-on }}
+    container: ${{ fromJSON(matrix.config.container) }}
+    defaults:
+      run:
+        # Avoid problems caused by different default shell for container jobs (sh) vs non-container jobs (bash).
+        shell: bash
+
     timeout-minutes: 90
 
     steps:
       - name: Checkout
+        if: fromJSON(matrix.config.container).image == null
         uses: actions/checkout@v4
 
+      - name: Checkout
+        # actions/checkout@v4 has dependency on a higher version of glibc than available in the Linux container.
+        if: fromJSON(matrix.config.container).image != null
+        uses: actions/checkout@v3
+
       - name: Install Node.js
+        if: fromJSON(matrix.config.container).image == null
         uses: actions/setup-node@v3
         with:
           node-version: ${{ env.NODE_VERSION }}
           registry-url: 'https://registry.npmjs.org'
           cache: 'yarn'
 
       - name: Install Python 3.x
+        if: fromJSON(matrix.config.container).image == null
         uses: actions/setup-python@v4
         with:
           python-version: '3.x'
@@ -285,7 +324,6 @@ jobs:
           version: 3.x
 
       - name: Package
-        shell: bash
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           AC_USERNAME: ${{ secrets.AC_USERNAME }}

.github/workflows/check-containers.yml

@@ -0,0 +1,58 @@
+name: Check Containers
+
+on:
+  pull_request:
+    paths:
+      - ".github/workflows/check-containers.ya?ml"
+      - "**.Dockerfile"
+      - "**/Dockerfile"
+  push:
+    paths:
+      - ".github/workflows/check-containers.ya?ml"
+      - "**.Dockerfile"
+      - "**/Dockerfile"
+  repository_dispatch:
+  schedule:
+    # Run periodically to catch breakage caused by external changes.
+    - cron: "0 7 * * MON"
+  workflow_dispatch:
+
+jobs:
+  run:
+    name: Run (${{ matrix.image.path }})
+    runs-on: ubuntu-latest
+    permissions: {}
+    services:
+      registry:
+        image: registry:2
+        ports:
+          - 5000:5000
+
+    env:
+      IMAGE_NAME: name/app:latest
+      REGISTRY: localhost:5000
+
+    strategy:
+      fail-fast: false
+      matrix:
+        image:
+          - path: .github/workflows/assets/linux.Dockerfile
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Build and push to local registry
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          file: ${{ matrix.image.path }}
+          push: true
+          tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+
+      - name: Run container
+        run: |
+          docker \
+            run \
+              --rm \
+              ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

.github/workflows/push-container-images.yml

@@ -0,0 +1,61 @@
+name: Push Container Images
+
+on:
+  pull_request:
+    paths:
+      - ".github/workflows/push-container-images.ya?ml"
+  push:
+    paths:
+      - ".github/workflows/push-container-images.ya?ml"
+      - "**.Dockerfile"
+      - "**/Dockerfile"
+  repository_dispatch:
+  schedule:
+    # Run periodically to catch breakage caused by external changes.
+    - cron: "0 8 * * MON"
+  workflow_dispatch:
+
+jobs:
+  push:
+    name: Push (${{ matrix.image.name }})
+    if: github.repository == 'arduino/arduino-ide'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+
+    strategy:
+      fail-fast: false
+      matrix:
+        image:
+          - path: .github/workflows/assets/linux.Dockerfile
+            name: ${{ github.repository }}/linux
+            registry: ghcr.io
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Log in to the Container registry
+        uses: docker/login-action@v3
+        with:
+          password: ${{ secrets.GITHUB_TOKEN }}
+          registry: ${{ matrix.image.registry }}
+          username: ${{ github.repository_owner }}
+
+      - name: Extract metadata for image
+        id: metadata
+        uses: docker/metadata-action@v5
+        with:
+          images: ${{ matrix.image.registry }}/${{ matrix.image.name }}
+
+      - name: Build and push image
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          file: ${{ matrix.image.path }}
+          labels: ${{ steps.metadata.outputs.labels }}
+          # Workflow is triggered on relevant events for the sake of a "dry run" validation but image is only pushed to
+          # registry on commit to the main branch.
+          push: ${{ github.ref == 'refs/heads/main' }}
+          tags: ${{ steps.metadata.outputs.tags }}