Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

run specified steps outside of container before the container is launched #812

Closed
igagis opened this issue Nov 18, 2020 · 3 comments
Closed
Labels
enhancement New feature or request

Comments

@igagis
Copy link

igagis commented Nov 18, 2020

Let's consider the following workflow:

name: ci
on: [push, pull_request]
jobs:
  linux:
    runs-on: ubuntu-latest
    container: debian:buster
    steps:
      - name: git clone
        uses: actions/checkout@main
      - name: build
        run: make

for job linux it will first launch the container, then checkout the source code (inside of the already running container?), then run the build step inside of the container.

What I want to be able to do is to add some pre-steps which would be run outside of the container and before the container is launched. In particular, I need this to install qemu and binfmt packages.
So, the workflow would look something like this:

name: ci
on: [push, pull_request]
jobs:
  linux:
    runs-on: ubuntu-latest
    container: igagis/my_image:arm
    pre-steps:
      - name: install qemu and binfmt
        run: apt update && apt install --yes binfmt-support qemu-user-static
    steps:
      - name: git clone
        uses: actions/checkout@main
      - name: build
        run: make

Note, that I changed the container image to my custom built image for ARM architecture, i.e. all binaries inside of the image are ARM binaries.
After installing qemu and binfmt it is possible to run ARM binaries on x86_64 architecture, using emulation. This approach works for me on Travis-CI, but there I start the docker manually and, thus, I have full control over what I run in host environment and what inside of the container. Now I'm trying to move to github actions and have this problem that it is only possible to either run everything inside of the container or everything in host ubuntu virtual machine.

@igagis igagis added the enhancement New feature or request label Nov 18, 2020
@umarcor
Copy link

umarcor commented Nov 20, 2020

@igagis, the following alternatives might work for you. You can optionally setup QEMU for a target architecture only (see dbhi/qus).

name: ci
on: [push, pull_request]
jobs:
  linux:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@main

      - uses: dbhi/qus/action@master

      - name: build
        uses: docker://debian:buster
        with:
          args: make
name: ci
on: [push, pull_request]
jobs:
  linux:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@main

      - name: Setup QEMU
        run: docker run --rm --privileged aptman/qus -s -- -p

      - name: build
        uses: docker://debian:buster
        with:
          args: make
name: ci
on: [push, pull_request]
jobs:
  linux:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@main

      - name: Setup QEMU
        run: apt update && apt install --yes binfmt-support qemu-user-static

      - name: build
        uses: docker://debian:buster
        with:
          args: make

@igagis
Copy link
Author

igagis commented Nov 20, 2020

Thanks!
I have also submitted actions/runner-images#2095

I understand that these pre-steps do not look like beautiful solution.

@mirabilos
Copy link

This is more difficult than it seems: GitHub fucks up argument quoting and passing, so the step.with.args cannot realistically be more than a simple shell script invocation, definitely not an inline command :/

PastaPastaPasta added a commit to dashpay/dash that referenced this issue Feb 25, 2025
…ional test logs as artifacts

3461c14 ci: tentatively drop multiprocess and tsan functional tests (Kittywhiskers Van Gogh)
5db8fa0 ci: cache previous releases if running `linux64` variant (Kittywhiskers Van Gogh)
cca0d89 ci: add functional tests for linux builds (Kittywhiskers Van Gogh)
fc2efb6 ci: upload functional test logs as artifacts (Kittywhiskers Van Gogh)
b25e846 ci: handle space exhaustion by deleting files we don't need (Kittywhiskers Van Gogh)
0a1e635 ci: add reusable workflow for running functional tests (Kittywhiskers Van Gogh)
57cf278 ci: use helper script to bundle artifacts (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional Information

  * [`actions/cache`](https://github.com/marketplace/actions/cache) allows specification of directories that want to be cached using glob expressions, which extend further into defining exclusions, needed due keep cache sizes maintainable ([source](https://github.com/dashpay/dash/blob/bb469687d3d936f82fd8e8fbe0934eec5e17df5e/.gitlab-ci.yml#L128-L138)).

    * Unfortunately, the implementation of globbing with respect to exclusions is more-or-less broken (see [actions/toolkit#713](actions/toolkit#713 (comment))) with the requirement that the inclusion depth should match the depth of the exclusion. Attempting to play by these rules more or less fails ([build](https://github.com/kwvg/dash/actions/runs/13344612118/job/37273624710#step:5:4634)).

    * Attempting to use third party actions like [`tj-actions/glob`](https://github.com/marketplace/actions/glob-match) provide for a much more saner experience but they enumerate individual files that match patterns, not folders. This means that when we pass them to `actions/cache`, we breach the arguments length limit ([build](https://github.com/kwvg/dash/actions/runs/13343953711/job/37272121153#step:9:4409)).

      * Modifying `ulimit` to get around this isn't very feasible due to odd behavior surrounding it (see [actions/runner#3421](actions/runner#3421)) and the general consensus is to save to a file and have the next action read from file ([source](https://stackoverflow.com/a/71349472)).

         [`tj-actions/glob`](https://github.com/marketplace/actions/glob-match) graciously does this with the `paths-output-file` output but it takes two to play and [`actions/cache`](https://github.com/marketplace/actions/cache) does not accept files (`path` must be a newline-delimited string).

    The path of least resistance, it seems, is to use a script to bundle our cache into a neat input and leave [`actions/cache`](https://github.com/marketplace/actions/cache) out of it entirely, this is the approach taken.

  * As we aren't using self-hosted runners, we are subject to GitHub's limits for everything, runner space, total artifact storage budget, total cache storage budget.

    * Caches that not **accessed** in 7 days are evicted and there's a 10 GB budget for all caches ([source](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy)) and GitHub will evict oldest (presumably by **creation**?) caches to make sure that limit is adhered to.

      * What makes this limit troubling is the immutable nature of caches as unlike GitLab, which is more conductive to shared caches ([source](https://github.com/dashpay/dash/blob/bb469687d3d936f82fd8e8fbe0934eec5e17df5e/.gitlab-ci.yml#L55-L69)), GitHub insists on its immutability (see [actions/toolkit#505](actions/toolkit#505)) and the only way to "update" a cache is to structure your cache key to allow the updated content to reflect in the key itself or delete the cache and create a new one, which brings race condition concerns ([comment](#6406 (review))).

        Sidenote, overwriting contents are allowed for artifacts ([source](https://github.com/actions/upload-artifact/blob/65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08/README.md#overwriting-an-artifact)), just not caches.

      * This means we need to be proactive in getting rid of caches with a short shelf life to avoid more long lasting caches (like `depends-sources`) from being evicted due to old age as we breach the 10 GB limit. We cannot just set a short retention period as GitHub doesn't offer you to do that with caches like they do with artifacts ([source](https://github.com/actions/upload-artifact/blob/65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08/README.md#retention-period)).

        * ~~While doing this properly would require us to implement a cache reaper workflow, this still needed to be addressed now as the contents of `build-ci` need to be passed onto the functional test workflow and this creates an ephemeral cache with a short lifespan that threatens longer-living (but older) caches.~~

       ~~This is currently approached by deleting `build-ci` (output) caches when the functional test runner is successful, we let the cache stick around if the build fails to allow for rerunning failed instances.~~

       ~~If for whatever reason a successful build has to be rerun, the build workflow would need to be rerun (though the `ccache` cache will speed this up significantly) to generate the output cache again for the test workflow to succeed. Failing to do this will result in a cache miss and run failure.~~ **Edit:** Switched to using artifacts to mitigate cache thrashing concerns. Auto-expiration is a huge plus, too.

    * Runners are limited to 14 GB of **addressable** storage space ([source](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories)) and breaching this limit will cause the runners to fail.

      * Our TSan ([build](https://github.com/kwvg/dash/actions/runs/13355816205/job/37298587344#step:5:1178)) and multiprocess ([build](https://github.com/kwvg/dash/actions/runs/13355816205/job/37298658464#step:5:1190)) test variants breach this limit when collecting logs and deleting the `build-ci` (see 2153b0b) cache doesn't make enough of a dent to help ([build](https://github.com/kwvg/dash/actions/runs/13356474530)).

        * Omitting the logs from successful runs would be a regression in content for our `test_logs` artifacts and therefore wasn't considered.

      * While third-party actions like [`AdityaGarg8/remove-unwanted-software`](https://github.com/marketplace/actions/maximize-build-disk-space-only-remove-unwanted-software) can bring significant space savings ([build](https://github.com/kwvg/dash/actions/runs/13357806504/job/37302970610#step:2:150)), they cannot be run in jobs that utilize the [`container`](https://docs.github.com/en/actions/writing-workflows/choosing-where-your-workflow-runs/running-jobs-in-a-container) context (so, all jobs after the container creation workflow) as they'd be executed inside the container when we want to affect the runner underneath ([build](https://github.com/kwvg/dash/actions/runs/13357260369/job/37301757225#step:3:29), notice the step being run after "Initialize containers").
        * There are no plans to implement definable "pre-steps" (see [actions/runner#812](actions/runner#812)) and the only way to implement "before" and "after" steps is by self-hosting ([source](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/running-scripts-before-or-after-a-job)).

       This has been sidestepped tentatively by omitting TSan and multiprocess builds as any attempted fixes would require precision gutting to get borderline space savings which could easily be nullified by future code changes that occupy more space and such measures are better reserved for future PRs.

    * Artifacts share their storage quota with GitHub Packages ([source](https://docs.github.com/en/billing/managing-billing-for-your-products/managing-billing-for-github-packages/about-billing-for-github-packages#about-billing-for-github-packages)) and artifacts by default linger around for 90 days ([source](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/storing-and-sharing-data-from-a-workflow#about-workflow-artifacts)) and considering that each run can generate multi-gigabyte total artifacts, testing the upper limit runs the risk of being very expensive. **Edit:** It appears pricing is reflective of artifacts in private repos, public repos don't seem to run this risk.

      * ~~All artifacts generated have an expiry of one day (compared to GitLab's three days, [source](https://github.com/dashpay/dash/blob/bb469687d3d936f82fd8e8fbe0934eec5e17df5e/.gitlab-ci.yml#L165), but they are self-hosted).~~ **Edit:** Artifacts now have an expiry of three days, matching GitLab.

      * Artifacts are compressed as ZIP archives and there is no way around that as of now (see [actions/upload-artifact#109](actions/upload-artifact#109 (comment))) and the permissions loss it entails is acknowledged by GitHub and their solution is... to put them in a tarball ([source](https://github.com/actions/upload-artifact/blob/65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08/README.md#permission-loss)).

      To keep size under control, artifacts contain `zstd-5` compressed tarballs (compression level determined by benchmarks, see below) generated by bundling scripts alongside their checksum.

      <details>

      <summary>Benchmarks:</summary>

      ```
      $ zstd -b1 -e22 -T0 artifacts-linux64_multiprocess.tar
      1#_multiprocess.tar :1586411520 -> 537552492 (x2.951), 2396.2 MB/s, 1367.8 MB/s
      2#_multiprocess.tar :1586411520 -> 499098623 (x3.179), 2131.8 MB/s, 1306.6 MB/s
      3#_multiprocess.tar :1586411520 -> 474452284 (x3.344), 1371.6 MB/s, 1245.6 MB/s
      4#_multiprocess.tar :1586411520 -> 470931621 (x3.369),  620.3 MB/s, 1239.1 MB/s
      5#_multiprocess.tar :1586411520 -> 459075785 (x3.456),  457.2 MB/s, 1230.1 MB/s
      6#_multiprocess.tar :1586411520 -> 449594612 (x3.529),  415.3 MB/s, 1289.7 MB/s
      7#_multiprocess.tar :1586411520 -> 446208421 (x3.555),  282.6 MB/s, 1296.3 MB/s
      8#_multiprocess.tar :1586411520 -> 442797797 (x3.583),  254.3 MB/s, 1338.4 MB/s
      9#_multiprocess.tar :1586411520 -> 438690318 (x3.616),  210.8 MB/s, 1331.5 MB/s
      10#_multiprocess.tar :1586411520 -> 437195147 (x3.629),  164.1 MB/s, 1337.4 MB/s
      11#_multiprocess.tar :1586411520 -> 436501141 (x3.634),  108.2 MB/s, 1342.5 MB/s
      12#_multiprocess.tar :1586411520 -> 436405679 (x3.635),  102.7 MB/s, 1344.0 MB/s
      13#_multiprocess.tar :1586411520 -> 436340981 (x3.636),   65.9 MB/s, 1344.0 MB/s
      14#_multiprocess.tar :1586411520 -> 435626720 (x3.642),   61.5 MB/s, 1346.9 MB/s
      15#_multiprocess.tar :1586411520 -> 434882716 (x3.648),   49.4 MB/s, 1352.9 MB/s
      16#_multiprocess.tar :1586411520 -> 411221852 (x3.858),   33.6 MB/s, 1049.2 MB/s
      17#_multiprocess.tar :1586411520 -> 399523001 (x3.971),   26.0 MB/s, 1003.7 MB/s
      18#_multiprocess.tar :1586411520 -> 379278765 (x4.183),   21.0 MB/s,  897.5 MB/s
      19#_multiprocess.tar :1586411520 -> 378022246 (x4.197),   14.7 MB/s,  896.0 MB/s
      20#_multiprocess.tar :1586411520 -> 375741653 (x4.222),   14.0 MB/s,  877.6 MB/s
      21#_multiprocess.tar :1586411520 -> 373303486 (x4.250),   11.9 MB/s,  866.8 MB/s
      22#_multiprocess.tar :1586411520 -> 358172556 (x4.429),   6.09 MB/s,  884.9 MB/s
      ```

      </details>

        > **Note:** As mentioned above, we use similar bundling scripts for the outputs cache but unlike artifacts, we cannot disable their compression routines or even adjust compression levels (see [actions/tookit#544](actions/toolkit#544))

  ## Notes

  * ~~If we add or remove binaries in terms of compile output, `bundle-build.sh` needs to be updated. Without updating it, it will fail if it cannot find the files it was looking for and it will not include files that it wasn't told to include.~~ No longer applicable.

  ## Breaking Changes

  None expected.

  ## Checklist

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)**
  - [x] I have made corresponding changes to the documentation **(note: N/A)**
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  PastaPastaPasta:
    utACK 3461c14
  UdjinM6:
    ACK 3461c14

Tree-SHA512: ef55bc10902c57673ffd9bee6562b362a87658e4c51e543b8553bf48c41544a302d6acad7c5a30395fbfcfd085354251a07327c3e78c93c750585496926be9f6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants