Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ulimit in self-hosted runner's shell mismatch the ulimit set in machine #3421

Open
igorluppi opened this issue Aug 6, 2024 · 1 comment
Open

ulimit in self-hosted runner's shell mismatch the ulimit set in machine #3421

igorluppi opened this issue Aug 6, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@igorluppi
Copy link

Describe the bug
When ulimit -a is performed in regular machine' shell we got:

POSIX message queues (bytes, -q) 4194304

But Github runners are limiting it to another value. If we run the same command in runner shell:

POSIX message queues (bytes, -q) 819200

The runner is installed in the same machine, although it cannot use the same resource as set.

In our /etc/security/limits.conf we have:

* hard msgqueue 4194304
<$USER> hard msgqueue 4194304
root hard msgqueue 4194304
* soft msgqueue 4194304
<$USER> soft msgqueue 4194304
root soft msgqueue 4194304

How can Github self-hosted runner access the same amount of memory?

To Reproduce
Just run ulimit -a in Github runner shell, if you have changed the default value for the desired ulimit.

Expected behavior
When we run ulimit -a we should see POSIX message queues (bytes, -q) 4194304

Actual behavior
When we run ulimit -a we see POSIX message queues (bytes, -q) 819200

Runner Version and Platform

Version of your runner? Current runner version: '2.317.0'

OS of the machine running the runner? Ubuntu 22.04

What's not working?

> Run ulimit -q 4194304
ulimit -q 4194304 
Error: Process completed with exit code 1.
@igorluppi igorluppi added the bug Something isn't working label Aug 6, 2024
@polarathene
Copy link

In our /etc/security/limits.conf we have

Limits are often managed by systemd now IIRC. This has been the case at least for file descriptors where the default limits (at least for root?) are set to 1024 soft and 524288 hard. On Github CI runners these have been observed as 65536 for both IIRC.

Limits config you reference may be applicable to specific users, you've not shared much context about the self hosted runner regarding that.

See this reference, try grep the filesystem for LimitMSGQUEUE. I could be mistaken, just a hint as I ran into this sort of issue with Docker containers in the past where the limit varied based on distro and caused much confusion to troubleshoot the cause on misbehaving systems.

@kwvg kwvg mentioned this issue Feb 17, 2025
Merged
5 tasks
PastaPastaPasta added a commit to dashpay/dash that referenced this issue Feb 25, 2025
@PastaPastaPasta
Merge #6583: ci: run functional tests on GitHub Actions, upload funct…
379c935 
…ional test logs as artifacts

3461c14 ci: tentatively drop multiprocess and tsan functional tests (Kittywhiskers Van Gogh)
5db8fa0 ci: cache previous releases if running `linux64` variant (Kittywhiskers Van Gogh)
cca0d89 ci: add functional tests for linux builds (Kittywhiskers Van Gogh)
fc2efb6 ci: upload functional test logs as artifacts (Kittywhiskers Van Gogh)
b25e846 ci: handle space exhaustion by deleting files we don't need (Kittywhiskers Van Gogh)
0a1e635 ci: add reusable workflow for running functional tests (Kittywhiskers Van Gogh)
57cf278 ci: use helper script to bundle artifacts (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional Information

  * [`actions/cache`](https://github.com/marketplace/actions/cache) allows specification of directories that want to be cached using glob expressions, which extend further into defining exclusions, needed due keep cache sizes maintainable ([source](https://github.com/dashpay/dash/blob/bb469687d3d936f82fd8e8fbe0934eec5e17df5e/.gitlab-ci.yml#L128-L138)).

    * Unfortunately, the implementation of globbing with respect to exclusions is more-or-less broken (see [actions/toolkit#713](actions/toolkit#713 (comment))) with the requirement that the inclusion depth should match the depth of the exclusion. Attempting to play by these rules more or less fails ([build](https://github.com/kwvg/dash/actions/runs/13344612118/job/37273624710#step:5:4634)).

    * Attempting to use third party actions like [`tj-actions/glob`](https://github.com/marketplace/actions/glob-match) provide for a much more saner experience but they enumerate individual files that match patterns, not folders. This means that when we pass them to `actions/cache`, we breach the arguments length limit ([build](https://github.com/kwvg/dash/actions/runs/13343953711/job/37272121153#step:9:4409)).

      * Modifying `ulimit` to get around this isn't very feasible due to odd behavior surrounding it (see [actions/runner#3421](actions/runner#3421)) and the general consensus is to save to a file and have the next action read from file ([source](https://stackoverflow.com/a/71349472)).

         [`tj-actions/glob`](https://github.com/marketplace/actions/glob-match) graciously does this with the `paths-output-file` output but it takes two to play and [`actions/cache`](https://github.com/marketplace/actions/cache) does not accept files (`path` must be a newline-delimited string).

    The path of least resistance, it seems, is to use a script to bundle our cache into a neat input and leave [`actions/cache`](https://github.com/marketplace/actions/cache) out of it entirely, this is the approach taken.

  * As we aren't using self-hosted runners, we are subject to GitHub's limits for everything, runner space, total artifact storage budget, total cache storage budget.

    * Caches that not **accessed** in 7 days are evicted and there's a 10 GB budget for all caches ([source](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy)) and GitHub will evict oldest (presumably by **creation**?) caches to make sure that limit is adhered to.

      * What makes this limit troubling is the immutable nature of caches as unlike GitLab, which is more conductive to shared caches ([source](https://github.com/dashpay/dash/blob/bb469687d3d936f82fd8e8fbe0934eec5e17df5e/.gitlab-ci.yml#L55-L69)), GitHub insists on its immutability (see [actions/toolkit#505](actions/toolkit#505)) and the only way to "update" a cache is to structure your cache key to allow the updated content to reflect in the key itself or delete the cache and create a new one, which brings race condition concerns ([comment](#6406 (review))).

        Sidenote, overwriting contents are allowed for artifacts ([source](https://github.com/actions/upload-artifact/blob/65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08/README.md#overwriting-an-artifact)), just not caches.

      * This means we need to be proactive in getting rid of caches with a short shelf life to avoid more long lasting caches (like `depends-sources`) from being evicted due to old age as we breach the 10 GB limit. We cannot just set a short retention period as GitHub doesn't offer you to do that with caches like they do with artifacts ([source](https://github.com/actions/upload-artifact/blob/65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08/README.md#retention-period)).

        * ~~While doing this properly would require us to implement a cache reaper workflow, this still needed to be addressed now as the contents of `build-ci` need to be passed onto the functional test workflow and this creates an ephemeral cache with a short lifespan that threatens longer-living (but older) caches.~~

       ~~This is currently approached by deleting `build-ci` (output) caches when the functional test runner is successful, we let the cache stick around if the build fails to allow for rerunning failed instances.~~

       ~~If for whatever reason a successful build has to be rerun, the build workflow would need to be rerun (though the `ccache` cache will speed this up significantly) to generate the output cache again for the test workflow to succeed. Failing to do this will result in a cache miss and run failure.~~ **Edit:** Switched to using artifacts to mitigate cache thrashing concerns. Auto-expiration is a huge plus, too.

    * Runners are limited to 14 GB of **addressable** storage space ([source](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories)) and breaching this limit will cause the runners to fail.

      * Our TSan ([build](https://github.com/kwvg/dash/actions/runs/13355816205/job/37298587344#step:5:1178)) and multiprocess ([build](https://github.com/kwvg/dash/actions/runs/13355816205/job/37298658464#step:5:1190)) test variants breach this limit when collecting logs and deleting the `build-ci` (see 2153b0b) cache doesn't make enough of a dent to help ([build](https://github.com/kwvg/dash/actions/runs/13356474530)).

        * Omitting the logs from successful runs would be a regression in content for our `test_logs` artifacts and therefore wasn't considered.

      * While third-party actions like [`AdityaGarg8/remove-unwanted-software`](https://github.com/marketplace/actions/maximize-build-disk-space-only-remove-unwanted-software) can bring significant space savings ([build](https://github.com/kwvg/dash/actions/runs/13357806504/job/37302970610#step:2:150)), they cannot be run in jobs that utilize the [`container`](https://docs.github.com/en/actions/writing-workflows/choosing-where-your-workflow-runs/running-jobs-in-a-container) context (so, all jobs after the container creation workflow) as they'd be executed inside the container when we want to affect the runner underneath ([build](https://github.com/kwvg/dash/actions/runs/13357260369/job/37301757225#step:3:29), notice the step being run after "Initialize containers").
        * There are no plans to implement definable "pre-steps" (see [actions/runner#812](actions/runner#812)) and the only way to implement "before" and "after" steps is by self-hosting ([source](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/running-scripts-before-or-after-a-job)).

       This has been sidestepped tentatively by omitting TSan and multiprocess builds as any attempted fixes would require precision gutting to get borderline space savings which could easily be nullified by future code changes that occupy more space and such measures are better reserved for future PRs.

    * Artifacts share their storage quota with GitHub Packages ([source](https://docs.github.com/en/billing/managing-billing-for-your-products/managing-billing-for-github-packages/about-billing-for-github-packages#about-billing-for-github-packages)) and artifacts by default linger around for 90 days ([source](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/storing-and-sharing-data-from-a-workflow#about-workflow-artifacts)) and considering that each run can generate multi-gigabyte total artifacts, testing the upper limit runs the risk of being very expensive. **Edit:** It appears pricing is reflective of artifacts in private repos, public repos don't seem to run this risk.

      * ~~All artifacts generated have an expiry of one day (compared to GitLab's three days, [source](https://github.com/dashpay/dash/blob/bb469687d3d936f82fd8e8fbe0934eec5e17df5e/.gitlab-ci.yml#L165), but they are self-hosted).~~ **Edit:** Artifacts now have an expiry of three days, matching GitLab.

      * Artifacts are compressed as ZIP archives and there is no way around that as of now (see [actions/upload-artifact#109](actions/upload-artifact#109 (comment))) and the permissions loss it entails is acknowledged by GitHub and their solution is... to put them in a tarball ([source](https://github.com/actions/upload-artifact/blob/65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08/README.md#permission-loss)).

      To keep size under control, artifacts contain `zstd-5` compressed tarballs (compression level determined by benchmarks, see below) generated by bundling scripts alongside their checksum.

      <details>

      <summary>Benchmarks:</summary>

      ```
      $ zstd -b1 -e22 -T0 artifacts-linux64_multiprocess.tar
      1#_multiprocess.tar :1586411520 -> 537552492 (x2.951), 2396.2 MB/s, 1367.8 MB/s
      2#_multiprocess.tar :1586411520 -> 499098623 (x3.179), 2131.8 MB/s, 1306.6 MB/s
      3#_multiprocess.tar :1586411520 -> 474452284 (x3.344), 1371.6 MB/s, 1245.6 MB/s
      4#_multiprocess.tar :1586411520 -> 470931621 (x3.369),  620.3 MB/s, 1239.1 MB/s
      5#_multiprocess.tar :1586411520 -> 459075785 (x3.456),  457.2 MB/s, 1230.1 MB/s
      6#_multiprocess.tar :1586411520 -> 449594612 (x3.529),  415.3 MB/s, 1289.7 MB/s
      7#_multiprocess.tar :1586411520 -> 446208421 (x3.555),  282.6 MB/s, 1296.3 MB/s
      8#_multiprocess.tar :1586411520 -> 442797797 (x3.583),  254.3 MB/s, 1338.4 MB/s
      9#_multiprocess.tar :1586411520 -> 438690318 (x3.616),  210.8 MB/s, 1331.5 MB/s
      10#_multiprocess.tar :1586411520 -> 437195147 (x3.629),  164.1 MB/s, 1337.4 MB/s
      11#_multiprocess.tar :1586411520 -> 436501141 (x3.634),  108.2 MB/s, 1342.5 MB/s
      12#_multiprocess.tar :1586411520 -> 436405679 (x3.635),  102.7 MB/s, 1344.0 MB/s
      13#_multiprocess.tar :1586411520 -> 436340981 (x3.636),   65.9 MB/s, 1344.0 MB/s
      14#_multiprocess.tar :1586411520 -> 435626720 (x3.642),   61.5 MB/s, 1346.9 MB/s
      15#_multiprocess.tar :1586411520 -> 434882716 (x3.648),   49.4 MB/s, 1352.9 MB/s
      16#_multiprocess.tar :1586411520 -> 411221852 (x3.858),   33.6 MB/s, 1049.2 MB/s
      17#_multiprocess.tar :1586411520 -> 399523001 (x3.971),   26.0 MB/s, 1003.7 MB/s
      18#_multiprocess.tar :1586411520 -> 379278765 (x4.183),   21.0 MB/s,  897.5 MB/s
      19#_multiprocess.tar :1586411520 -> 378022246 (x4.197),   14.7 MB/s,  896.0 MB/s
      20#_multiprocess.tar :1586411520 -> 375741653 (x4.222),   14.0 MB/s,  877.6 MB/s
      21#_multiprocess.tar :1586411520 -> 373303486 (x4.250),   11.9 MB/s,  866.8 MB/s
      22#_multiprocess.tar :1586411520 -> 358172556 (x4.429),   6.09 MB/s,  884.9 MB/s
      ```

      </details>

        > **Note:** As mentioned above, we use similar bundling scripts for the outputs cache but unlike artifacts, we cannot disable their compression routines or even adjust compression levels (see [actions/tookit#544](actions/toolkit#544))

  ## Notes

  * ~~If we add or remove binaries in terms of compile output, `bundle-build.sh` needs to be updated. Without updating it, it will fail if it cannot find the files it was looking for and it will not include files that it wasn't told to include.~~ No longer applicable.

  ## Breaking Changes

  None expected.

  ## Checklist

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)**
  - [x] I have made corresponding changes to the documentation **(note: N/A)**
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  PastaPastaPasta:
    utACK 3461c14
  UdjinM6:
    ACK 3461c14

Tree-SHA512: ef55bc10902c57673ffd9bee6562b362a87658e4c51e543b8553bf48c41544a302d6acad7c5a30395fbfcfd085354251a07327c3e78c93c750585496926be9f6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@polarathene @igorluppi