Actions stuck on "Job is about to start running on the hosted runner" for hours

[dimanlm]

Describe the bug
Ubuntu latest actions get stuck on "Job is about to start running on the hosted runner: Hosted Agent (hosted)". Usually after 3 hours it says if the build passed or not.
This problem started after I added more tests to my project (NodeJS - Jest)

To Reproduce
Steps to reproduce the behavior:

1. Push a new commit
2. Run "Actions"
3. See error

Expected behavior
There are 24 tests, and usually it takes 15 seconds (maximum) to run them locally.

Runner Version and Platform

Version of your runner? Ubuntu-latest

Jobs in the workflow:

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [16.x] 
    steps:
    - uses: actions/checkout@v2
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v2
      with:
        node-version: ${{ matrix.node-version }}
        cache: 'npm'
    
    - name: Install Node dependencies
      run: npm i

    - name: Run unit and integration tests
      run: npm test

    - name: SonarCloud scan
      uses: sonarsource/sonarcloud-github-action@master
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

What's not working?

"Run unit and integration tests" final log:

Job Log Output

[nikola-jokic]

Hi @dimanlm, thanks for submitting this issue.

Could you please provide _diag file so we can investigate what is causing this issue?

[dimanlm]

Hello @nikola-jokic, thank you for your answer.

I suppose _diag file is generated on self-hosted runners. That's not my case, so I attach you the logs that I could download from the last run.
pipeline_log.txt
logs_42(1).zip

[lithiumtoast]

Hey @dimanlm,

I had the same issue. I was able to use ubuntu-18.04 instead of ubuntu-latest which then I could get my workflow running.

[dimanlm]

Hey @dimanlm,

I had the same issue. I was able to use ubuntu-18.04 instead of ubuntu-latest which then I could get my workflow running.

Hi @lithiumtoast ,

Thank you for your reply. I found out that open handles after jest tests were causing Github Actions runner to be blocked at this step waiting for the timeout. Those open handles were left by mongoose (mongoose.connect and .connection.close()). To fix this issue, I created a test-teardown-globals.js file, as suggested on this thread: ladjs/supertest#520 (comment)
I will keep your suggestion to put ubuntu-18.4 in my workflow though :)

[darius-sas]

In my case the problem was that the server running the self-hosted runner ran out of space (full of docker images!).
Maybe this is useful to someone.

[subooom]

Hey, I initially created the directory under /home/ubuntu and it was all working. I later decided to move the directory to /var/www/html and it stopped working.

Waiting for a runner to pick up this job...

[jscherb1]

@darius-sas - how did you identify the server was full of docker images? And what did you do to remove those images?

[darius-sas]

@darius-sas - how did you identify the server was full of docker images? And what did you do to remove those images?

Just out of luck actually...

You can use docker system prune to remove all unused images and containers! You can Google yourself how to remove old images

[therealdwright]

I have experienced this issue since the closure of this issue as recently as 20 minutes ago.

[roskelleycj]

I have experienced this issue since the closure of this issue as recently as 20 minutes ago.

Ditto, just 30m ago this issue has occurred. And https://www.githubstatus.com/ shows no issues.

Seems like an issue in the GitHub Action infrastructure.

[nasirky]

It is currently happening for us as well on certain workflows (we are using self-hosted runners)

[qpwo]

My whole host machine gets totally frozen every day or two and i have to hard reboot it

[dev-rinchin]

I had the same issue right now for macos-latest

https://github.com/sb-ai-lab/LightAutoML/actions/runs/10386491995/job/28757888335
logs_27165338198.zip