System.IO.IOException: No space left on device

[pemistahl]

Description

A GitHub workflow of mine tried to produce 61 Python extension modules for a Rust program. The very last one failed with the following stacktrace:

You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 32 MB

System.IO.IOException: No space left on device : '/home/runner/runners/2.308.0/_diag/Worker_20230824-111853-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
System.IO.IOException: No space left on device : '/home/runner/runners/2.308.0/_diag/Worker_20230824-111853-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Common.Tracing.Error(Exception exception)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/runners/2.308.0/_diag/Worker_20230824-111853-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at System.Diagnostics.TraceSource.Flush()
   at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing)
   at GitHub.Runner.Common.TraceManager.Dispose()
   at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing)
   at GitHub.Runner.Common.HostContext.Dispose()
   at GitHub.Runner.Worker.Program.Main(String[] args)

I have never encountered something like this before. I read that there is gigabytes of space available for the runner, so how did I manage to consume all the available space with my workflow? Is it the Rust or Python toolchains that were installed by the workflow? What can I do to prevent this problem in the future? Is 61 Python modules simply too much?

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

https://github.com/pemistahl/grex/actions/runs/5962663607

Is it regression?

I think so.

Expected behavior

I expected all Python extension modules to be built and uploaded successfully.

Actual behavior

The build of the very last extension module has failed because of too few disk space.

Repro steps

See the workflow: https://github.com/pemistahl/grex/actions/runs/5962663607

[ilia-shipitsin]

@pemistahl , your concern is valid, failed github worker leaves no chance to add any debug step.

the easy thing to try is adding "df -h" as a first step, it will show how much space is avaialable before build.

[ilia-shipitsin]

@pemistahl , I forked your repo and added "df -h" in the very beginning

https://github.com/ilia-shipitsin/grex/actions/runs/5976435738/job/16214256839#step:4:13

as you can see, there's 22Gb free.
we cannot tell what happens to that space, I suspect it was utilized by build process.

can you try to add "du" somewhere before build fails (as well as "df -h" step in the beginning)? (because for failed build github agent is not functional)

[ilia-shipitsin]

it is interesting that build succeeded in my fork

https://github.com/ilia-shipitsin/grex/actions

I talked to my colleagues, they told me that "sometimes" vm is provisioned with small disk. it is bug.

I suggest to wait for "df -h" output from you. Whether it shows enough space or not.

[pemistahl]

Hi @ilia-shipitsin, thanks a lot for your quick response. It turns out that the Maturin GitHub action is responsible for this problem. It is not the runner's fault. I've opened a bug report there already. So I think this issue here can be closed.