Buffering needs to be configurable

[javier-alvarez]

I got this exception because my request was 300MB and I was running in an Azure Cloud Service. Azure Cloud services cannot write to temp storage they need to configure a disk space and write to a specific directory. It was very difficult to find that I had to set a environment variable (ASPNETCORE_TEMP) to configure this. https://github.com/aspnet/HttpAbstractions/blob/07d115400e4f8c7a66ba239f230805f03a14ee3d/src/Microsoft.AspNetCore.Http/Internal/BufferingHelper.cs#L25-L26

Expected: This should be a first class config and documented.

System.IO.IOException: There is not enough space on the disk.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.BeginWriteCore(Byte[] bytes, Int32 offset, Int32 numBytes, AsyncCallback userCallback, Object stateObject)
   at System.IO.FileStream.FlushWrite(Boolean calledFromFinalizer)
   at System.IO.FileStream.BeginWriteAsync(Byte[] array, Int32 offset, Int32 numBytes, AsyncCallback userCallback, Object stateObject)
   at System.IO.FileStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.<ReadAsync>d__35.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.<DrainAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Formatters.JsonInputFormatter.<ReadRequestBodyAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.<BindModelAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.<BindModelAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

[Tratcher]

Yes this needs to be better integrated with the config system.

I do have to ask why you're trying to buffer a 300MB upload? At a certain point buffering becomes impractical and you need to find an alternative algorithm for processing your data.

[javier-alvarez]

We need this to be able to upload machine learning models that are a big binary blob. This is not a frequent call. Only done by admins of the API. We are not worried about scalability of this call.

[Tratcher]

What happens to it once it's buffered? In memory processing, or copied to permanent storage? Any reason those operations can't be streamed?

[javier-alvarez]

In memory processing, we need to do a few lightweight checks in the API before we save it.

[javier-alvarez]

• Is it possible to disable the buffering completely?
• Is it possible to set the maximum body size per API route?

[Tratcher]

Buffering is off by default. How are you reading the request body? Some of the MVC components may opt to buffer.

Yes, there's a RequestSizeLimitAttribute. But watch out for IIS's conflicting limit.

[javier-alvarez]

If you check in the stack trace JsonInputFormatter is in the stack. We are using the usual model binding.
We are using self hosted HttpSys.

[Tratcher]

HttpSys has the same request body size feature.

You're trying to model bind 300MB of JSON? Turning off model binding seems like a more pressing question than turning off buffering. @rynowak

[javier-alvarez]

Yes we are using NSwag to autogenerate the client. It would be awesome if we could auto-generate the same client but with protobuf instead of json. 300MB and it is compressed.

[RehanSaeed]

I'm trying to enable a read-only file system in Docker (See https://github.com/RehanSaeed/ReadOnlyDockerTest). Buffering to a temp file system is not going to work in that scenario.

Some of the MVC components may opt to buffer.

Which ones exactly? Is Buffering only turned on when EnableRewind on the response body is used?

[Tratcher]

The JSON and XML model binders buffer the request body before reading it as they don't have async parsing APIs.

[RehanSaeed]

Will this be the case with the new JSON API's in .NET Core 3?

[Tratcher]

No, that's one of the main things we're planning to fix.

[RehanSaeed]

Microsoft happened to have blogged about this today:
https://devblogs.microsoft.com/aspnet/re-reading-asp-net-core-request-bodies-with-enablebuffering/

[davidfowl]

@Tratcher what's the work here?

[Tratcher]

ASPNETCORE_TEMP is one of the only configurations in the platform that can only be configured as an environment variable, it's not read from IConfiguration. It's consumed by the EnableBuffering APIs:
https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.http.httprequestrewindextensions.enablebuffering?view=aspnetcore-5.0

These APIs need to be updated to resolve the temp directory from DI -> Options -> Config -> Environment.

[davidfowl]

I remember this discussion on how we do this. I'm not sure it's worth the trouble flowing that to these APIs. Do we need better docs about ASPNETCORE_TEMP?

[Tratcher]

It's mentioned in the linked doc.

It stands out because it's the only thing in the framework that's configured this way.

[davidfowl]

I think that's fine, the effort to flow configuration to this part of the stack isn't worth it.

[Tratcher]

Duplicate of #9466

[RehanSaeed]

@Tratcher Since we have async API's for JSON now, Is your earlier statement no longer true and can we now safely enable readonly Docker file systems?

The JSON and XML model binders buffer the request body before reading it as they don't have async parsing APIs.

[Tratcher]

For JSON, yes the async model binding parser is now the default. For XML you still need to buffer.