[C#] Stream file uploads instead of copy all bytes into memory

[hacki11]

ApiClient::ParameterToFile will copy complete file into ram before upload to rest service.

what about streaming a file to avoid OutOfMemory Exceptions on large files like:

public FileParameter ParameterToFile(string name, Stream stream)
        {
            var fp = new FileParameter
            {
                Writer = s =>
                {
                    using (StreamReader file = new StreamReader(stream))
                    {
                        file.BaseStream.CopyTo(s);
                    }
                },
                Name = name
            };


            if (stream is FileStream)
                fp.FileName = Path.GetFileName(((FileStream) stream).Name);
            else
                fp.FileName = "no_file_name_provided";
            return fp;
        }

current implementation:
ApiClient.mustache

if (stream is FileStream)
                return FileParameter.Create(name, ReadAsBytes(stream), Path.GetFileName(((FileStream)stream).Name));
            else
                return FileParameter.Create(name, ReadAsBytes(stream), "no_file_name_provided");

[wing328]

@hacki11 thanks for the suggestion. Do you have cycle to submit a PR with the suggested change?

Ref: https://github.com/swagger-api/swagger-codegen/blob/master/CONTRIBUTING.md

[hacki11]

Unfortunately, RestSharp 105.1.0 loads the complete body into memory before send.
The current RestSharp version has an issue not setting the Content-Length attribute where the rest server gets an eof exception....

[wing328]

@hacki11 are you referring to this issue? restsharp/RestSharp#814

Roughly how large is your file?

[hacki11]

@wing328 i did not know there is already an issue - nice investigation!

it can easily grow to several hundred MB to 1 GB.
Due to migrating my project to C# (java before) the problem arises. Java can stream perfectly with jaxrs.

[wing328]

@hacki11 you may want to try compiling the latest version (unofficial release) as another user has confirmed that it did address some of the issue with file upload: #2175

[hacki11]

i compiled the RestSharp master and the EOF Error on server side due to missing Content-Length is fixed now.
But the execute function of RestSharp seems to still load the whole stream into memory before streaming the content to the server.
There was already a stackoverflow discussion SO but it did not work for me

[wing328]

If you perform a debug, did you hit the following line mentioned in the SO comment?

https://github.com/restsharp/RestSharp/blob/62335a677c92ad45d9dc3214ff6765b597d8642f/RestSharp/Http.Sync.cs#L226

[hacki11]

yes, I hit that line and it uses the Action of the FileParameter Writer Property.
After RestSharp ran the line above file.BaseStream.CopyTo(s); the process memory fills up with the amount of the file that is sent (tested with 250mb file).
i did assume that with this stream, the file will not be loaded completely into the ram.
i am not sure where the issue lies exactly.

[wing328]

@hacki11 I would suggest 2 things:

1. test with just RestSharp with a simple script to upload large files to see if you're experiencing the same so as so isolate the problem with RestSharp

2. upload a larger file (e.g. 2GB) to make sure you get the out of memory exception (so as to confirm the RestSharp is not smart enough to stream large files by not loading the whole file into memory)

[jimschubert]

It looks like RestSharp supports file streaming via WriteMultipartFormData. You'd have to set AlwaysMultipartFormData = true to hit this case:

RestRequest request = new RestRequest("/upload", Method.POST) 
{ 
  AlwaysMultipartFormData = true 
};

[wing328]

@hacki11 did you have a chance to review the feedback from @jimschubert and give it a try?

[hacki11]

it doesn't make a difference setting AlwaysMultipartFormData = true because the caller of the WriteMultipartFormData function checks the HasFiles flag which is true in my case:

Http.Sync.cs:
if (this.HasFiles || this.AlwaysMultipartFormData)
{
    this.WriteMultipartFormData(requestStream);
}

For completeness i set the AlwaysMultipartFormData = true, but the process consumed 1GB memory like the file to be uploaded.

Debugging makes me wonder where the issue lies.
WriteMultipartFormData tries to write the stream to the webrequest stream (seems correct to me):

This function calls my Writer Action in the FileParameter object:

After executing the CopyTo function memory fills up

---

After some tests i end up with a OutOfMemory Exception already at this line using (Stream requestStream = webRequest.GetRequestStream()) but can't tell really why.

[jimschubert]

@hacki11 thanks for your investigation work.

Looks like you'd get the exception on that line because ParameterToFile is flushing the stream.

Do you have a test project repo you used for investigation, or were you testing against your codebase?

[chenzhekl]

Seems to be related to this restsharp/RestSharp#889

[wing328]

@chenzhekl thanks for the pointer. You may be interested in this task as well: #7012

[nevergiveupc]

Is this problem solved or not? @hacki11 @wing328

[alexeyzimarev]

Don't use FileParameter. Use AddFile overload that accepts the stream writer delegate.