Increase serialization max size or make it user driven for SystemTextJson

[BRLN1]

Background and motivation

I'm trying to serialize a big pdf of around 200MB, while trying to do so I'm getting following error:

---> System.ArgumentException: The JSON value of length 304063897 is too large and not supported.\r\n
   at System.Text.Json.ThrowHelper.ThrowArgumentException_ValueTooLarge(Int32 tokenLength)\r\n
   at System.Text.Json.Utf8JsonWriter.WriteStringValue(ReadOnlySpan`1 value)\r\n
   at System.Text.Json.Serialization.Converters.StringConverter.Write(Utf8JsonWriter writer, String value, JsonSerializerOptions options)\r\n
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)\r\n
   at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)\r\n
   at System.Text.Json.Serialization.JsonConverter`1.WriteCoreAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)\r\n
   at System.Text.Json.JsonSerializer.WriteCore[TValue](JsonConverter jsonConverter, Utf8JsonWriter writer, TValue& value, JsonSerializerOptions options, WriteStack& state)\r\n
   at System.Text.Json.JsonSerializer.WriteAsyncCore[TValue](Stream utf8Json, TValue value, Type inputType, JsonSerializerOptions options, CancellationToken cancellationToken)\r\n
   at System.Net.Http.Json.JsonContent.SerializeToStreamAsyncCore(Stream targetStream, Boolean async, CancellationToken cancellationToken)\r\n
   at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)\r\n
   at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)\r\n
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)\r\n

I found out that current size limitation for that is 125MB. That is definitely too little. I'd make sense to inscrease the cap, possibly to 2GB(like string that contains base64 of pdf or post request) or simply make that user defined.

If I'm missing any solution to that problem, I'd appreciate some tips,
Thanks!

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

I'm trying to serialize a big pdf of around 200MB, while trying to do so I'm getting following error:
---> System.ArgumentException: The JSON value of length 304063897 is too large and not supported.\r\n
at System.Text.Json.ThrowHelper.ThrowArgumentException_ValueTooLarge(Int32 tokenLength)\r\n
at System.Text.Json.Utf8JsonWriter.WriteStringValue(ReadOnlySpan1 value)\r\n at System.Text.Json.Serialization.Converters.StringConverter.Write(Utf8JsonWriter writer, String value, JsonSerializerOptions options)\r\n at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)\r\n
at System.Text.Json.Serialization.JsonConverter1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)\r\n at System.Text.Json.Serialization.JsonConverter1.WriteCoreAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)\r\n
at System.Text.Json.JsonSerializer.WriteCore[TValue](JsonConverter jsonConverter, Utf8JsonWriter writer, TValue& value, JsonSerializerOptions options, WriteStack& state)\r\n
at System.Text.Json.JsonSerializer.WriteAsyncCore[TValue](Stream utf8Json, TValue value, Type inputType, JsonSerializerOptions options, CancellationToken cancellationToken)\r\n
at System.Net.Http.Json.JsonContent.SerializeToStreamAsyncCore(Stream targetStream, Boolean async, CancellationToken cancellationToken)\r\n
at System.Net.Http.HttpContent.g__WaitAsync|56_0(ValueTask copyTask)\r\n
at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)\r\n
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)\r\n

I found out that current size limitation for that is 125MB. That is definitely too little. I'd make sense to inscrease the cap, possibly to 2GB(like string that contains base64 of pdf or post request) or simply make that user defined.

If I'm missing any solution to that problem, I'd appreciate some tips,
Thanks!

API Proposal

namespace System.Collections.Generic
{
    public class MyFancyCollection<T> : IEnumerable<T>
    {
        public void Fancy(T item);
    }
}

API Usage

// Fancy the value
var c = new MyFancyCollection<int>();
c.Fancy(42);

// Getting the values out
foreach (var v in c)
    Console.WriteLine(v);

Alternative Designs

No response

Risks

No response

Author:	BRLN1
Assignees:	-
Labels:	api-suggestion, area-System.Text.Json, untriaged
Milestone:	-

[eiriktsarpalis]

This appears to be by design:

runtime/src/libraries/System.Text.Json/src/System/Text/Json/JsonConstants.cs

Lines 71 to 78 in c690d27

	// The maximum number of characters allowed when writing raw UTF-16 JSON. This is the maximum length that we can guarantee can
	// be safely transcoded to UTF-8 and fit within an integer-length span, given the max expansion factor of a single character (3).
	public const int MaxUtf16RawValueLength = int.MaxValue / MaxExpansionFactorWhileTranscoding;
	public const int MaxEscapedTokenSize = 1_000_000_000; // Max size for already escaped value.
	public const int MaxUnescapedTokenSize = MaxEscapedTokenSize / MaxExpansionFactorWhileEscaping; // 166_666_666 bytes
	public const int MaxBase64ValueTokenSize = (MaxEscapedTokenSize >> 2) * 3 / MaxExpansionFactorWhileEscaping; // 125_000_000 bytes
	public const int MaxCharacterTokenSize = MaxEscapedTokenSize / MaxExpansionFactorWhileEscaping; // 166_666_666 characters

Basically the restriction is in place in order to guarantee that the escaped token does fit in a single span segment. That being said, the constant seems to be derived from the rather pessimistic assumption that every single character in the input string needs escaping, in the worst expansion possible.

I wonder if delaying the check until after we have detected that the string needs escaping might make sense here:

runtime/src/libraries/System.Text.Json/src/System/Text/Json/Writer/Utf8JsonWriter.WriteValues.String.cs

Lines 82 to 96 in 4e7cf80

	private void WriteStringEscape(ReadOnlySpan<char> value)
	{
	int valueIdx = JsonWriterHelper.NeedsEscaping(value, _options.Encoder);
	Debug.Assert(valueIdx >= -1 && valueIdx < value.Length);
	if (valueIdx != -1)
	{
	WriteStringEscapeValue(value, valueIdx);
	}
	else
	{
	WriteStringByOptions(value);
	}
	}

cc @ahsonkhan @bartonjs @krwq

[Tornhoof]

Not a solution, but maybe a workaround, if you need to serialize the pdf for Elasticsearch ingest pipelines, you can also use CBOR to send the data, instead of json.