Generic Writer Support #10
Conversation
This mirrors recent changes submitted for the deserialization API and provides a generic system for custom writer support to control how and where the byte stream generated during serialization is written. The change as-is is fully backwards compatible and passes the existing and expanded test cases. Primary use cases would align with existing async serialization attempts but don't put the complexity of thread management or throttling logic into the library, instead leaving the minutia of how those aspects are handled to callers. It also enables some other advanced use cases such as streaming the data to a destination without buffering it fully beforehand, which could enable efficiencies elsewhere.
|
No comments come to mind other than it's nice to see you again :) |
|
You as well :). I just created a branch named readwrite-staging, what do you think about combining your two PRs and targeting that branch? I really like the flexibility your approach provides here. I do like having at least the async/throttled aspect be part of the API surface directly, though, rather than requiring everyone to implement a custom reader/writer, but maybe the API surface could be implemented on top of this infrastructure rather than the slightly more invasive approach in #11. If you combine and retarget the PRs, I'll merge it and then spend some time looking at how to combine them. |
|
Sounds good - I'll look at doing that some point this week. |
|
Closing in favor of the merged #12 as discussed. |
This mirrors recent changes submitted for the deserialization API (#8) and provides a system for custom writer support, allowing control of how and where the byte stream generated during serialization is written. The change as-is is fully backwards compatible and passes the existing and expanded test cases - which can be used as examples.
Primary use cases would align with existing async serialization attempts (#4, #7) but don't put the complexity of thread management or throttling logic into the library, or risk breaking backwards compatibility. Instead the minutia of how those aspects are handled are left the responsibility of callers. It also enables some other advanced use cases such
as streaming the data to a destination without buffering it fully beforehand, which could enable efficiencies elsewhere.
This PR will likely conflict with #8 slightly if either is merged as some documentation and utility related changes are common between the two; otherwise however I'm happy with this PR as-is and can resolve those conflicts if either is merged.
As with #8 this defines a "Writer" protocol which expects that - at minimum - an object be supplied via the options table to
SerializeEx()which provides aWriteStringfunction. If this object is not supplied, or if the object supplied does not meet this requirement, it will be ignored and the existing approach of writing strings to a temporary buffer will continue to be used.WriteString:
function(writer, str)(required)This function will be called each time the library submits a byte string that was created as result of serializing data.
Flush:
function(writer)(optional)If specified, this function will be called at the end of the serialization process. It may return any number of values - including zero - all of which will be passed through to the caller of
SerializeEx()verbatim.The default behavior if this function is not specified - and if the writer is otherwise valid - is a no-op that returns no values. In such a case it is assumed that the WriteString method will have either passed the strings it received on elsewhere, which seems a reasonable default given the advanced nature of this feature.
As with the deserialization/reader PR, this requires relatively minimal changes to the internals of the library - just a few changes to how a couple of operations are delegated.