transfer() semantics around preserving resizability

[syg]

    > * https://streams.spec.whatwg.org/#transfer-array-buffer

• https://webidl.spec.whatwg.org/#arraybuffer-transfer , although it does not appear to be used directly yet.

Also add https://html.spec.whatwg.org/multipage/structured-data.html to that list. We'll want this to preserve resizability:

const arrayBuffer = new ArrayBuffer(1024, { maxByteLength: 2048 });
const transferredArrayBuffer = structuredClone(arrayBuffer, { transfer: [arrayBuffer] });
console.log(transferredArrayBuffer.resizable === true);

when we transfer ArrayBuffers on the web platform, we should preserve their resizability. And apparently this spec has an algorithm which does that.

Looks like I misread the explainer. 😅 As currently written, transfer() always returns a non-resizable ArrayBuffer.

Should we change transfer() to preserve resizability? Or should there be a different method (or option for transfer()) which does that? I suppose this answers one of the open questions on this proposal:

Should there be a transferResizable() that reallocs into another resizable ArrayBuffer?

Are there compelling use cases for this?

So yes, there are compelling use cases! 😁

• Readable byte streams use transferring to take ownership of a ArrayBuffer while doing a BYOB read, and then returns ownership back to the user once it's filled the buffer. If the input buffer is resizable, we'll want to preserve that property in the returned buffer.
• Transferring a resizable ArrayBuffer between realms (through postMessage()) should also preserve resizability. One possible use case could be making readable byte streams transferable, allowing the BYOB request to be transferred and filled by another realm. (At the moment, transferring a readable byte stream results in a "default" readable stream without BYOB support, but we should be able to extend the Streams spec to allow for BYOB.)

_Originally posted by @MattiasBuelens in #3

---

Splitting off this question into its own issue. Currently transfer() always produces a fixed-length ArrayBuffer.

It seems good to give it the additional capability to produce resizable buffers. I'd like to defer this to a follow-on proposal, but there is some future proofing to do here.

The two interlinked API design questions I think are:

1. How to tell transfer() what kind of ArrayBuffer to return?
2. Should transfer() preserve the resizability of its receiver by default? If so, how do you override that default behavior?

One natural way to address (1) is to add an options bag to transfer(), mirroring what the constructor accepts: transfer(newLen, { maxByteLength }). If a { maxByteLength } options bag is passed, then transfer returns a resizable buffer, just like the constructor.

However, this design cannot also satisfy (2) if it is decided (2) should preserve resizability by default. Preserving by default means that if the { maxByteLength } options bag is not passed, that it gets this value from the receiver. But if that's the case, there's no way to tell transfer to make a fixed-length ArrayBuffer from a resizable one. And moving a resizable buffer to a fixed one is one of the original use cases: to do a zero-copy move and also free up the reserved virtual memory after resizing is no longer needed.

There are a few choices moving forward:

A. Stick with the current design of transfer(), i.e. always return fixed buffers. When we extend transfer() to produce resizable buffers, an explicit options bag must always be passed. It does not preserve resizability by default.

B. Make transfer() preserve resizability if the options bag is undefined. Since this means you cannot transfer from resizable into a fixed length buffer, add a new method fix() (name up for bikeshed).

C. Have a transfer() preserve resizability if both parameters are undefined. That is, make the nullary transfer() a special overload that perform a "move as is". transfer(newLen) always produces a fixed-length buffer of newLen, and transfer(newLen, { maxByteLength }) always produces a resizable buffer.

I think I prefer A > C >>> B. Thoughts?

[domenic]

My main use case for transfer() is the one described in https://github.com/domenic/proposal-arraybuffer-transfer/tree/d4e00037420b87d0b5662c82b74d56b4ba1562ad#the-problem . From that perspective, the default behavior should be to preserve resizability, and changing the length is a non-goal. (The same goes for Streams's use cases, essentially.)

So to me, I think B > C >> A. From my perspective, it seems like a big win if a method named transfer() always preserves the "type" of the original buffer, and you need a separate method method (fix()) to switch types. Second-best is having the default behavior preserve the type, and in far last place is having the default behavior switch types.

[syg]

Interesting! Thanks for your perspective. I plan to have a quick discussion at the next plenary.

[syg]

Result from November 2022 TC39 is to split out transfer() into its own proposal and demote it to Stage 2 to further explore the design space.

Some variant of (B) above seems to be the most popular option.

Other ideas that were floated:

• Have transfer() always preserve resizability, and have an in-place fix(). (From @jridgewell)
• How far can we go with copy-on-write ArrayBuffers and an in-place detach()? (From @mhofman)

@jridgewell's idea sgtm from an API design perspective, I need to think more about the implementation implications of this.

@mhofman's idea about CoW ABs give me pause about implementation complexity and breaking the "ABs should be implementable as having fixed data pointers from construction, for the sake of security" design goal.

[syg]

The upshot of #4 is that transfer() should not be shipped with the rest of the spec.

cc @Constellation

[syg]

The spun out proposal repo is up at https://github.com/tc39/proposal-arraybuffer-transfer. Closing this thread; please have further discussions in that repo.