encoding small message has buffer bytelength of 8192

[redgetan]

protobuf.js version: 6.8.0

I'm encoding a very small payload and having it sent over socket io as an arraybuffer. However, for some reason, the bytelength of the underlying buffer of my encoded message is 8192, which you can see in this example (https://jsfiddle.net/fLp62wvx/1/) even though the payload is very small. Is this expected behavior?

[dcodeIO]

This is because the library uses a node-like buffer pool internally. The allocated backing buffer is 8k, but byteOffset and byteLength of the returned Uint8Array point to just the relevant part of it.

[amc6]

Is there a "correct" way to handle this in the browser? I've worked around it by copying it to a new Uint8Array, but that seems suboptimal.

[dcodeIO]

When working with unknown-length data, using a buffer pool has its advantages. For instance, node.js also uses one but also provides APIs that accept views (there: Buffers), unlike WebSocket, which doesn't accept views (here: Uint8Arrays) in its .send() method - I still wonder why. Nonetheless, the performance benefit gained by using a buffer pool should still outweigh the final copying step, because the final length is known then.

Of course you can also experiment with this yourself by overriding protobuf.util.pool (see link above for a reference) with your own implementation (i.e. one that doesn't pool at all).

[amc6]

Makes sense. Thanks

[sergerusso]

quick workaround to get ArrayBuffer with correct length
Uint8ArrayInstance.slice().buffer

[chengsu]

I'm sure that my messages are all below 512 bytes. Does it matters if I change the pool size to 512 ? Should this be a configuration ? May 8192 cause the memery usage grow fast?

[larrymyers]

I can confirm that @sergerusso fix with slice() works without issue. No need to call slice().buffer since XHR2 supports data views directly.

This prevent the proto unmarshall errors due to the server receiving a buffer that has a bunch of trailing zeros for padding out the internal pool that protobuf.js uses.

[rynop]

I'm trying to get this working on IE11 and the Uint8Array object has no method slice(). I know IE11 has no marketshare, but its a requirement of my client.

Anyone have an idea for a workaround?

[dcodeIO]

Does IE11 have .subarray? Should be roughly the same.

[rynop]

The following polyfill gets around the slice() err on IE11.

// Polyfill for Uint8Array.slice for IE and Safari
if (!Uint8Array.prototype.slice) {
    Object.defineProperty(Uint8Array.prototype, 'slice', {
        value: Array.prototype.slice
    });
}

However, I don't think it actually works. Because when I POST my protobuf to my RPC server (Twirp), my server throws the following 500 err:

failed to parse request proto: proto: can't skip unknown wire type 7. And for some reason I can't get at what exactly was POSTed from IE11 dev console.

Without getting this thread too far off track, @dcodeIO does protobuf.js support IE11? If so I'll keep going down the rabbit hole.

[dcodeIO]

I think it should, but can't give any hard guarantees. Such errors usually appear when the data is either truncated or not encoded properly, with the latter usually happening when binary becomes somehow converted to (utf8) text. This must be avoided, i.e. by encoding the data as base64 before POSTing it, and decoding again at the receiver's end.

[rynop]

I figured it out. The typedarray-slice is true polyfill implementation of slice for all TypedArray objs (Uint8Array included). @dcodeIO your prompt and thorough responses were much appreciated. Thank you for a great OSS Lib.

[zj1024]

quick workaround to get ArrayBuffer with correct length
Uint8ArrayInstance.slice().buffer

it works. Great

[maelp]

Hmmm... shouldn't we expect the message.finish() to return the correctly-sized buffer "by default"? it seems a bit hacky-ish to have to do message.finish().slice().buffer to send the correctly-sized message no?

[soritesparadox]

Still an issue in 2024... 🙃

Using slice() worked well for me though.