You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Current implementation use Buffer as the internal container for the data been uploaded, which has a size limit of buffer.constants.MAX_LENGTH. Need to store the data for retry.
On 32-bit architectures, this value currently is (2^30)-1 (~1GB). On 64-bit architectures, this value currently is (2^31)-1 (~2GB).
This becomes a limitation as Jumb Putblock now supports a maximum block size of 4000MB.
Initial idea
Replace the Buffer with an array of Buffer?
ArrayBuffer has no hard limit, could it be of any use?
Manipulating ArrayBuffer with DataView seems to have a performance disadvantage to Buffer. Also, underlying http module - node-fetch do not support transferring 4GB ArrayBuffer yet as it use Uint8Array as the data container internally.
Current implementation use Buffer as the internal container for the data been uploaded, which has a size limit of buffer.constants.MAX_LENGTH. Need to store the data for retry.
On 32-bit architectures, this value currently is (2^30)-1 (~1GB). On 64-bit architectures, this value currently is (2^31)-1 (~2GB).
This becomes a limitation as Jumb Putblock now supports a maximum block size of 4000MB.
Initial idea
Manipulating ArrayBuffer with DataView seems to have a performance disadvantage to Buffer. Also, underlying http module - node-fetch do not support transferring 4GB ArrayBuffer yet as it use Uint8Array as the data container internally.
TODO
The text was updated successfully, but these errors were encountered: