Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[storage][blob & file-datalake] fix uploadStream to support block size larger than buffer.constants.MAX_LENGTH #9403

Closed
2 tasks
ljian3377 opened this issue Jun 9, 2020 · 1 comment · Fixed by #11005
Assignees
Labels
Client This issue points to a problem in the data-plane of the library. Storage Storage Service (Queues, Blobs, Files)

Comments

@ljian3377
Copy link
Member

ljian3377 commented Jun 9, 2020

Current implementation use Buffer as the internal container for the data been uploaded, which has a size limit of buffer.constants.MAX_LENGTH. Need to store the data for retry.
On 32-bit architectures, this value currently is (2^30)-1 (~1GB). On 64-bit architectures, this value currently is (2^31)-1 (~2GB).

This becomes a limitation as Jumb Putblock now supports a maximum block size of 4000MB.

Initial idea

  • Replace the Buffer with an array of Buffer?
  • ArrayBuffer has no hard limit, could it be of any use?
    Manipulating ArrayBuffer with DataView seems to have a performance disadvantage to Buffer. Also, underlying http module - node-fetch do not support transferring 4GB ArrayBuffer yet as it use Uint8Array as the data container internally.

TODO

@ljian3377 ljian3377 added this to the [2020] July milestone Jun 9, 2020
@ramya-rao-a ramya-rao-a added Client This issue points to a problem in the data-plane of the library. Storage Storage Service (Queues, Blobs, Files) labels Jun 9, 2020
@ghost
Copy link

ghost commented Jun 9, 2020

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Client This issue points to a problem in the data-plane of the library. Storage Storage Service (Queues, Blobs, Files)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants