Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Improve ZipFS.readSync performance #1411

Closed
1 task done
markdingram opened this issue May 26, 2020 · 1 comment
Closed
1 task done

[Feature] Improve ZipFS.readSync performance #1411

markdingram opened this issue May 26, 2020 · 1 comment
Labels
enhancement New feature or request

Comments

@markdingram
Copy link
Contributor

  • I'd be willing to implement this feature

Describe the user story

@aws-cdk/aws-s3-deployment has a 10mb Zip file inside it that hashes on each run with

https://github.com/aws/aws-cdk/blob/a5c06a3a27b39a5315d0cfd0d34b3c1b25cfc464/packages/%40aws-cdk/core/lib/fs/fingerprint.ts

Currently gets stuck due to #1409, after that the it looks like performance is slow due to ZipFS.readSync calling ZipFS.readFileSync afresh on each read into the buffer.

There's a test repo to reproduce - https://github.com/markdingram/yarn-zip

$ yarn run slow
starting
read: 281.731ms 8192 8192
read: 480.655ms 8192 16384
read: 663.234ms 8192 24576
read: 845.223ms 8192 32768
read: 1027.200ms 8192 40960
read: 1210.914ms 8192 49152
read: 1391.626ms 8192 57344
read: 1590.551ms 8192 65536
read: 1764.586ms 8192 73728
read: 1944.802ms 8192 81920
read: 2139.621ms 8192 90112
read: 2333.735ms 8192 98304
read: 2525.436ms 8192 106496
....

Ejecting is a workaround.

Describe the solution you'd like

Cache the file contents in the ZipFS.fds Map on first read to serve subsequent requests.

Describe the drawbacks of your solution

Memory pressure if Zip Files aren't closed after use by existing consumers? Or many files are opened concurrently.

Describe alternatives you've considered

Store an open native File handle in the ZipFS.fds map on first read. Seems more complicated than the proposed approach?

Additional context

Add any other context or screenshots about the feature request here.

Repro - https://github.com/markdingram/yarn-zip

@markdingram markdingram added the enhancement New feature or request label May 26, 2020
@merceyz
Copy link
Member

merceyz commented Aug 29, 2020

Closing as fixed since #1743 started caching the read results

@merceyz merceyz closed this as completed Aug 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants