Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

store complete size in transmute zstd compressor #60

Merged
merged 22 commits into from
May 8, 2023
Merged

Conversation

dholth
Copy link
Contributor

@dholth dholth commented Mar 31, 2023

Description

Fix #57

Copy link
Member

@jezdez jezdez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly some comments

conda_package_streaming/transmute.py Outdated Show resolved Hide resolved
conda_package_streaming/transmute.py Outdated Show resolved Hide resolved
Comment on lines 103 to 104
pkg_dat = pkg_file._file.read()
pkg_stream.write(pkg_dat)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remind me, does this trigger reading the whole file into memory?

Suggested change
pkg_dat = pkg_file._file.read()
pkg_stream.write(pkg_dat)
pkg_stream.write(pkg_file._file.read())

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably need copyfileobj.

@dholth dholth requested a review from jezdez May 5, 2023 10:56
@dholth dholth merged commit b6a8e12 into main May 8, 2023
@dholth dholth deleted the spooled-transmute branch May 8, 2023 17:54
@github-actions github-actions bot added the locked label May 8, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 8, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update transmute to use SpooledTemporaryFile instead of streaming directly to zip, for zstd reasons
2 participants