Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deep copy causes memory flare on unspill #7407

Closed
crusaderky opened this issue Dec 14, 2022 · 0 comments · Fixed by dask/zict#74 or #7435
Closed

Deep copy causes memory flare on unspill #7407

crusaderky opened this issue Dec 14, 2022 · 0 comments · Fixed by dask/zict#74 or #7435
Assignees
Labels

Comments

@crusaderky
Copy link
Collaborator

crusaderky commented Dec 14, 2022

When spilling a key with pickle5 buffers (e.g. numpy or pandas), you'll get n+2 frames, with the first two being the header and non-buffer pickle output.
This data is written contiguously to a file.

When unspilling the key, the pickle5 buffers are deep-copied by deserialization; this is both time-consuming (#7351 (comment)) and causes a memory flare.

Note that this issue only affects uncompressible data (or when lz4 has not been installed); when the buffers are compressed, there is an unavoidable moment where you have both the compressed and uncompressed copy in memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
1 participant