You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When spilling a key with pickle5 buffers (e.g. numpy or pandas), you'll get n+2 frames, with the first two being the header and non-buffer pickle output.
This data is written contiguously to a file.
When unspilling the key, the pickle5 buffers are deep-copied by deserialization; this is both time-consuming (#7351 (comment)) and causes a memory flare.
Note that this issue only affects uncompressible data (or when lz4 has not been installed); when the buffers are compressed, there is an unavoidable moment where you have both the compressed and uncompressed copy in memory.
The text was updated successfully, but these errors were encountered:
When spilling a key with pickle5 buffers (e.g. numpy or pandas), you'll get n+2 frames, with the first two being the header and non-buffer pickle output.
This data is written contiguously to a file.
When unspilling the key, the pickle5 buffers are deep-copied by deserialization; this is both time-consuming (#7351 (comment)) and causes a memory flare.
Note that this issue only affects uncompressible data (or when lz4 has not been installed); when the buffers are compressed, there is an unavoidable moment where you have both the compressed and uncompressed copy in memory.
The text was updated successfully, but these errors were encountered: