You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Spark supports fetch the contiguous shuffle blocks in batch, which is enabled by default (by conf spark.sql.adaptive.fetchShuffleBlocksInBatch). This feature has a big performance improvement in our production.
However, currently, since ColumnarBatchSerializer's supportsRelocationOfSerializedObjects return false, so that this feature cann't take effect.
In fact, the arrow serialization does support reloation if we don't write schema (which is default to true) and don't write EOS (which is an optional in arrow rpc serialization format)
Spark supports fetch the contiguous shuffle blocks in batch, which is enabled by default (by conf
spark.sql.adaptive.fetchShuffleBlocksInBatch
). This feature has a big performance improvement in our production.However, currently, since ColumnarBatchSerializer's
supportsRelocationOfSerializedObjects
return false, so that this feature cann't take effect.In fact, the arrow serialization does support reloation if we don't write schema (which is default to true) and don't write EOS (which is an optional in arrow rpc serialization format)
https://wesm.github.io/arrow-site-test/format/IPC.html#streaming-format

https://github.com/apache/arrow/blob/maint-12.0.1/cpp/src/arrow/ipc/message.cc#L496
The text was updated successfully, but these errors were encountered: