Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LZ4 compression for Apache Arrow #2339

Merged
merged 3 commits into from
Aug 14, 2023
+5,926 −2,324
Merged

LZ4 compression for Apache Arrow #2339

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b760fdf
Perspective reading LZ4 compressed RecordBatches.
sinistersnare Aug 7, 2023
a0fdc7d
Upgrade Arrow to 12.0.0
texodus Aug 13, 2023
4c775aa
Add `compression` options to `to_arrow()`, update superstore examples.
texodus Aug 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ and/or [Jupyterlab](https://jupyterlab.readthedocs.io/en/stable/).
### Examples

<!-- Examples -->
<table><tbody><tr><td>editable</td><td>csv</td><td>fractal</td></tr><tr><td><a href="https://perspective.finos.org/block?example=editable"><img height="125" src="https://perspective.finos.org/blocks/editable/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=csv"><img height="125" src="https://perspective.finos.org/blocks/csv/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=fractal"><img height="125" src="https://perspective.finos.org/blocks/fractal/preview.png"></img></a></td></tr><tr><td>market</td><td>raycasting</td><td>evictions</td></tr><tr><td><a href="https://perspective.finos.org/block?example=market"><img height="125" src="https://perspective.finos.org/blocks/market/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=raycasting"><img height="125" src="https://perspective.finos.org/blocks/raycasting/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=evictions"><img height="125" src="https://perspective.finos.org/blocks/evictions/preview.png"></img></a></td></tr><tr><td>streaming</td><td>covid</td><td>movies</td></tr><tr><td><a href="https://perspective.finos.org/block?example=streaming"><img height="125" src="https://perspective.finos.org/blocks/streaming/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=covid"><img height="125" src="https://perspective.finos.org/blocks/covid/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=movies"><img height="125" src="https://perspective.finos.org/blocks/movies/preview.png"></img></a></td></tr><tr><td>superstore</td><td>citibike</td><td>olympics</td></tr><tr><td><a href="https://perspective.finos.org/block?example=superstore"><img height="125" src="https://perspective.finos.org/blocks/superstore/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=citibike"><img height="125" src="https://perspective.finos.org/blocks/citibike/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=olympics"><img height="125" src="https://perspective.finos.org/blocks/olympics/preview.png"></img></a></td></tr><tr><td>jupyterlab</td><td>magic</td><td>nft</td></tr><tr><td><a href="http://beta.mybinder.org/v2/gh/finos/perspective/master?urlpath=lab/tree/examples/jupyter-notebooks"><img height="125" src="https://perspective.finos.org/img/jupyterlab.png"></img></a></td><td><a href="https://texodus.github.io/mtg-perspective/?seasons-in-the-abyss-67"><img height="125" src="https://perspective.finos.org/img/mtg_preview.png"></img></a></td><td><a href="https://sc1f.github.io/pudgy-penguin-perspective/"><img height="125" src="https://raw.githubusercontent.com/sc1f/pudgy-penguin-perspective/pages/meta.png"></img></a></td></tr><tr><td>nypd ccrb</td></tr><tr><td><a href="https://texodus.github.io/nypd-ccrb/"><img height="125" src="https://texodus.github.io/nypd-ccrb/preview.png"></img></a></td></tr></tbody></table>
<table><tbody><tr><td>editable</td><td>file</td><td>fractal</td></tr><tr><td><a href="https://perspective.finos.org/block?example=editable"><img height="125" src="https://perspective.finos.org/blocks/editable/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=file"><img height="125" src="https://perspective.finos.org/blocks/file/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=fractal"><img height="125" src="https://perspective.finos.org/blocks/fractal/preview.png"></img></a></td></tr><tr><td>market</td><td>raycasting</td><td>evictions</td></tr><tr><td><a href="https://perspective.finos.org/block?example=market"><img height="125" src="https://perspective.finos.org/blocks/market/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=raycasting"><img height="125" src="https://perspective.finos.org/blocks/raycasting/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=evictions"><img height="125" src="https://perspective.finos.org/blocks/evictions/preview.png"></img></a></td></tr><tr><td>streaming</td><td>covid</td><td>movies</td></tr><tr><td><a href="https://perspective.finos.org/block?example=streaming"><img height="125" src="https://perspective.finos.org/blocks/streaming/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=covid"><img height="125" src="https://perspective.finos.org/blocks/covid/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=movies"><img height="125" src="https://perspective.finos.org/blocks/movies/preview.png"></img></a></td></tr><tr><td>superstore</td><td>citibike</td><td>olympics</td></tr><tr><td><a href="https://perspective.finos.org/block?example=superstore"><img height="125" src="https://perspective.finos.org/blocks/superstore/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=citibike"><img height="125" src="https://perspective.finos.org/blocks/citibike/preview.png"></img></a></td><td><a href="https://perspective.finos.org/block?example=olympics"><img height="125" src="https://perspective.finos.org/blocks/olympics/preview.png"></img></a></td></tr><tr><td>jupyterlab</td><td>magic</td><td>nft</td></tr><tr><td><a href="http://beta.mybinder.org/v2/gh/finos/perspective/master?urlpath=lab/tree/examples/jupyter-notebooks"><img height="125" src="https://perspective.finos.org/img/jupyterlab.png"></img></a></td><td><a href="https://texodus.github.io/mtg-perspective/?seasons-in-the-abyss-67"><img height="125" src="https://perspective.finos.org/img/mtg_preview.png"></img></a></td><td><a href="https://sc1f.github.io/pudgy-penguin-perspective/"><img height="125" src="https://raw.githubusercontent.com/sc1f/pudgy-penguin-perspective/pages/meta.png"></img></a></td></tr><tr><td>nypd ccrb</td></tr><tr><td><a href="https://texodus.github.io/nypd-ccrb/"><img height="125" src="https://texodus.github.io/nypd-ccrb/preview.png"></img></a></td></tr></tbody></table>
<!-- Examples -->

### Documentation
Expand Down
2 changes: 1 addition & 1 deletion cmake/arrow.txt.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ project(arrow-download NONE)
include(ExternalProject)
ExternalProject_Add(apachearrow
GIT_REPOSITORY https://github.com/apache/arrow.git
GIT_TAG apache-arrow-8.0.0
GIT_TAG apache-arrow-12.0.0
SOURCE_DIR "${CMAKE_BINARY_DIR}/arrow-src"
BINARY_DIR "${CMAKE_BINARY_DIR}/arrow-build"
CONFIGURE_COMMAND ""
Expand Down
41 changes: 27 additions & 14 deletions cmake/arrow/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,15 @@ set(ARROW_SRCS
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/array_dict.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/array_nested.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/array_primitive.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/array_run_end.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/builder_adaptive.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/builder_base.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/builder_binary.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/builder_decimal.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/builder_dict.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/builder_nested.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/builder_primitive.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/builder_run_end.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/builder_union.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/concatenate.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/array/data.cc
Expand Down Expand Up @@ -76,6 +78,7 @@ set(ARROW_SRCS
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/io/stdio.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/io/transform.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/async_util.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/atfork_internal.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/basic_decimal.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/bit_block_counter.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/bit_run_reader.cc
Expand All @@ -87,8 +90,9 @@ set(ARROW_SRCS
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/byte_size.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/cancel.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/compression.cc
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/compression_zstd.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/compression_lz4.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/counting_semaphore.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/cpu_info.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/debug.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/decimal.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/delimiting.cc
Expand All @@ -100,6 +104,7 @@ set(ARROW_SRCS
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/key_value_metadata.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/memory.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/mutex.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/ree_util.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/string.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/string_builder.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/task_group.cc
Expand Down Expand Up @@ -130,7 +135,7 @@ set(ARROW_SRCS
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/csv/column_decoder.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/csv/options.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/csv/parser.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/csv/reader.cc
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/csv/reader.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/csv/writer.cc

# IPC
Expand All @@ -139,7 +144,6 @@ set(ARROW_SRCS
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/ipc/message.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/ipc/metadata_internal.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/ipc/options.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/ipc/reader.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/ipc/writer.cc

# Compute
Expand All @@ -148,20 +152,13 @@ set(ARROW_SRCS
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/api_vector.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/cast.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/exec.cc

# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/exec/aggregate_node.cc
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/exec/exec_plan.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/exec/expression.cc

# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/exec/filter_node.cc
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/exec/project_node.cc
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/exec/source_node.cc
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/exec/sink_node.cc
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/exec/order_by_impl.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/expression.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/function.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/function_internal.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/function_internal.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernel.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/ordering.cc


# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/registry.cc
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/aggregate_basic.cc
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/aggregate_mode.cc
Expand All @@ -175,6 +172,7 @@ set(ARROW_SRCS
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/scalar_boolean.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/scalar_cast_boolean.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/scalar_cast_dictionary.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/scalar_cast_extension.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/scalar_cast_internal.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/scalar_cast_nested.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/scalar_cast_numeric.cc
Expand All @@ -196,6 +194,7 @@ set(ARROW_SRCS
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/vector_nested.cc
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/vector_replace.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/vector_selection.cc
${PSP_CPP_SRC}/src/cpp/vendor/arrow_compute_registry.cpp

# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/vector_sort.cc
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/compute/kernels/row_encoder.cc
Expand All @@ -214,10 +213,15 @@ set(ARROW_SRCS
if(PSP_PYTHON_BUILD)
set(ARROW_SRCS
${ARROW_SRCS}
# use standard reader in Python builds.
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/ipc/reader.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/csv/reader.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/io/file.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/tensor/coo_converter.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/tensor/csf_converter.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/tensor/csx_converter.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/cpu_info.cc
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/union_util.cc

# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/util/time.cc
# ${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/vendored/double-conversion/bignum-dtoa.cc
Expand All @@ -230,6 +234,12 @@ if(PSP_PYTHON_BUILD)
${ARROW_SRCS}
${CMAKE_BINARY_DIR}/arrow-src/cpp/src/arrow/vendored/musl/strptime.c)
endif()
else()
set(ARROW_SRCS
${ARROW_SRCS}
# Use our vendored reader that does not use threads.
${PSP_CPP_SRC}/src/cpp/vendor/single_threaded_reader.cpp
${PSP_CPP_SRC}/src/cpp/vendor/arrow_single_threaded_reader.cpp)
endif()

set_property(SOURCE util/io_util.cc
Expand Down Expand Up @@ -274,13 +284,16 @@ add_library(arrow STATIC ${ARROW_SRCS})

target_compile_definitions(arrow PUBLIC ARROW_NO_DEPRECATED_API)
target_compile_definitions(arrow PUBLIC ARROW_STATIC)
# target_compile_definitions(arrow PUBLIC ARROW_WITH_ZSTD=ON)
target_compile_definitions(arrow PUBLIC ARROW_WITH_LZ4)

# will need built boost filesystem and system .lib to work, even though
# perspective itself does not use those dependencies
target_link_libraries(arrow
${double-conversion_LIBRARIES}
${Boost_FILESYSTEM_LIBRARY}
${Boost_SYSTEM_LIBRARY}
lz4_static
${ARROW_TEST_LINK_TOOLCHAIN})

# find_package(Flatbuffers)
Expand Down
16 changes: 16 additions & 0 deletions cmake/lz4.txt.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
cmake_minimum_required(VERSION 3.7.2)

project(lz4-download NONE)

include(ExternalProject)
ExternalProject_Add(lz4
GIT_REPOSITORY https://github.com/lz4/lz4.git
GIT_TAG v1.9.4
SOURCE_DIR "${CMAKE_BINARY_DIR}/lz4-src"
BINARY_DIR "${CMAKE_BINARY_DIR}/lz4-build"
CONFIGURE_COMMAND ""
BUILD_COMMAND ""
INSTALL_COMMAND ""
TEST_COMMAND ""
CMAKE_ARGS " -DBUILD_SHARED_LIBS=OFF -DBUILD_STATIC_LIBS=ON -DCMAKE_MODULE_PATH=${CMAKE_MODULE_PATH} -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}"
)
8 changes: 7 additions & 1 deletion cmake/modules/FindInstallDependency.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ cmake_minimum_required(VERSION 3.7.2)
# Helper to grab dependencies from remote sources #
# ##################################################
function(psp_build_dep name cmake_file)
if(EXISTS ${CMAKE_BINARY_DIR}/${name}-build)
if(EXISTS ${CMAKE_BINARY_DIR}/${name}-build AND NOT name STREQUAL "lz4")
psp_build_message("${Cyan}Dependency found - not rebuilding - ${CMAKE_BINARY_DIR}/${name}-build${ColorReset}")
else()
configure_file(${cmake_file} ${name}-download/CMakeLists.txt)
Expand Down Expand Up @@ -46,6 +46,12 @@ function(psp_build_dep name cmake_file)
add_subdirectory(${CMAKE_BINARY_DIR}/${name}-src
${CMAKE_BINARY_DIR}/${name}-build
EXCLUDE_FROM_ALL)
elseif(${name} STREQUAL lz4)
# lz4's CMakeLists.txt is in a subdir, build/cmake
add_subdirectory(${CMAKE_BINARY_DIR}/${name}-src/build/cmake
${CMAKE_BINARY_DIR}/${name}-build
EXCLUDE_FROM_ALL)
include_directories(${CMAKE_BINARY_DIR}/${name}-src/lib)
else()
add_subdirectory(${CMAKE_BINARY_DIR}/${name}-src
${CMAKE_BINARY_DIR}/${name}-build
Expand Down
8 changes: 4 additions & 4 deletions cpp/perspective/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ project(psp)
include(CheckCCompilerFlag)

set(CMAKE_BUILD_TYPE "Release")
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD 17)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should update/remove gnu flag to match:

perspective/cpp/perspective/CMakeLists.txt

Line 414 in 88e0878

set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=c++1y")

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this tested?

set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

# CMAKE POLICIES
Expand Down Expand Up @@ -380,6 +380,7 @@ psp_build_message("${Cyan}Building minimal Apache Arrow${ColorReset}")
psp_build_dep("rapidjson" "${PSP_CMAKE_MODULE_PATH}/rapidjson.txt.in")
psp_build_dep("double-conversion" "${PSP_CMAKE_MODULE_PATH}/double-conversion.txt.in")
psp_build_dep("flatbuffers" "${PSP_CMAKE_MODULE_PATH}/flatbuffers.txt.in")
psp_build_dep("lz4" "${PSP_CMAKE_MODULE_PATH}/lz4.txt.in")

# Build minimal arrow itself
psp_build_dep("arrow" "${PSP_CMAKE_MODULE_PATH}/arrow.txt.in")
Expand Down Expand Up @@ -414,7 +415,6 @@ if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
endif()

set(SOURCE_FILES
${PSP_CPP_SRC}/src/cpp/vendor/arrow_compute_registry.cpp
${PSP_CPP_SRC}/src/cpp/aggregate.cpp
${PSP_CPP_SRC}/src/cpp/aggspec.cpp
${PSP_CPP_SRC}/src/cpp/arg_sort.cpp
Expand Down Expand Up @@ -509,7 +509,7 @@ set(PYTHON_SOURCE_FILES ${SOURCE_FILES}
)

set(WASM_SOURCE_FILES ${SOURCE_FILES}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should remove whole block if not used anymore

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

used here

${PSP_CPP_SRC}/src/cpp/vendor/arrow_single_threaded_reader.cpp
# ${PSP_CPP_SRC}/src/cpp/vendor/arrow_single_threaded_reader.cpp
)

set(PYTHON_BINDING_SOURCE_FILES
Expand Down Expand Up @@ -626,7 +626,7 @@ elseif(PSP_CPP_BUILD OR PSP_PYTHON_BUILD)
set_property(TARGET psppy PROPERTY INSTALL_RPATH ${CMAKE_INSTALL_RPATH} ${module_origin_path})

target_compile_options(psp PRIVATE -fvisibility=hidden)
target_compile_options(psppy PRIVATE -fvisibility=hidden)
target_compile_options(psppy PRIVATE -fvisibility=hidden)
else()
target_compile_options(psppy PRIVATE -Wdeprecated-declarations)
endif()
Expand Down
6 changes: 5 additions & 1 deletion cpp/perspective/src/cpp/arrow_csv.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -404,7 +404,11 @@ namespace apachearrow {
auto read_options = arrow::csv::ReadOptions::Defaults();
auto parse_options = arrow::csv::ParseOptions::Defaults();
auto convert_options = arrow::csv::ConvertOptions::Defaults();

// #ifdef PSP_PARALLEL_FOR
// read_options.use_threads = true;
// #else
// read_options.use_threads = false;
// #endif
read_options.use_threads = false;
parse_options.newlines_in_values = true;

Expand Down
9 changes: 5 additions & 4 deletions cpp/perspective/src/cpp/emscripten.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,9 +207,10 @@ namespace binding {
template <typename CTX_T>
t_val
to_arrow(std::shared_ptr<View<CTX_T>> view, std::int32_t start_row,
std::int32_t end_row, std::int32_t start_col, std::int32_t end_col) {
std::shared_ptr<std::string> s
= view->to_arrow(start_row, end_row, start_col, end_col, true);
std::int32_t end_row, std::int32_t start_col, std::int32_t end_col,
bool compress) {
std::shared_ptr<std::string> s = view->to_arrow(
start_row, end_row, start_col, end_col, true, compress);
return str_to_arraybuffer(s)["buffer"];
}

Expand All @@ -226,7 +227,7 @@ namespace binding {
t_val
get_row_delta(std::shared_ptr<View<CTX_T>> view) {
auto slice = view->get_row_delta();
auto row_delta = view->data_slice_to_arrow(slice, false);
auto row_delta = view->data_slice_to_arrow(slice, false, false);
return str_to_arraybuffer(row_delta)["buffer"];
}

Expand Down
Loading