Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add foreign table providers #921

Merged
merged 28 commits into from
Nov 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
5cd598a
testing FFI for table provider
timsaucer Sep 9, 2024
7a476d2
Was able to get round trip schema from datafusion -> delta table -> d…
timsaucer Sep 9, 2024
f830058
Expand file structure
timsaucer Sep 9, 2024
eec590f
WIP on execution plan
timsaucer Sep 9, 2024
f7e7aca
Working through execution plan FFI
timsaucer Oct 5, 2024
803e6ff
Using datafusion-proto for execution plan properties
timsaucer Oct 6, 2024
9df2f86
Adding plan properties parsing from ffi
timsaucer Oct 6, 2024
73a1899
Standardize naming for FFI structs
timsaucer Oct 6, 2024
0d23d55
Intermediate testing and troubleshooting
timsaucer Oct 8, 2024
7d42f73
Adding record batch stream ffi representation
timsaucer Oct 9, 2024
405a89f
Mimimum viable product demonstrating foreign table provider
timsaucer Oct 9, 2024
d122393
Move ffi module to datafusion core
timsaucer Oct 14, 2024
88afcc7
Modifications need to compile against latest DF
timsaucer Oct 14, 2024
5e386e6
Set DF to 42.0.0
timsaucer Oct 18, 2024
2764bca
Rebasing and pulling in a few changes for DF43.0
timsaucer Nov 1, 2024
fbf54f1
Add wrapper for register table provider
timsaucer Nov 11, 2024
01a15f1
Suppress deprecation warning
timsaucer Nov 11, 2024
7f58117
Add example for FFI table provider
timsaucer Nov 11, 2024
cee25b8
Add pytest for FFI module to CI
timsaucer Nov 11, 2024
a284554
Add license text
timsaucer Nov 11, 2024
5551ef9
Change the name of the FFI table provider test so it doesn't try to r…
timsaucer Nov 11, 2024
50b4bf2
Build example in build stage to be used during test stage
timsaucer Nov 11, 2024
468ef5b
Combine pytests into one stage
timsaucer Nov 11, 2024
a0a39fd
Fix path for unit test
timsaucer Nov 11, 2024
9529bea
Installing maturin for ffi test in test script
timsaucer Nov 11, 2024
15458c7
Need to install the wheel for unit test
timsaucer Nov 11, 2024
64dcbd3
Add online documentation about using custom table providers
timsaucer Nov 12, 2024
e31e478
Raise an error if method is not implemented when it is expected
timsaucer Nov 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,9 @@ jobs:
repo-token: ${{ secrets.GITHUB_TOKEN }}

- name: Build Python package
run: maturin build --release --strip --features substrait
run: |
maturin build --release --strip --features substrait

- name: List Mac wheels
run: find target/wheels/

Expand Down
9 changes: 9 additions & 0 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,15 @@ jobs:
pip install -e . -vv
pytest -v .

- name: FFI unit tests
run: |
source venv/bin/activate
pip install -e . -vv
pip install maturin==1.5.1
cd examples/ffi-table-provider
maturin develop --release --strip
pytest python/tests/_test_table_provider.py

- name: Cache the generated dataset
id: cache-tpch-dataset
uses: actions/cache@v4
Expand Down
Loading
Loading