Initial PyTorch Dataset support #134
Conversation
| from libcpp.string cimport string | ||
|
|
||
| from pathlib import Path | ||
|
|
||
| from pyarrow import Table | ||
| from pyarrow._dataset cimport FileFormat, FileWriteOptions, CFileWriteOptions, CScanner, CDataset, Dataset | ||
|
|
There was a problem hiding this comment.
These changes were due to running isort
| @@ -49,13 +49,19 @@ def scanner( | |||
| data: Union[str, Path, ds.Dataset], | |||
| columns: Optional[str] = None, | |||
| filter: Optional[pc.Expression] = None, | |||
| batch_size: Optional[int] = None, | |||
There was a problem hiding this comment.
Let's add docstrings for the function?
python/setup.py
Outdated
| @@ -69,7 +67,8 @@ | |||
| ext_modules=cythonize(extensions, language_level="3"), | |||
| zip_safe=False, | |||
| install_requires=["pyarrow>=9,<10"], | |||
| extras_require={"test": ["pytest>=6.0", "pandas", "duckdb", "click"]}, | |||
| extras_require={"test": ["pytest>=6.0", "pandas", "duckdb", "click"], | |||
| "pytorch": ["torch"]}, | |||
There was a problem hiding this comment.
I'm wondering if we should just ask the user to manually pip install torch? I feel like Rikai's extras system is way too complicated and hard to maintain. I'm wondering whether we should keep it simple for now.
python/lance/tests/test_pytorch.py
Outdated
|
|
||
| import pytest | ||
|
|
||
| pytest.importorskip("torch") |
There was a problem hiding this comment.
torch = pytest.importorskip("torch")? Save the import torch call afterwards
There was a problem hiding this comment.
Oh, i dont know this works. TIL
| @@ -16,10 +16,10 @@ | |||
| from pathlib import Path | |||
There was a problem hiding this comment.
Changes in this whole file is just formatting right?
.github/workflows/python.yml
Outdated
| @@ -31,7 +31,7 @@ jobs: | |||
| run: ./tools/build_wheel.sh $(echo cp${{ matrix.python-version }} | sed "s/\.//") | |||
| - name: Pip install | |||
| run: | | |||
| python -m pip install $(ls wheels/*.whl)[test] | |||
| python -m pip install $(ls wheels/*.whl)[test,pytorch] | |||
There was a problem hiding this comment.
install pytorch directly?
Basic and experimental implementation of
torch.Dataset.Closes #109