Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use CREATE TABLE for small parquet #1617

Merged
merged 1 commit into from
Aug 29, 2024
Merged

Conversation

mbostock
Copy link
Member

Fixes #1614.

@mbostock mbostock requested a review from Fil August 28, 2024 20:01
@Fil
Copy link
Contributor

Fil commented Aug 28, 2024

Did you have a way to estimate (guesstimate) that threshold value? Do we want to set up some kind of experiment?

@mbostock
Copy link
Member Author

No idea what the right threshold is. I’d guess it depends on your server too.

@mbostock
Copy link
Member Author

mbostock commented Aug 29, 2024

Is there a different value that you would prefer? It would be easy to change in the future so I don’t think we need to try too hard here.

@Fil
Copy link
Contributor

Fil commented Aug 29, 2024

No idea, I think it's probably a good guess. We could do do some experiments but it doesn't have to be blocking. Let's merge now and revisit later if needed.

@mbostock mbostock merged commit bbea2c5 into main Aug 29, 2024
4 checks passed
@mbostock mbostock deleted the mbostock/optimize-duckdb-parquet branch August 29, 2024 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimize DuckDBClient’s handling of small Parquet files
2 participants