Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consolidate various configurations options, remove unrelated batch_size #1565

Closed
yjshen opened this issue Jan 14, 2022 · 0 comments · Fixed by #1562
Closed

Consolidate various configurations options, remove unrelated batch_size #1565

yjshen opened this issue Jan 14, 2022 · 0 comments · Fixed by #1562
Labels
api change Changes the API exposed to users of the crate datafusion Changes in the datafusion crate enhancement New feature or request

Comments

@yjshen
Copy link
Member

yjshen commented Jan 14, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We currently have three Configurations: ExecutionConfig, PhysicalPlanConfig and RuntimeConfig. They all share batch_size, which is only meaningful for query execution.

As we get RuntimeEnv plumbing through all execute() methods, we've got RuntimeConfig available to all executing code. Therefore, we'd better consolidate these configs and better separate their responsibilities.

Describe the solution you'd like

  1. Keep the batch_size only in RuntimeConfig, remove all unrelated batch_size passing during query planning.
  2. Rename PhysicalPlanConfig to FileScanConfig that clarifies its usage scenario.

Describe alternatives you've considered

Additional context
#1526 (comment) as well for context.

@yjshen yjshen added the enhancement New feature or request label Jan 14, 2022
@alamb alamb changed the title Consolidate various configurations Consolidate various configurations options Jan 16, 2022
@alamb alamb added datafusion Changes in the datafusion crate api change Changes the API exposed to users of the crate labels Feb 10, 2022
@alamb alamb changed the title Consolidate various configurations options Consolidate various configurations options, remove unrelated batch_size Feb 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api change Changes the API exposed to users of the crate datafusion Changes in the datafusion crate enhancement New feature or request
Projects
None yet
2 participants