Add standalone example of using the SQL frontend #11088
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@andygrove or @jdye64 I wonder if you might have some time to review this PR as I think it is relevant to to your use of DataFusion |
github-actions
bot
removed
the
documentation
11 tasks
|
I am working on getting the CI to run successfully on this PR, see #11173 |
|
🤔 this might also make https://github.com/apache/datafusion/blob/main/datafusion/sql/examples/sql.rs redundant |
mustafasrepo
reviewed
Jul 2, 2024
| // The initial LogicalPlan is a mechanical translation from the parsed SQL | ||
| // and often can not run without the Analyzer passes. | ||
| // | ||
| // In this example, `person.age` is actually a different data type (Int32) |
There was a problem hiding this comment.
At the source schema data type defined as Uint8
mustafasrepo
reviewed
Jul 2, 2024
| // (Int64). However, this plan is not as efficient as it could be, as it | ||
| // will require casting *each row* of the input to UInt64 before comparison | ||
| // to 21 and 32. To optimize this query's performance, it is better to cast | ||
| // the constants once at plan time to Int32. |
There was a problem hiding this comment.
I guess this should be to Uint8
There was a problem hiding this comment.
Good catch -- fixed in 6152fd6
mustafasrepo
approved these changes
Jul 2, 2024
comphead
reviewed
Jul 2, 2024
| let context_provider = MyContextProvider::default(); | ||
| let sql_to_rel = SqlToRel::new(&context_provider); | ||
| let logical_plan = sql_to_rel.sql_statement_to_plan(statements[0].clone())?; | ||
| println!( |
There was a problem hiding this comment.
maybe we can use assert! here to let the user know the expected output?
There was a problem hiding this comment.
that is a good idea -- I will do so (the existing examples are inconsistent with that).
comphead
reviewed
Jul 2, 2024
| config.options(), | ||
| observe_analyzer, | ||
| )?; | ||
| println!( |
comphead
reviewed
Jul 2, 2024
| // 4. Pushed the filter into the scan | ||
| let optimized_plan = | ||
| Optimizer::new().optimize(analyzed_plan, &config, observe_optimizer)?; | ||
| println!( |
comphead
reviewed
Jul 2, 2024
comphead
approved these changes
Jul 2, 2024
|
Thanks everyone for the comments and feedback |
comphead
added a commit
to comphead/arrow-datafusion
that referenced
this pull request
Jul 8, 2024
* Add standalone example of using the SQL frontend * Disable debug info for all example jobs * Revert "Disable debug info for all example jobs" This reverts commit f222a10. * fix type * Int32 --> Int8 * Use assert_eq rather than println * Update datafusion-examples/examples/sql_frontend.rs Co-authored-by: Oleks V <[email protected]> --------- Co-authored-by: Oleks V <[email protected]>
findepi
pushed a commit
to findepi/datafusion
that referenced
this pull request
Jul 16, 2024
* Add standalone example of using the SQL frontend * Disable debug info for all example jobs * Revert "Disable debug info for all example jobs" This reverts commit f222a10. * fix type * Int32 --> Int8 * Use assert_eq rather than println * Update datafusion-examples/examples/sql_frontend.rs Co-authored-by: Oleks V <[email protected]> --------- Co-authored-by: Oleks V <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #10855
Rationale for this change
The current
rewrite_expr.rsexample has three distinct examples:AnalyzerRuleOptimizerRuleEach I think should be its own example to make them easier to find and more full featured
What changes are included in this PR?
sql_frontend.rsAre these changes tested?
Yes by CI
Are there any user-facing changes?