docs: rust usage documentation #3089
Conversation
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
|
I thought github supported draft PRs 😅 Maybe confusing it with another VCS |
Abdullahsab3
changed the title
docs/src/rust/read_cdf.rs
Outdated
| arrow_cast::pretty::print_batches(&cdf)?; | ||
| let batches = collect_batches( | ||
| cdf.properties().output_partitioning().partition_count(), | ||
| &cdf, | ||
| ctx, | ||
| ).await?; | ||
| arrow_cast::pretty::print_batches(&batches)?; | ||
|
|
||
|
|
||
| Ok(()) | ||
| } No newline at end of file | ||
| } | ||
|
|
||
| async fn collect_batches( | ||
| num_partitions: usize, | ||
| stream: &impl ExecutionPlan, | ||
| ctx: SessionContext, | ||
| ) -> Result<Vec<RecordBatch>, Box<dyn std::error::Error>> { | ||
| let mut batches = vec![]; | ||
| for p in 0..num_partitions { | ||
| let data: Vec<RecordBatch> = | ||
| collect_sendable_stream(stream.execute(p, ctx.task_ctx())?).await?; | ||
| batches.extend_from_slice(&data); | ||
| } | ||
| Ok(batches) | ||
| } |
There was a problem hiding this comment.
I think the CDF interface changed at some point. I was able to get around it by using the DF EXecutionPlan interface and a helper function I copied from delta-rs tests. Perhaps a more user-friendly interface for it is needed here?
Abdullahsab3
changed the title
Abdullahsab3
requested review from
wjones127,
rtyler,
fvaleye,
roeap,
ion-elgreco and
hntd187
as code owners
github-actions
bot
added
binding/python
Abdullahsab3
changed the title
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #3089 +/- ##
=======================================
Coverage 72.52% 72.53%
=======================================
Files 128 128
Lines 41201 41201
Branches 41201 41201
=======================================
+ Hits 29882 29886 +4
- Misses 9408 9410 +2
+ Partials 1911 1905 -6 ☔ View full report in Codecov by Sentry. |
| let builder = deltalake::DeltaTableBuilder::from_uri(bucket_table_path).with_storage_options(storage_options); | ||
| builder.build()?.verify_deltatable_existence().await?; | ||
| // true | ||
| ``` | ||
|
|
||
|
|
||
| ## Custom Storage Backends |
There was a problem hiding this comment.
If I understand correctly this is specific to the Python variant of the library, since it's using pyArrow, right?
|
There's some formatting fixes that need to happen before merging here, but I will squash the history and tidy things up before merging, thanks! |
Signed-off-by: Abdullahsab3 <[email protected]>
Description
This PR is intended to add high-level (mainly usage) documentation of delta-rs using the Rust API.
Related Issue(s)
closes #3088
Progress: