feat(python): add capability to read unity catalog (uc://) uris #3113
Conversation
github-actions
bot
added
binding/python
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
omkar-foss
force-pushed
the
feat-uc-python
branch
from
00254dc to
8fcd7f4
Compare
This adds capability to read directly from uc:// uris using the local catalog-unity crate. This also exposes the UC temporary credentials in storage_options of the `DeltaTable` instance so polars or similar readers can use it. Signed-off-by: Omkar P <[email protected]>
omkar-foss
force-pushed
the
feat-uc-python
branch
from
8fcd7f4 to
0824abe
Compare
omkar-foss
changed the title
Signed-off-by: Omkar P <[email protected]>
| @@ -266,14 +266,14 @@ pub struct TableSummary { | |||
| pub struct Table { | |||
| /// Username of table creator. | |||
| #[serde(default)] | |||
| pub created_by: String, | |||
| pub created_by: Option<String>, | |||
There was a problem hiding this comment.
There are instances I encountered where created_by and updated_by are null in UC Get Tables response, so we need to keep this as an Option<String> to handle it.
|
|
||
| let mut options = storage_options.clone().unwrap_or_default(); | ||
| if !uc_token.is_empty() { | ||
| options.insert("UNITY_CATALOG_TEMPORARY_TOKEN".to_string(), uc_token); |
There was a problem hiding this comment.
Have added this key UNITY_CATALOG_TEMPORARY_TOKEN for Polars or similar readers to be able reuse the credentials. cc: @ion-elgreco
| /// Allow http url (e.g. http://localhost:8080/api/2.1/...) | ||
| /// Supported keys: | ||
| /// - `unity_allow_http_url` | ||
| AllowHttpUrl, |
There was a problem hiding this comment.
This allows users to work with a local (non-https) Unity Catalog REST API with delta-rs.
| let (table_path, uc_token) = if UnityCatalogBuilder::is_unity_catalog_uri(table_uri) { | ||
| match rt().block_on(UnityCatalogBuilder::get_uc_location_and_token(table_uri)) { | ||
| Ok(tup) => tup, | ||
| Err(err) => return Err(PyRuntimeError::new_err(err.to_string())), | ||
| } | ||
| } else { | ||
| (table_uri.to_string(), "".to_string()) | ||
| }; | ||
|
|
||
| let mut options = storage_options.clone().unwrap_or_default(); | ||
| if !uc_token.is_empty() { | ||
| options.insert("UNITY_CATALOG_TEMPORARY_TOKEN".to_string(), uc_token); |
There was a problem hiding this comment.
I think this would be a better fit for code inside an object store factory
There was a problem hiding this comment.
Als you might have to wait before @hntd187's code gets merged (#3078)
Yep will have to wait for #3078 to get merged before this one, the temp credentials part is in there :)
I think this would be a better fit for code inside an object store factory
Yes, this code is currently duplicated in new() and is_deltatable(). Shall I move it to a private function? Or object store factory works too, let me know what you have in mind about that.
Description
This adds capability to read directly from uc:// uris using the local catalog-unity crate. This also exposes the UC temporary credentials in storage_options of the
DeltaTableinstance so polars or similar readers can use it.Related Issue(s)
Documentation
N/A