BEAC-167: Create a crate that instantiates an object store based on URL, settings and environment variables #569
Conversation
lizardoluis
requested review from
gruuya and
mildbyte
and removed request for
gruuya
lizardoluis
force-pushed
the
BEAC-167-object-store
branch
from
1c3a92f to
6f9ea5e
Compare
lizardoluis
force-pushed
the
BEAC-167-object-store
branch
from
6f9ea5e to
d9a49f1
Compare
lizardoluis
changed the title
| HashMap::new() | ||
| } | ||
| } | ||
|
|
||
| #[derive(Deserialize, Debug, PartialEq, Eq, Clone)] | ||
| pub struct S3 { |
There was a problem hiding this comment.
We should re-use the configs from your crate here (same for GCS, etc.).
There was a problem hiding this comment.
@gruuya This change will be a snow ball as these structs are used in several parts of the code. Would be ok to address this change in a separate PR, so that we reduce the complexity of it?
object_store_factory/src/aws.rs
Outdated
| ("AWS_ACCESS_KEY_ID", AmazonS3ConfigKey::AccessKeyId), | ||
| ("AWS_SECRET_ACCESS_KEY", AmazonS3ConfigKey::SecretAccessKey), | ||
| ("AWS_DEFAULT_REGION", AmazonS3ConfigKey::Region), | ||
| ("AWS_ENDPOINT", AmazonS3ConfigKey::Endpoint), | ||
| ("AWS_SESSION_TOKEN", AmazonS3ConfigKey::Token), | ||
| ( | ||
| "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", | ||
| AmazonS3ConfigKey::ContainerCredentialsRelativeUri, | ||
| ), |
There was a problem hiding this comment.
Seems like the first elements are redundant, since you can get them from the second element(s): config_key.as_ref().to_uppercase()
There was a problem hiding this comment.
I changed it completely to be more robust and accept more variables. I implemented something similar to their from_env() algorithm like here https://docs.rs/object_store/latest/src/object_store/aws/builder.rs.html#429-443.
| /// # Errors | ||
| /// | ||
| /// * If the URL scheme is unsupported or there is an error parsing the URL or options, an error is returned. | ||
| pub async fn build_object_store_from_opts( |
There was a problem hiding this comment.
This should also support building a local FS object store using the file:// protocol
There was a problem hiding this comment.
Should be resolve now with 26bb61e.
…/seafowl into BEAC-167-object-store
This PR breaks the object store creation logic into a separate crate
object_store_factory.The crate allows building object stores from various configurations and URLs. The key updates include:
build_object_store_from_configfunction:build_object_store_from_optsfunction: