Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: service doc for s3 #2376

Merged
merged 4 commits into from
May 31, 2023
Merged

docs: service doc for s3 #2376

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5588c46
docs: service doc for s3
suyanhanx May 31, 2023
62aed1d
try
suyanhanx May 31, 2023
f737bee
fix
suyanhanx May 31, 2023
54fc88c
add region in via_map examples
suyanhanx May 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions core/src/services/azblob/docs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Capabilities
## Capabilities

This service can be used to:

Expand All @@ -15,7 +15,7 @@ This service can be used to:
- [x] presign
- [ ] blocking

# Configuration
## Configuration

- `root`: Set the work dir for backend.
- `container`: Set the container name for backend.
Expand All @@ -25,18 +25,18 @@ This service can be used to:

Refer to public API docs for more information.

# Examples
## Examples

This example works on [Azurite](https://github.com/Azure/Azurite) for local developments.

## Start local blob service
### Start local blob service

```shell
docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite
az storage container create --name test --connection-string "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;"
```

## Init OpenDAL Operator
### Init OpenDAL Operator

### Via Builder

Expand Down
10 changes: 4 additions & 6 deletions core/src/services/azdfs/docs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ As known as `abfs`, `azdfs` or `azdls`.

This service will visist the [ABFS](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-abfs-driver) URI supported by [Azure Data Lake Storage Gen2](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction).

# Capabilities
## Capabilities

This service can be used to:

Expand All @@ -18,7 +18,7 @@ This service can be used to:
- [ ] presign
- [ ] blocking

# Configuration
## Configuration

- `root`: Set the work dir for backend.
- `filesystem`: Set the filesystem name for backend.
Expand All @@ -28,9 +28,7 @@ This service can be used to:

Refer to public API docs for more information.

# Example

## Init OpenDAL Operator
## Example

### Via Builder

Expand Down Expand Up @@ -69,4 +67,4 @@ async fn main() -> Result<()> {

Ok(())
}
```
```
10 changes: 5 additions & 5 deletions core/src/services/cos/docs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Capabilities
## Capabilities

This service can be used to:

Expand All @@ -14,7 +14,7 @@ This service can be used to:
- [ ] presign
- [ ] blocking

# Configuration
## Configuration

- `root`: Set the work directory for backend
- `bucket`: Set the container name for backend
Expand All @@ -24,9 +24,9 @@ This service can be used to:

You can refer to [`CosBuilder`]'s docs for more information

# Example
## Example

## Via Builder
### Via Builder

```rust
use anyhow::Result;
Expand Down Expand Up @@ -54,4 +54,4 @@ async fn main() -> Result<()> {

Ok(())
}
```
```
4 changes: 2 additions & 2 deletions core/src/services/dashmap/docs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Capabilities
## Capabilities

This service can be used to:

Expand All @@ -12,4 +12,4 @@ This service can be used to:
- [ ] list
- [x] scan
- [ ] presign
- [ ] blocking
- [ ] blocking
6 changes: 3 additions & 3 deletions core/src/services/fs/docs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Capabilities
## Capabilities

This service can be used to:

Expand All @@ -15,13 +15,13 @@ This service can be used to:
- [ ] ~~presign~~
- [x] blocking

# Configuration
## Configuration

- `root`: Set the work dir for backend.

Refer to public API docs for more information.

# Example
## Example

### Via Builder

Expand Down
225 changes: 2 additions & 223 deletions core/src/services/s3/backend.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,232 +55,11 @@ static ENDPOINT_TEMPLATES: Lazy<HashMap<&'static str, &'static str>> = Lazy::new
});

const DEFAULT_WRITE_MIN_SIZE: usize = 8 * 1024 * 1024;

/// Aws S3 and compatible services (including minio, digitalocean space, Tencent Cloud Object Storage(COS) and so on) support.
/// For more information about s3-compatible services, refer to [Compatible Services](#compatible-services).
///
/// # Capabilities
///
/// This service can be used to:
///
/// - [x] stat
/// - [x] read
/// - [x] write
/// - [x] create_dir
/// - [x] delete
/// - [x] copy
/// - [ ] rename
/// - [x] list
/// - [x] scan
/// - [x] presign
/// - [ ] blocking
///
/// # Configuration
///
/// - `root`: Set the work dir for backend.
/// - `bucket`: Set the container name for backend.
/// - `endpoint`: Set the endpoint for backend.
/// - `region`: Set the region for backend.
/// - `access_key_id`: Set the access_key_id for backend.
/// - `secret_access_key`: Set the secret_access_key for backend.
/// - `security_token`: Set the security_token for backend.
/// - `default_storage_class`: Set the default storage_class for backend.
/// - `server_side_encryption`: Set the server_side_encryption for backend.
/// - `server_side_encryption_aws_kms_key_id`: Set the server_side_encryption_aws_kms_key_id for backend.
/// - `server_side_encryption_customer_algorithm`: Set the server_side_encryption_customer_algorithm for backend.
/// - `server_side_encryption_customer_key`: Set the server_side_encryption_customer_key for backend.
/// - `server_side_encryption_customer_key_md5`: Set the server_side_encryption_customer_key_md5 for backend.
/// - `disable_config_load`: Disable aws config load from env
/// - `enable_virtual_host_style`: Enable virtual host style.
///
/// Refer to [`S3Builder`]'s public API docs for more information.
///
/// # Temporary security credentials
///
/// OpenDAL now provides support for S3 temporary security credentials in IAM.
///
/// The way to take advantage of this feature is to build your S3 backend with `Builder::security_token`.
///
/// But OpenDAL will not refresh the temporary security credentials, please keep in mind to refresh those credentials in time.
///
/// # Server Side Encryption
///
/// OpenDAL provides full support of S3 Server Side Encryption(SSE) features.
///
/// The easiest way to configure them is to use helper functions like
///
/// - SSE-KMS: `server_side_encryption_with_aws_managed_kms_key`
/// - SSE-KMS: `server_side_encryption_with_customer_managed_kms_key`
/// - SSE-S3: `server_side_encryption_with_s3_key`
/// - SSE-C: `server_side_encryption_with_customer_key`
///
/// If those functions don't fulfill need, low-level options are also provided:
///
/// - Use service managed kms key
/// - `server_side_encryption="aws:kms"`
/// - Use customer provided kms key
/// - `server_side_encryption="aws:kms"`
/// - `server_side_encryption_aws_kms_key_id="your-kms-key"`
/// - Use S3 managed key
/// - `server_side_encryption="AES256"`
/// - Use customer key
/// - `server_side_encryption_customer_algorithm="AES256"`
/// - `server_side_encryption_customer_key="base64-of-your-aes256-key"`
/// - `server_side_encryption_customer_key_md5="base64-of-your-aes256-key-md5"`
///
/// After SSE have been configured, all requests send by this backed will attach those headers.
///
/// Reference: [Protecting data using server-side encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html)
///
/// # Example
///
/// ## Basic Setup
///
/// ```no_run
/// use std::sync::Arc;
///
/// use anyhow::Result;
/// use opendal::services::S3;
/// use opendal::Operator;
///
/// #[tokio::main]
/// async fn main() -> Result<()> {
/// // Create s3 backend builder.
/// let mut builder = S3::default();
/// // Set the root for s3, all operations will happen under this root.
/// //
/// // NOTE: the root must be absolute path.
/// builder.root("/path/to/dir");
/// // Set the bucket name, this is required.
/// builder.bucket("test");
/// // Set the endpoint.
/// //
/// // For examples:
/// // - "https://s3.amazonaws.com"
/// // - "http://127.0.0.1:9000"
/// // - "https://oss-ap-northeast-1.aliyuncs.com"
/// // - "https://cos.ap-seoul.myqcloud.com"
/// //
/// // Default to "https://s3.amazonaws.com"
/// builder.endpoint("https://s3.amazonaws.com");
/// // Set the access_key_id and secret_access_key.
/// //
/// // OpenDAL will try load credential from the env.
/// // If credential not set and no valid credential in env, OpenDAL will
/// // send request without signing like anonymous user.
/// builder.access_key_id("access_key_id");
/// builder.secret_access_key("secret_access_key");
///
/// let op: Operator = Operator::new(builder)?.finish();
///
/// Ok(())
/// }
/// ```
///
/// ## S3 with SSE-C
///
/// ```no_run
/// use anyhow::Result;
/// use log::info;
/// use opendal::services::S3;
/// use opendal::Operator;
///
/// #[tokio::main]
/// async fn main() -> Result<()> {
/// let mut builder = S3::default();
///
/// // Setup builders
///
/// // Enable SSE-C
/// builder.server_side_encryption_with_customer_key("AES256", "customer_key".as_bytes());
///
/// let op = Operator::new(builder)?.finish();
/// info!("operator: {:?}", op);
///
/// // Writing your testing code here.
///
/// Ok(())
/// }
/// ```
///
/// ## S3 with SSE-KMS and aws managed kms key
///
/// ```no_run
/// use anyhow::Result;
/// use log::info;
/// use opendal::services::S3;
/// use opendal::Operator;
///
/// #[tokio::main]
/// async fn main() -> Result<()> {
/// let mut builder = S3::default();
///
/// // Setup builders
///
/// // Enable SSE-KMS with aws managed kms key
/// builder.server_side_encryption_with_aws_managed_kms_key();
///
/// let op = Operator::new(builder)?.finish();
/// info!("operator: {:?}", op);
///
/// // Writing your testing code here.
///
/// Ok(())
/// }
/// ```
///
/// ## S3 with SSE-KMS and customer managed kms key
///
/// ```no_run
/// use anyhow::Result;
/// use log::info;
/// use opendal::services::S3;
/// use opendal::Operator;
///
/// #[tokio::main]
/// async fn main() -> Result<()> {
/// let mut builder = S3::default();
///
/// // Setup builders
///
/// // Enable SSE-KMS with customer managed kms key
/// builder.server_side_encryption_with_customer_managed_kms_key("aws_kms_key_id");
///
/// let op = Operator::new(builder)?.finish();
/// info!("operator: {:?}", op);
///
/// // Writing your testing code here.
///
/// Ok(())
/// }
/// ```
///
/// ## S3 with SSE-S3
///
/// ```no_run
/// use anyhow::Result;
/// use log::info;
/// use opendal::services::S3;
/// use opendal::Operator;
///
/// #[tokio::main]
/// async fn main() -> Result<()> {
/// let mut builder = S3::default();
///
/// // Setup builders
///
/// // Enable SSE-S3
/// builder.server_side_encryption_with_s3_key();
///
/// let op = Operator::new(builder)?.finish();
/// info!("operator: {:?}", op);
///
/// // Writing your testing code here.
///
/// Ok(())
/// }
/// ```
///
/// # Compatible Services
#[doc = include_str!("docs.md")]
#[doc = include_str!("compatible_services.md")]
#[derive(Default)]
pub struct S3Builder {
Expand Down
Loading