Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add S3 support #3258

Merged
merged 1 commit into from
Sep 19, 2022
Merged

Add S3 support #3258

merged 1 commit into from
Sep 19, 2022

Conversation

fyrestone
Copy link
Contributor

@fyrestone fyrestone commented Sep 15, 2022

What do these changes do?

  • Add S3 support (The s3fs and fsspec Python packages are required).

An example to read csv from s3.

import mars
import mars.dataframe as md

mars.new_session()
# Pass endpoint_url / aws_access_key_id / aws_secret_access_key to read_csv.
mdf = md.read_csv("s3://bucket/example.csv", index_col=0, storage_options={
    "client_kwargs": {
        "endpoint_url": "http://192.168.1.12:9000",
        "aws_access_key_id": "RwDeqMoctbLG3yly",
        "aws_secret_access_key": "uwinWm1hTAGJ6Wnipa4tbE4SwO3Mx6Ek",
    }})
# Export environment vars AWS_ENDPOINT_URL / AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY.
mdf = md.read_csv("s3://bucket/example.csv", index_col=0)
r = mdf.head(1000).execute()
print(r)

Related issue number

Fixes #xxxx

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass, see here for how to run them

@fyrestone fyrestone added the type: feature New feature label Sep 15, 2022
@fyrestone fyrestone marked this pull request as ready for review September 16, 2022 02:15
@fyrestone fyrestone requested a review from a team as a code owner September 16, 2022 02:15
Copy link
Contributor

@zhongchun zhongchun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Copy link
Contributor

@chaokunyang chaokunyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chaokunyang chaokunyang merged commit c69eea9 into mars-project:master Sep 19, 2022
qianduoduo0904 pushed a commit to qianduoduo0904/mars that referenced this pull request Sep 22, 2022
Co-authored-by: 刘宝 <[email protected]>
(cherry picked from commit c69eea9)
UranusSeven pushed a commit to xorbitsai/mars that referenced this pull request Sep 22, 2022
Co-authored-by: 刘宝 <[email protected]>
(cherry picked from commit c69eea9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: feature New feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants