fix(ingest/s3): incorrectly parsing path in s3_uri

[eagle-25]

Background

• While ingesting S3 datasets with a pipeline, I encountered a warning: Unable to find any files in the folder s3://your-bucket/your-path/year=2024/. Skipping..., but the file was definitely present.

Changes

As-is

• The path was misparsed due to using lstrip() with s3://<bucket-name>//.

dir = "s3://bucket/test9/abc.txt"
path_to_process = dir.rstrip("\\").lstrip("s3://bucket//")

print(path_to_process) # result: '9/abc.txt', expected: test9/abc.txt

To-be

• Use urlparse to parse the path in s3_uri.

from urllib.parse import urlparse

dir = "s3://bucket/test/abc.txt"
path_to_process = urlparse(dir).path.lstrip("/")

print(path_to_process) # result: 'test9/abc.txt'

How to reproduce

• You can reproduce this bug with the following code.

from datahub.ingestion.run.pipeline import Pipeline

DATAHUB_SERVER = "http://localhost:8080"
DEFAULT_SINK = {"type": "datahub-rest", "config": {"server": DATAHUB_SERVER}}

S3_PATH = "s3://your-bucket/your-path/path/{table}/{partition_key[0]}={partition[0]}/*.parquet"

s3_config = {
        "source": {
            "type": "s3",
            "config": {
                "path_spec": {
                    "include": S3_PATH,
                },
                "aws_config": {
                    "aws_region": "ap-northeast-2",
                },
            },
        },
        "sink": DEFAULT_SINK,
    }

pipeline = Pipeline.create(s3_config)
pipeline.run()

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[treff7es]

It looks good, thanks for the fix