Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8422 list multipart uploads #8531

Open
wants to merge 50 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
e7fb7b8
Revert "docs: clarify auditing availble on enterprise (#8289)"
ItamarYuran Oct 23, 2024
cd5b80d
Merge branch 'master' of github.com:treeverse/lakeFS
ItamarYuran Jan 2, 2025
299032a
Merge branch 'master' of github.com:treeverse/lakeFS
ItamarYuran Jan 19, 2025
d9ea540
no work no mess
ItamarYuran Jan 20, 2025
7b1500d
got it
ItamarYuran Jan 21, 2025
08ad93c
f**** auditing
ItamarYuran Jan 21, 2025
1f7bf56
with tests
ItamarYuran Jan 21, 2025
2fbe42a
slight adjustment
ItamarYuran Jan 21, 2025
11f8d46
cross ya fingaz
ItamarYuran Jan 21, 2025
08977ec
yalla
ItamarYuran Jan 21, 2025
749f305
yalla
ItamarYuran Jan 21, 2025
7d121bd
tests dont yet work
ItamarYuran Jan 22, 2025
e04cd1b
check output
ItamarYuran Jan 22, 2025
dfc6699
check output
ItamarYuran Jan 22, 2025
8aa34f5
clean
ItamarYuran Jan 22, 2025
f8c8215
no tests
ItamarYuran Jan 22, 2025
cff37f2
tests
ItamarYuran Jan 23, 2025
219d4f4
ooooo baby
ItamarYuran Jan 23, 2025
0a37251
moving further
ItamarYuran Jan 23, 2025
52ef352
check errr
ItamarYuran Jan 23, 2025
ea30b22
check errr
ItamarYuran Jan 23, 2025
62f9e44
err not found
ItamarYuran Jan 23, 2025
1bdd16f
err not found
ItamarYuran Jan 23, 2025
644a390
yalla
ItamarYuran Jan 23, 2025
22c1756
check
ItamarYuran Jan 23, 2025
3d9225e
check
ItamarYuran Jan 23, 2025
b1d588b
schlafen
ItamarYuran Jan 23, 2025
f6d95c7
adapter
ItamarYuran Jan 23, 2025
9a4c5e1
namespace
ItamarYuran Jan 26, 2025
033784a
thank yoou
ItamarYuran Jan 26, 2025
2542855
names
ItamarYuran Jan 26, 2025
c74b714
refactor
ItamarYuran Jan 27, 2025
deb1fd3
refactored now
ItamarYuran Jan 27, 2025
5922ef8
no error
ItamarYuran Jan 27, 2025
2f740f4
trimmming
ItamarYuran Jan 27, 2025
f91771d
corrections
ItamarYuran Jan 27, 2025
b3e5884
fixin
ItamarYuran Jan 27, 2025
5a20568
yalla
ItamarYuran Jan 27, 2025
62f8bbb
testing key marker
ItamarYuran Jan 28, 2025
8f55605
is truncated
ItamarYuran Jan 28, 2025
645e5cf
with error not implemented
ItamarYuran Jan 28, 2025
4ecb691
testts will now pass
ItamarYuran Jan 28, 2025
58961d1
truncated for real
ItamarYuran Jan 28, 2025
fa64a69
tests
ItamarYuran Jan 29, 2025
82f74a3
final
ItamarYuran Jan 29, 2025
dfbee5c
kadima
ItamarYuran Jan 30, 2025
d21b831
final adjusments
ItamarYuran Jan 30, 2025
8ea2f8e
max uploads added
ItamarYuran Jan 30, 2025
eaf8972
kadima
ItamarYuran Jan 30, 2025
ad126b7
nu
ItamarYuran Jan 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions esti/s3_gateway_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,23 @@ func TestListMultipartUploads(t *testing.T) {
require.Contains(t, keys, obj1)
require.Contains(t, keys, obj2)

// testing maxuploads - only first upload should return
maxUploads := aws.Int32(1)
output, err = s3Client.ListMultipartUploads(ctx, &s3.ListMultipartUploadsInput{Bucket: resp1.Bucket, MaxUploads: maxUploads})
require.NoError(t, err, "failed to list multipart uploads")
keys = extractUploadKeys(output)
require.Contains(t, keys, obj1)
require.NotContains(t, keys, obj2)

// testing key marker and upload id marker for pagination. only records after marker should return
keyMarker := output.NextKeyMarker
uploadIDMarker := output.NextUploadIdMarker
output, err = s3Client.ListMultipartUploads(ctx, &s3.ListMultipartUploadsInput{Bucket: resp1.Bucket, MaxUploads: maxUploads, KeyMarker: keyMarker, UploadIdMarker: uploadIDMarker})
require.NoError(t, err, "failed to list multipart uploads")
keys = extractUploadKeys(output)
require.NotContains(t, keys, obj1)
require.Contains(t, keys, obj2)

// finish first mpu check only second appear
_, err = s3Client.CompleteMultipartUpload(ctx, completeInput1)
nopcoder marked this conversation as resolved.
Show resolved Hide resolved
require.NoError(t, err, "failed to complete multipart upload")
Expand All @@ -355,6 +372,23 @@ func TestListMultipartUploads(t *testing.T) {
keys = extractUploadKeys(output)
require.NotContains(t, keys, obj1)
require.Contains(t, keys, obj2)

// unsupported headers, expect error
delimiter := aws.String("/")
prefix := aws.String("prefix")
encodingType := types.EncodingTypeUrl

output, err = s3Client.ListMultipartUploads(ctx, &s3.ListMultipartUploadsInput{Bucket: resp1.Bucket, Delimiter: delimiter})
require.Error(t, err)
require.Contains(t, err.Error(), "NotImplemented")

output, err = s3Client.ListMultipartUploads(ctx, &s3.ListMultipartUploadsInput{Bucket: resp1.Bucket, Prefix: prefix})
require.Error(t, err)
require.Contains(t, err.Error(), "NotImplemented")

output, err = s3Client.ListMultipartUploads(ctx, &s3.ListMultipartUploadsInput{Bucket: resp1.Bucket, EncodingType: encodingType})
require.Error(t, err)
require.Contains(t, err.Error(), "NotImplemented")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these checks should be in a different test @nopcoder WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in general you are right but because we are going to skip this one for now for control-plane, might be nice to keep it in one spot. WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the multipart upload was the only test that we used esti to check as it requires 5M uploads per part.
Agree with Idean - this part can be part the adapter package test as it should represent the spec we supprot.

}

func extractUploadKeys(output *s3.ListMultipartUploadsOutput) []string {
Expand Down
1 change: 1 addition & 0 deletions pkg/block/adapter.go
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ type ListMultipartUploadsResponse struct {
Uploads []types.MultipartUpload
NextUploadIDMarker *string
NextKeyMarker *string
IsTruncated bool
}

// CreateMultiPartUploadOpts contains optional arguments for
Expand Down
1 change: 1 addition & 0 deletions pkg/block/s3/adapter.go
Original file line number Diff line number Diff line change
Expand Up @@ -881,6 +881,7 @@ func (a *Adapter) ListMultipartUploads(ctx context.Context, obj block.ObjectPoin
Uploads: resp.Uploads,
NextUploadIDMarker: resp.NextUploadIdMarker,
NextKeyMarker: resp.NextKeyMarker,
IsTruncated: *resp.IsTruncated,
nopcoder marked this conversation as resolved.
Show resolved Hide resolved
}
return &mpuResp, nil
}
Expand Down
12 changes: 12 additions & 0 deletions pkg/gateway/operations/listobjects.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ const (
QueryParamMaxUploads = "max-uploads"
QueryParamUploadIDMarker = "upload-id-marker"
QueryParamKeyMarker = "key-marker"
// missing implementation - will return error
QueryParampPrefix = "prefix"
QueryParampEncodingType = "encoding-type"
QueryParampDelimiter = "delimiter"
)

type ListObjects struct{}
Expand Down Expand Up @@ -409,6 +413,13 @@ func handleListMultipartUploads(w http.ResponseWriter, req *http.Request, o *Rep
maxUploadsStr := query.Get(QueryParamMaxUploads)
uploadIDMarker := query.Get(QueryParamUploadIDMarker)
keyMarker := query.Get(QueryParamKeyMarker)
prefix := query.Get(QueryParampPrefix)
delimiter := query.Get(QueryParampDelimiter)
ItamarYuran marked this conversation as resolved.
Show resolved Hide resolved
encodingType := query.Get(QueryParampEncodingType)
if prefix != "" || delimiter != "" || encodingType != "" {
_ = o.EncodeError(w, req, gatewayerrors.ErrNotImplemented, gatewayerrors.Codes.ToAPIErr(gatewayerrors.ErrNotImplemented))
return
}
opts := block.ListMultipartUploadsOpts{}
if maxUploadsStr != "" {
maxUploads, err := strconv.ParseInt(maxUploadsStr, 10, 32)
Expand Down Expand Up @@ -462,6 +473,7 @@ func handleListMultipartUploads(w http.ResponseWriter, req *http.Request, o *Rep
Uploads: uploads,
NextKeyMarker: *mpuResp.NextKeyMarker,
NextUploadIDMarker: *mpuResp.NextUploadIDMarker,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the AWS's helper function to get the value and check if the value is not nil - aws.StringValue

IsTruncated: mpuResp.IsTruncated,
}
o.EncodeResponse(w, req, resp, http.StatusOK)
}
1 change: 1 addition & 0 deletions pkg/gateway/serde/xml.go
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@ type ListMultipartUploadsOutput struct {
Uploads []Upload `xml:"Upload"`
NextKeyMarker string `xml:"NextKeyMarker"`
NextUploadIDMarker string `xml:"NextUploadIDMarker"`
IsTruncated bool `xml:"IsTruncated"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please verify that if there is no pagination required, does S3 remove these fields - which means that we need to set them as omitempty

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in case there is no pagination, itTruncated does not appear in s3's answer

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is why we should use omitempty

}

type VersioningConfiguration struct {
Expand Down
Loading