Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AzDatalake] Cleanup + Improvements #21222

Merged
merged 63 commits into from
Jul 24, 2023
Merged
Show file tree
Hide file tree
Changes from 57 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
e8167a2
Enable gocritic during linting (#20715)
jhendrixMSFT Apr 28, 2023
86627ae
Cosmos DB: Enable merge support (#20716)
ealsur Apr 28, 2023
8ac8c6d
[azservicebus, azeventhubs] Stress test and logging improvement (#20710)
richardpark-msft May 1, 2023
9111616
update proxy version (#20712)
azure-sdk May 1, 2023
d6bf190
Return an error when you try to send a message that's too large. (#20…
richardpark-msft May 1, 2023
e2693bd
Changes in test that is failing in pipeline (#20693)
siminsavani-msft May 2, 2023
03f0ac3
[azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20…
richardpark-msft May 2, 2023
838842d
[azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726)
richardpark-msft May 3, 2023
20b4dd8
Update changelog with latest features (#20730)
jhendrixMSFT May 3, 2023
745d967
pass along the artifact name so we can override it later (#20732)
azure-sdk May 3, 2023
6dfd0cb
[azeventhubs] Fixing checkpoint store race condition (#20727)
richardpark-msft May 3, 2023
ed7f3c7
Fix azidentity troubleshooting guide link (#20736)
chlowell May 3, 2023
b2cddab
[Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (…
Alancere May 4, 2023
2a8d96d
add sdk/resourcemanager/postgresql/armpostgresql live test (#20685)
Alancere May 4, 2023
0d22aed
add sdk/resourcemanager/eventhub/armeventhub live test (#20686)
Alancere May 4, 2023
5fa7df4
add sdk/resourcemanager/compute/armcompute live test (#20048)
Alancere May 4, 2023
c005ed6
sdk/resourcemanager/network/armnetwork live test (#20331)
Alancere May 4, 2023
36f766d
add sdk/resourcemanager/cosmos/armcosmos live test (#20705)
Alancere May 4, 2023
9c9d62a
Increment package version after release of azcore (#20740)
azure-sdk May 4, 2023
8bc3450
[azeventhubs] Improperly resetting etag in the checkpoint store (#20737)
richardpark-msft May 4, 2023
e1a6152
Eng workflows sync and branch cleanup additions (#20743)
azure-sdk May 4, 2023
04b463d
[azeventhubs] Latest start position can also be inclusive (ie, get th…
richardpark-msft May 4, 2023
8849196
Update GitHubEventProcessor version and remove pull_request_review pr…
azure-sdk May 5, 2023
27f5ee0
Rename DisableAuthorityValidationAndInstanceDiscovery (#20746)
chlowell May 5, 2023
2eec707
fix (#20707)
Alancere May 6, 2023
22db2d4
AzFile (#20739)
souravgupta-msft May 8, 2023
0cbfd88
azfile: Fixing connection string parsing logic (#20798)
souravgupta-msft May 8, 2023
d54fb08
[azadmin] fix flaky test (#20758)
gracewilcox May 8, 2023
ad8ebd9
Prepare azidentity v1.3.0 for release (#20756)
chlowell May 8, 2023
e2a6f70
Fix broken podman link (#20801)
azure-sdk May 8, 2023
a59d912
[azquery] update doc comments (#20755)
gracewilcox May 8, 2023
bd3b467
Fixed contribution section (#20752)
bobtabor-msft May 8, 2023
132a01a
[azeventhubs,azservicebus] Some API cleanup, renames (#20754)
richardpark-msft May 8, 2023
8db51ca
Add supporting features to enable distributed tracing (#20301) (#20708)
jhendrixMSFT May 9, 2023
4a66b4f
Restore ARM CAE support for azcore beta (#20657)
chlowell May 9, 2023
7d4a3cb
Upgrade to stable azcore (#20808)
chlowell May 9, 2023
068c3be
Increment package version after release of data/azcosmos (#20807)
azure-sdk May 9, 2023
8e0f66e
Updating changelog (#20810)
souravgupta-msft May 9, 2023
ce926c4
Add fake package to azcore (#20711)
jhendrixMSFT May 9, 2023
1a145c5
Updating CHANGELOG.md (#20809)
siminsavani-msft May 9, 2023
90dfc5c
changelog (#20811)
tasherif-msft May 9, 2023
c7eda59
Increment package version after release of storage/azfile (#20813)
azure-sdk May 9, 2023
7fac0b5
Update changelog (azblob) (#20815)
siminsavani-msft May 9, 2023
498a2ef
[azquery] migration guide (#20742)
gracewilcox May 9, 2023
ccb967e
Increment package version after release of monitor/azquery (#20820)
azure-sdk May 9, 2023
f4e6a22
[keyvault] prep for release (#20819)
gracewilcox May 10, 2023
8fd8eda
Merge branch 'main' into feature/azdatalake
tasherif-msft May 11, 2023
c94fa00
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft May 11, 2023
fc0b2b5
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jun 12, 2023
6fb1694
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jun 19, 2023
4f7fe43
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jun 26, 2023
3dac9d0
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jul 4, 2023
a0a861b
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jul 7, 2023
124e27e
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jul 19, 2023
133e6dd
fixed datalake errors + moved to internal path
tasherif-msft Jul 19, 2023
3052a35
delegation key + constants
tasherif-msft Jul 20, 2023
a3fad77
removed test
tasherif-msft Jul 20, 2023
4048359
further cleanup
tasherif-msft Jul 20, 2023
b3f98b1
renamed support and client fixes
tasherif-msft Jul 20, 2023
42e9c62
added tests
tasherif-msft Jul 20, 2023
5a1a8ea
handle error for rename
tasherif-msft Jul 21, 2023
30bb9ba
fixed response formatting
tasherif-msft Jul 22, 2023
fc268ea
cleanup
tasherif-msft Jul 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions sdk/storage/azdatalake/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
package azdatalake

import (
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/lease"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/exported"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/sas"
)
Expand Down Expand Up @@ -34,3 +35,47 @@ func ParseURL(u string) (URLParts, error) {
// ending at offset+count. A zero-value HTTPRange indicates the entire resource. An HTTPRange
// which has an offset but no zero value count indicates from the offset to the resource's end.
type HTTPRange = exported.HTTPRange

// ===================================== LEASE CONSTANTS ============================================================

// StatusType defines values for StatusType
type StatusType = lease.StatusType

const (
StatusTypeLocked StatusType = lease.StatusTypeLocked
StatusTypeUnlocked StatusType = lease.StatusTypeUnlocked
)

// PossibleStatusTypeValues returns the possible values for the StatusType const type.
func PossibleStatusTypeValues() []StatusType {
return lease.PossibleStatusTypeValues()
}

// DurationType defines values for DurationType
type DurationType = lease.DurationType

const (
DurationTypeInfinite DurationType = lease.DurationTypeInfinite
DurationTypeFixed DurationType = lease.DurationTypeFixed
)

// PossibleDurationTypeValues returns the possible values for the DurationType const type.
func PossibleDurationTypeValues() []DurationType {
return lease.PossibleDurationTypeValues()
}

// StateType defines values for StateType
type StateType = lease.StateType

const (
StateTypeAvailable StateType = lease.StateTypeAvailable
StateTypeLeased StateType = lease.StateTypeLeased
StateTypeExpired StateType = lease.StateTypeExpired
StateTypeBreaking StateType = lease.StateTypeBreaking
StateTypeBroken StateType = lease.StateTypeBroken
)

// PossibleStateTypeValues returns the possible values for the StateType const type.
func PossibleStateTypeValues() []StateType {
return lease.PossibleStateTypeValues()
}
240 changes: 117 additions & 123 deletions sdk/storage/azdatalake/datalakeerror/error_codes.go

Large diffs are not rendered by default.

5 changes: 0 additions & 5 deletions sdk/storage/azdatalake/directory/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -230,8 +230,3 @@ func (d *Client) SetHTTPHeaders(ctx context.Context, httpHeaders HTTPHeaders, op
// TODO: call into blob
return SetHTTPHeadersResponse{}, nil
}

// UndeletePath restores the specified path that was previously deleted. (dfs op/blob2).
func (d *Client) UndeletePath(ctx context.Context, path string, options *UndeletePathOptions) (UndeletePathResponse, error) {
return UndeletePathResponse{}, nil
}
33 changes: 4 additions & 29 deletions sdk/storage/azdatalake/directory/constants.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,37 +7,12 @@
package directory

import (
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/generated"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/path"
)

type ResourceType = generated.PathResourceType
type EncryptionAlgorithmType = path.EncryptionAlgorithmType

// TODO: consider the possibility of not exposing this and just pass it under the hood
const (
ResourceTypeFile ResourceType = generated.PathResourceTypeFile
ResourceTypeDirectory ResourceType = generated.PathResourceTypeDirectory
)

type RenameMode = generated.PathRenameMode

// TODO: consider the possibility of not exposing this and just pass it under the hood
const (
RenameModeLegacy RenameMode = generated.PathRenameModeLegacy
RenameModePosix RenameMode = generated.PathRenameModePosix
)

type SetAccessControlRecursiveMode = generated.PathSetAccessControlRecursiveMode

const (
SetAccessControlRecursiveModeSet SetAccessControlRecursiveMode = generated.PathSetAccessControlRecursiveModeSet
SetAccessControlRecursiveModeModify SetAccessControlRecursiveMode = generated.PathSetAccessControlRecursiveModeModify
SetAccessControlRecursiveModeRemove SetAccessControlRecursiveMode = generated.PathSetAccessControlRecursiveModeRemove
)

type EncryptionAlgorithmType = blob.EncryptionAlgorithmType

const (
EncryptionAlgorithmTypeNone EncryptionAlgorithmType = blob.EncryptionAlgorithmTypeNone
EncryptionAlgorithmTypeAES256 EncryptionAlgorithmType = blob.EncryptionAlgorithmTypeAES256
EncryptionAlgorithmTypeNone EncryptionAlgorithmType = path.EncryptionAlgorithmTypeNone
EncryptionAlgorithmTypeAES256 EncryptionAlgorithmType = path.EncryptionAlgorithmTypeAES256
)
214 changes: 30 additions & 184 deletions sdk/storage/azdatalake/directory/models.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@
package directory

import (
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/exported"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/generated"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/path"
"time"
)

Expand Down Expand Up @@ -73,80 +73,8 @@ type RenameOptions struct {
AccessConditions *AccessConditions
}

// GetPropertiesOptions contains the optional parameters for the Client.GetProperties method
type GetPropertiesOptions struct {
AccessConditions *AccessConditions
CPKInfo *CPKInfo
}

func (o *GetPropertiesOptions) format() *blob.GetPropertiesOptions {
if o == nil {
return nil
}
accessConditions := exported.FormatBlobAccessConditions(o.AccessConditions)
return &blob.GetPropertiesOptions{
AccessConditions: accessConditions,
CPKInfo: &blob.CPKInfo{
EncryptionKey: o.CPKInfo.EncryptionKey,
EncryptionAlgorithm: o.CPKInfo.EncryptionAlgorithm,
EncryptionKeySHA256: o.CPKInfo.EncryptionKeySHA256,
},
}
}

// ===================================== PATH IMPORTS ===========================================

// SetAccessControlOptions contains the optional parameters when calling the SetAccessControl operation. dfs endpoint
type SetAccessControlOptions struct {
// Owner is the owner of the path.
Owner *string
// Group is the owning group of the path.
Group *string
// ACL is the access control list for the path.
ACL *string
// Permissions is the octal representation of the permissions for user, group and mask.
Permissions *string
// AccessConditions contains parameters for accessing the path.
AccessConditions *AccessConditions
}

func (o *SetAccessControlOptions) format() (*generated.PathClientSetAccessControlOptions, *generated.LeaseAccessConditions, *generated.ModifiedAccessConditions, error) {
if o == nil {
return nil, nil, nil, nil
}
// call path formatter since we're hitting dfs in this operation
leaseAccessConditions, modifiedAccessConditions := exported.FormatPathAccessConditions(o.AccessConditions)
return &generated.PathClientSetAccessControlOptions{
Owner: o.Owner,
Group: o.Group,
ACL: o.ACL,
Permissions: o.Permissions,
}, leaseAccessConditions, modifiedAccessConditions, nil
}

// GetAccessControlOptions contains the optional parameters when calling the GetAccessControl operation.
type GetAccessControlOptions struct {
// UPN is the user principal name.
UPN *bool
// AccessConditions contains parameters for accessing the path.
AccessConditions *AccessConditions
}

func (o *GetAccessControlOptions) format() (*generated.PathClientGetPropertiesOptions, *generated.LeaseAccessConditions, *generated.ModifiedAccessConditions, error) {
action := generated.PathGetPropertiesActionGetAccessControl
if o == nil {
return &generated.PathClientGetPropertiesOptions{
Action: &action,
}, nil, nil, nil
}
// call path formatter since we're hitting dfs in this operation
leaseAccessConditions, modifiedAccessConditions := exported.FormatPathAccessConditions(o.AccessConditions)
return &generated.PathClientGetPropertiesOptions{
Upn: o.UPN,
Action: &action,
}, leaseAccessConditions, modifiedAccessConditions, nil
}

// SetAccessControlRecursiveOptions contains the optional parameters when calling the SetAccessControlRecursive operation. TODO: Design formatter
type SetAccessControlRecursiveOptions struct {
// ACL is the access control list for the path.
Expand Down Expand Up @@ -204,131 +132,49 @@ func (o *RemoveAccessControlRecursiveOptions) format() (*generated.PathClientSet
return nil, nil
}

// SetHTTPHeadersOptions contains the optional parameters for the Client.SetHTTPHeaders method.
type SetHTTPHeadersOptions struct {
AccessConditions *AccessConditions
}
// ================================= path imports ==================================

func (o *SetHTTPHeadersOptions) format() *blob.SetHTTPHeadersOptions {
if o == nil {
return nil
}
accessConditions := exported.FormatBlobAccessConditions(o.AccessConditions)
return &blob.SetHTTPHeadersOptions{
AccessConditions: accessConditions,
}
}

// HTTPHeaders contains the HTTP headers for path operations.
type HTTPHeaders struct {
// Optional. Sets the path's cache control. If specified, this property is stored with the path and returned with a read request.
CacheControl *string
// Optional. Sets the path's Content-Disposition header.
ContentDisposition *string
// Optional. Sets the path's content encoding. If specified, this property is stored with the blobpath and returned with a read
// request.
ContentEncoding *string
// Optional. Set the path's content language. If specified, this property is stored with the path and returned with a read
// request.
ContentLanguage *string
// Specify the transactional md5 for the body, to be validated by the service.
ContentMD5 []byte
// Optional. Sets the path's content type. If specified, this property is stored with the path and returned with a read request.
ContentType *string
}

func (o *HTTPHeaders) formatBlobHTTPHeaders() (*blob.HTTPHeaders, error) {
if o == nil {
return nil, nil
}
opts := blob.HTTPHeaders{
BlobCacheControl: o.CacheControl,
BlobContentDisposition: o.ContentDisposition,
BlobContentEncoding: o.ContentEncoding,
BlobContentLanguage: o.ContentLanguage,
BlobContentMD5: o.ContentMD5,
BlobContentType: o.ContentType,
}
return &opts, nil
}

func (o *HTTPHeaders) formatPathHTTPHeaders() (*generated.PathHTTPHeaders, error) {
// TODO: will be used for file related ops, like append
if o == nil {
return nil, nil
}
opts := generated.PathHTTPHeaders{
CacheControl: o.CacheControl,
ContentDisposition: o.ContentDisposition,
ContentEncoding: o.ContentEncoding,
ContentLanguage: o.ContentLanguage,
ContentMD5: o.ContentMD5,
ContentType: o.ContentType,
TransactionalContentHash: o.ContentMD5,
}
return &opts, nil
}
// GetPropertiesOptions contains the optional parameters for the Client.GetProperties method
type GetPropertiesOptions = path.GetPropertiesOptions

// SetMetadataOptions provides set of configurations for Set Metadata on path operation
type SetMetadataOptions struct {
AccessConditions *AccessConditions
CPKInfo *CPKInfo
CPKScopeInfo *CPKScopeInfo
}
// SetAccessControlOptions contains the optional parameters when calling the SetAccessControl operation. dfs endpoint
type SetAccessControlOptions = path.SetAccessControlOptions

func (o *SetMetadataOptions) format() *blob.SetMetadataOptions {
if o == nil {
return nil
}
accessConditions := exported.FormatBlobAccessConditions(o.AccessConditions)
return &blob.SetMetadataOptions{
AccessConditions: accessConditions,
CPKInfo: &blob.CPKInfo{
EncryptionKey: o.CPKInfo.EncryptionKey,
EncryptionAlgorithm: o.CPKInfo.EncryptionAlgorithm,
EncryptionKeySHA256: o.CPKInfo.EncryptionKeySHA256,
},
CPKScopeInfo: &blob.CPKScopeInfo{
EncryptionScope: o.CPKScopeInfo.EncryptionScope,
},
}
}
// GetAccessControlOptions contains the optional parameters when calling the GetAccessControl operation.
type GetAccessControlOptions = path.GetAccessControlOptions

// CPKInfo contains a group of parameters for the PathClient.Download method.
type CPKInfo struct {
EncryptionAlgorithm *EncryptionAlgorithmType
EncryptionKey *string
EncryptionKeySHA256 *string
}
type CPKInfo = path.CPKInfo

// CPKScopeInfo contains a group of parameters for the PathClient.SetMetadata method.
type CPKScopeInfo struct {
EncryptionScope *string
}
// GetSASURLOptions contains the optional parameters for the Client.GetSASURL method.
type GetSASURLOptions = path.GetSASURLOptions

// UndeletePathOptions contains the optional parameters for the Filesystem.UndeletePath operation.
type UndeletePathOptions struct {
// placeholder
}
// SetHTTPHeadersOptions contains the optional parameters for the Client.SetHTTPHeaders method.
type SetHTTPHeadersOptions = path.SetHTTPHeadersOptions

func (o *UndeletePathOptions) format() *UndeletePathOptions {
if o == nil {
return nil
}
return &UndeletePathOptions{}
}
// HTTPHeaders contains the HTTP headers for path operations.
type HTTPHeaders = path.HTTPHeaders

// SourceModifiedAccessConditions identifies the source path access conditions.
type SourceModifiedAccessConditions = generated.SourceModifiedAccessConditions
// SetMetadataOptions provides set of configurations for Set Metadata on path operation
type SetMetadataOptions = path.SetMetadataOptions

// SharedKeyCredential contains an account's name and its primary or secondary key.
type SharedKeyCredential = exported.SharedKeyCredential
type SharedKeyCredential = path.SharedKeyCredential

// AccessConditions identifies blob-specific access conditions which you optionally set.
type AccessConditions = exported.AccessConditions
type AccessConditions = path.AccessConditions

// SourceAccessConditions identifies blob-specific access conditions which you optionally set.
type SourceAccessConditions = path.SourceAccessConditions

// LeaseAccessConditions contains optional parameters to access leased entity.
type LeaseAccessConditions = exported.LeaseAccessConditions
type LeaseAccessConditions = path.LeaseAccessConditions

// ModifiedAccessConditions contains a group of parameters for specifying access conditions.
type ModifiedAccessConditions = exported.ModifiedAccessConditions
type ModifiedAccessConditions = path.ModifiedAccessConditions

// SourceModifiedAccessConditions contains a group of parameters for specifying access conditions.
type SourceModifiedAccessConditions = path.SourceModifiedAccessConditions

// CPKScopeInfo contains a group of parameters for the PathClient.SetMetadata method.
type CPKScopeInfo path.CPKScopeInfo
Loading