Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate AWS OpenSearch requests signing credential generation to AWS SDK V2 #51149

Merged
merged 8 commits into from
Jan 28, 2025

Conversation

gabrielcorado
Copy link
Contributor

Related to #14142

This PR updates the AWS signing service to accept using the AWS config provider (which uses AWS SDK V2). OpenSearch uses the service to sign the requests.

Note that the signer part still uses the SDK V1 signer (with the NewSignerV2 helper function). So, no changes are required to the signing logic for now. This will be updated in future PRs.

@gabrielcorado gabrielcorado added the no-changelog Indicates that a PR does not require a changelog entry label Jan 17, 2025
lib/srv/db/opensearch_test.go Show resolved Hide resolved
lib/srv/db/opensearch/engine.go Show resolved Hide resolved
Comment on lines +185 to +188
// TODO(gabrielcorado): once all service callers are updated to use
// AWSConfigProvider, make it required and remove session provider and
// credentials getter fallback.
func (s *SigningService) newSigner(ctx context.Context, signCtx *SigningCtx) (*v4.Signer, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I like how this makes it easy to migrate each dependent service in small PRs

@gabrielcorado gabrielcorado marked this pull request as ready for review January 17, 2025 12:59
@github-actions github-actions bot added database-access Database access related issues and PRs size/sm labels Jan 17, 2025
@github-actions github-actions bot requested a review from smallinsky January 17, 2025 12:59
Copy link
Contributor

@GavinFrazar GavinFrazar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took another look at this and it won't match prior behavior because the engine will not assume the role from the db AWS metadata to chain the roles together.

We can fix that by adding e.sessionCtx.Database.GetAWS().AssumeRoleARN to the SigningCtx and passing it when we call SignRequest.

Alternatively, we could add it to the SigningServiceConfig, e.g

func (s *SigningService) newSigner(ctx context.Context, signCtx *SigningCtx) (*v4.Signer, error) {
	if s.AWSConfigProvider != nil {
		awsCfg, err := s.AWSConfigProvider.GetConfig(ctx, signCtx.SigningRegion,
			awsconfig.WithAssumeRole(s.AWSRoleArn, s.AWSExternalID),
			awsconfig.WithAssumeRole(signCtx.AWSRoleArn, signCtx.AWSExternalID),

@gabrielcorado
Copy link
Contributor Author

Thanks for pointing it out. I've updated the SigningCtx to accept a base role.

Also, @smallinsky, if you can take a look at this, thanks!

Copy link
Contributor

@smallinsky smallinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if tested

@greedy52 greedy52 self-requested a review January 22, 2025 13:34
@greedy52
Copy link
Contributor

greedy52 commented Jan 22, 2025

		awsCfg, err := s.AWSConfigProvider.GetConfig(ctx, signCtx.SigningRegion,
			awsconfig.WithAssumeRole(signCtx.BaseAWSRoleARN, signCtx.AWSExternalID),
			awsconfig.WithAssumeRole(signCtx.AWSRoleArn, signCtx.AWSExternalID),
			awsconfig.WithCredentialsMaybeIntegration(signCtx.Integration),

Does that mean the existing code uses external ID twice for opensearch when chained? That would be a bug if we want to keep the logic consistent with other database types. Though I agree we should keep it backwards compatible for now. The problem is this logic in SignerService would need a variant if used for other database types.

At this point, i would rather make the GetConfig call outside signing service to avoid passing in so many things to the signing ctx and to keep SigningService single purpose.

@GavinFrazar
Copy link
Contributor

Does that mean the existing code uses external ID twice for opensearch when chained

Yes. It should have been chained like this:

if meta.AssumeRoleARN == "" {
signingCtx.AWSExternalID = meta.ExternalID
}
signedReq, err := signer.SignRequest(e.Context, outReq, signingCtx)

At this point, i would rather make the GetConfig call outside signing service to avoid passing in so many things to the signing ctx and to keep SigningService single purpose.

I agree that we should reduce the scope of signing service responsibilities.
I think if we take this suggestion then we can just do it for v2 sdk credentials for now and keep the prior behavior where signing service makes STS assume role call based on the signing ctx role arn for sdk v1.

@gabrielcorado
Copy link
Contributor Author

@GavinFrazar @greedy52 I've kept this change as simple as possible by adding another parameter to the SigingCtx and having the caller decide how to specify the external IDs.

This isn't ideal, but since the signing service will be reworked during the SDK V2 migration, any improvements on the API definition should come from there because we'll know precisely the requirements for V2. We'll also consider the app access usage.

@greedy52 greedy52 enabled auto-merge January 28, 2025 15:28
@greedy52 greedy52 added this pull request to the merge queue Jan 28, 2025
Merged via the queue into master with commit ad2774d Jan 28, 2025
41 checks passed
@greedy52 greedy52 deleted the gabrielcorado/opensearch-aws-sdk2-migration branch January 28, 2025 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
database-access Database access related issues and PRs no-changelog Indicates that a PR does not require a changelog entry size/sm
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants