Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optional assume role in raster-api for external S3 bucket read permissions #56

Merged
merged 21 commits into from
Jun 8, 2022

Conversation

anayeaye
Copy link
Collaborator

@anayeaye anayeaye commented Apr 28, 2022

What

  • Optional DELTA_RASTER_DATA_ACCESS_ROLE_ARN will be assumed by raster api to obtain credentials with GetObject permissions for internal and external S3 buckets. This is intended to support reading data from external data providers that are willing to add a limited read only bucket policy to the VEDA raster api data access role.
  • Catch up to IMPACT fork of eoAPI to add datasetparams and post processing functionality to the raster-api (this will allow the raster-api to support pre-defined multi band algorithms for map layers like HLS SWIR--see this explanation in a delta-config project discussion.
  • Raster-api GDAL, logging, and resource configuration changes for experimenting with the raster-api (Tune raster-api config #54 removes these temporary changes when stack is operational).

Why

The raster-api needs to stream cloud optimized data from externally hosted buckets. This PR adds the methods needed to assume a role with read permissions for external buckets.

How tested

  • Deployed test and dev stacks with a data access role created to enable the raster api to stream data from internal and external buckets that have policies permitting that role to GetObjects.
  • Confirmed that only gdal debug configuration changes are deployed if optional data access role arn is not provided.
  • Confirmed that the eoAPI post processing methods work in a us-west stack with manually inserted HLS credentials

Notes:
(1) This PR adds logging that we will not want to keep for the long term; issue #54 reminds us to revisit the raster-api configuration and scale back logging and tune GDAL settings.
(2) Earlier versions of this branch included an external edl-credential-rotation service to obtain earthdata credentials on behalf of the raster api. Credential rotation may be added in future development but this PR only adds the option to assume a role in the raster api for external s3 data access.

@anayeaye
Copy link
Collaborator Author

I stamped this with an ominous 'do not merge' message but the changes themselves are non-breaking. The raster api works as expected when the session tokens are removed from the raster-api environment. The issue only comes up if the edl service is running and adding session tokens to the raster api environment.

@anayeaye anayeaye marked this pull request as draft May 24, 2022 18:57
@anayeaye anayeaye marked this pull request as draft May 24, 2022 18:57
@alukach alukach force-pushed the feature/edl-4-rasterapi branch from e260808 to 1f2ab9e Compare June 3, 2022 00:08
@anayeaye anayeaye changed the title [WIP] Earth data login for raster-api Optional assume role in raster-api for external S3 bucket read permissions Jun 7, 2022
@anayeaye anayeaye marked this pull request as ready for review June 7, 2022 22:42
@anayeaye anayeaye requested a review from leothomas June 7, 2022 22:43
Copy link
Contributor

@leothomas leothomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! I'm curious where the value for GDAL_HTTP_RETRY_DELAY comes from. Seems very specific

@anayeaye
Copy link
Collaborator Author

anayeaye commented Jun 8, 2022

Looks good to me! I'm curious where the value for GDAL_HTTP_RETRY_DELAY comes from. Seems very specific

I took this (and all or the raster-api updates) from the IMPACT eoAPI but I don't know how @sharkinsspatial and @vincentsarago chose that very specific value. I added a note to the tech debt issue #54 that we should try to explain or remove this setting.

@anayeaye anayeaye merged commit a2e3c34 into develop Jun 8, 2022
@anayeaye anayeaye deleted the feature/edl-4-rasterapi branch June 8, 2022 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants