Use S3 VPC Gateway Endpoint

[matusdrobuliak66]

Concept

Currently, we talk to S3 via the internet. We should use the S3 Endpoint which will talk directly within AWS network and without costs if it is in the same region & load balancing cost should decrease

• https://tomgregory.com/aws/when-to-use-an-aws-s3-vpc-endpoint/
• https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html

Tasks

Preview Give feedback

1. Idea: Use S3 private VPC Endpoint to download solver results from comp jobs into rocket PCR

[mrnicegyu11]

On 27. Sept. 2024: Enabled AWS VPC S3 Gateway Endpoint for osparc-master-zmt.click via the AWS console (no terraform) in us-east-1

[pcrespov]

@YuryHrytsuk @mrnicegyu11 Could you please include here (or a link to it) some evidence (e.g., plots or metrics) that quantifies the extent of the improvement? This can be used for the review meeting. thx

[mrnicegyu11]

DirtyS3APIBenchmark-Copy1.zip
We have done cost and usage tests using the S3 VPC Gateway Endpoint cloud infrastructure, as also proposed by AWS Solution Architects in our in-person meeting. This comment is only concerned with evaluating whether the usage of the S3 Endpoint will lead to increased pulling/pushing speeds.

To test this, we have enabled the S3 VPC Gateway Endpoint on aws-master, and test for potential speedups w.r.t. osparc-staging.io, that does not have the S3 Endpoint activated. The test is done using the osparc public python API client to download and upload files via osparc. The testcode is executed inside a jupyter-math on the respective osparc platforms.

The jupyter notebook code executed is attached to this comment.

In our tests we see no speedup of aws-master w.r.t. osparc-staging.io.

time to upload&download a 256MB file, 10 attempts, time in seconds - osparc-staging.io:

time to upload&download a 256MB file, 10 attempts, time in seconds - aws-master:

@mguidon FYI

[mguidon]

Also, @mrnicegyu11 On what ec2 instance type have you been running those tests?

[mguidon]

@mrnicegyu11 Can this be closed?

[mrnicegyu11]

Yes this can be considered done, thx