Storage/handle file overwrite

[sanderegg]

What do these changes do?

This PR ensures that a file uploaded again first deletes the current file in the storage micro-service (both S3 and DB) before the links for uploading are returned.
The problem arose in #5102, where a file that was already uploaded before can be uploaded again only if no calls to GET /locations/{{location_id}}/files/{{file_id}}/metadata is done. If calls are done, there is a lazy mechanism in storage that fetches the file metadata from S3 to upgrade the database. As soon as this is called, then the upload information is wiped out of the database, which causes issues for multipart uploads (see remember how multipart upload works below)

Update from last changes:

• makes use of AWS versioning if any (which is also reflected in the tests) --> it will revert to the latest version of the file if versioning exists, and revert the database accordingly.
• re-use the fixtures from pytest-simcore
• upgraded local minio version from RELEASE.2020-05-16T01-33-21Z to RELEASE.2023-12-02T10-51-33Z in order to enable versioning (as agreed verbally, people will need to delete their minio docker volume docker volume rm ops_minio_data)

remember how multipart upload works

• ask storage to get an upload link (either 1 S3 link or N presigned links)
• in case of presigned links, storage creates the multipart upload and handles the whole procedure
• in case of S3 link which is the interesting in this case, storage cannot handle the procedure, therefore it marks the entry in file_metadata table as being currently uploaded, and an upload expiration date
• whenever a client asks for the file metadata, storage will find that the file is currently in upload, and will ask S3 whether the file did not arrive in between (lazy modus). if it finds the file, then it fills the database and returns it.

In this case, since there was previously a file, S3 has information even though the brand new file is not uploaded yet.

Related issue/s

• requires 🐛Correctly handle file type of on-demand clusters #5101
• partially solves Re-running a computational service that produce >5GiB file fails when uploading #5102

How to test

Dev Checklist

• No ENV changes or I properly updated ENV (read the instruction)

DevOps Checklist

[codecov]

Codecov Report

Merging #5108 (2afba7a) into master (2018f37) will increase coverage by 0.0%.
The diff coverage is 100.0%.

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #5108   +/-   ##
======================================
  Coverage    87.3%   87.4%           
======================================
  Files        1269    1269           
  Lines       52149   52174   +25     
  Branches     1129    1129           
======================================
+ Hits        45572   45611   +39     
+ Misses       6337    6323   -14     
  Partials      240     240           

Flag	Coverage Δ
integrationtests	64.8% <100.0%> (+<0.1%)	⬆️
unittests	85.2% <91.1%> (+<0.1%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...k/src/simcore_sdk/node_ports_common/filemanager.py	85.2% <100.0%> (+0.6%)	⬆️
...rage/src/simcore_service_storage/handlers_files.py	99.1% <ø> (ø)
...s/storage/src/simcore_service_storage/s3_client.py	98.8% <100.0%> (+0.1%)	⬆️
...rage/src/simcore_service_storage/simcore_s3_dsm.py	94.5% <100.0%> (+0.1%)	⬆️

... and 6 files with indirect coverage changes

[bisgaard-itis]

Nice! Thanks for fixing this

[GitHK]

👍

[ignapas]

🥇

[pcrespov]

🎉 amazing!

[matusdrobuliak66]

Thanks 👍

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[codeclimate]

Code Climate has analyzed commit 2afba7a and detected 0 issues on this pull request.

View more on Code Climate.