Improving storage performance

[GitHK]

What do these changes do?

The list_files function is now optimised to run with big datasets, and no longer depends on third party S3 calls when listing files in the storage service.

Removed

• database updates when retrieving the real file names (which might have been renamed) => responsible for 40% of the slowdown
• file metadata updates in the database after invoking an S3 call for each file when listing all the files => responsible for the majority of the reminding slowdown

Added

• a metadata_file_updater launched before returning the upload link for each file to be uploaded. It will try to update the metadata for each file. It uses an exponential backoff retry to account for huge file uploads. It also has a maximum amount of possible retries. If it fails to update the metadata, an error message will be logged.

Changes

Related issue number

Closes #1559

How to test

Start the project and make sure to have lots of files in your storage. With the old implementation performances degraded after 200 uploaded files.

Checklist

• Did you change any service's API? Then make sure to bundle document and upgrade version (make openapi-specs, git commit ... and then make version-*)
• Unit tests for the changes exist
• Runs in the swarm
• Documentation reflects the changes
• New module? Add your github username to .github/CODEOWNERS

[codecov]

Codecov Report

Merging #1659 into master will increase coverage by 5.4%.
The diff coverage is 72.1%.

@@           Coverage Diff            @@
##           master   #1659     +/-   ##
========================================
+ Coverage    68.2%   73.6%   +5.4%     
========================================
  Files         256     278     +22     
  Lines        9442   10918   +1476     
  Branches     1010    1179    +169     
========================================
+ Hits         6442    8046   +1604     
+ Misses       2777    2529    -248     
- Partials      223     343    +120     

Flag	Coverage Δ
#integrationtests	56.7% <ø> (?)
#unittests	67.4% <72.1%> (-0.9%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...vices/storage/src/simcore_service_storage/utils.py	69.5% <62.5%> (ø)
...ervices/storage/src/simcore_service_storage/dsm.py	65.0% <73.5%> (ø)
...es/storage/src/simcore_service_storage/settings.py	100.0% <0.0%> (ø)
...ces/storage/src/simcore_service_storage/datcore.py	15.4% <0.0%> (ø)
...s/storage/src/simcore_service_storage/db_tokens.py	90.0% <0.0%> (ø)
...s/storage/src/simcore_service_storage/resources.py	100.0% <0.0%> (ø)
...age/src/simcore_service_storage/datcore_wrapper.py	56.1% <0.0%> (ø)
...storage/src/simcore_service_storage/application.py	46.6% <0.0%> (ø)
...ervices/storage/src/simcore_service_storage/cli.py	75.7% <0.0%> (ø)
...es/storage/src/simcore_service_storage/__init__.py	100.0% <0.0%> (ø)
... and 44 more

[sanderegg]

I think you can use tenacity for the exponential backoff, and it works through a decorator:
https://github.com/jd/tenacity

[sanderegg]

why not use tenacity here? I think most of these features are available in the @Retry decorator.

stuff like this:

@retry(wait=wait_exponential(multiplier=1, min=4, max=10))
    def wait_exponential_1():
        print("Wait 2^x * 1 second between each retry starting with 4 seconds, then up to 10 seconds, then 10 seconds afterwards")
        raise Exception

[GitHK]

Because it was too generic for my use case. I needed to sleep only for a subsection of the function

Edit: I have skipped tenacity on purpose here. Also the exponential backoff generator implementation comes from the backoff library.

[odeimaiz]

Demo worked like a charm.