✨ Webserver reports copy progress

[sanderegg]

What do these changes do?

webserver micro-service

• POST /projects now reports progress of copy as % and gives info on how much data was already copied in its message
• frontend will receive a progress between 0.0 and 1.0 together with a message of containing the progress in absolute data size value

storage micro-service

• changes POST /{api_vtag}/simcore-s3/folders to a long running task (e.g. Accepted, get status, get result)

@odeimaiz : this will need your 🪄 here

Related issue/s

NOTE: first #3290 must go in

How to test

Checklist

[codecov]

Codecov Report

Merging #3287 (07c1174) into master (07c1174) will not change coverage.
The diff coverage is n/a.

❗ Current head 07c1174 differs from pull request most recent head a83b282. Consider uploading reports for the commit a83b282 to get more accurate results

@@          Coverage Diff           @@
##           master   #3287   +/-   ##
======================================
  Coverage    76.2%   76.2%           
======================================
  Files         321     321           
  Lines       10863   10863           
  Branches     1541    1541           
======================================
  Hits         8288    8288           
  Misses       2367    2367           
  Partials      208     208           

Flag	Coverage Δ
unittests	76.2% <0.0%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[GitHK]

Very nice, I've appreciated the long running tasks interface.

[pcrespov]

Excellent work.
Some minor comments and one request: can we limit the digits in the progress to at most 3 ? e.g. for a progress between 0 - 1, I find a bit strange to have to deal with the digits that are not highlighted 0.051234567899

[pcrespov]

curious that the generator takes now the get instead of the put path to find the schema. I guess this has to do with the version issue we saw some days ago.

[sanderegg]

yes I'll regenerate it again at the office to be sure...

[pcrespov]

Just to see if i understood: here you are connecting two long-running tasks

So this mean that the webserver waits (i.e. polls storage until tasks is completed) until the all the data is fully copied. since this coroutine is awaited within create_project. Then the webserver handler for CREATE project is built as a long running task as well.

[sanderegg]

yes 2 long running tasks are connected here.

technically the create_project was already waiting for storage to complete. The long running task allows:

• no super large request anymore (the request can spend several minutes)
• it is potentially possible to retry in case of network issue (retrials, this is not yet the case in the frontend, but the client does retry on connection lost (60 seconds)
• progress reporting is now possible (and is implemented as such)
• potentially could also return faster if the request completed fast (not implemented)

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
0.0% Duplication

[elisabettai]

I've just tried to copy a "large" (10-20 Gb on AWS staging) and it fails with a 500.

[sanderegg]

@elisabettai : the frontend was improved regarding retries as well in #3326
this is not yet in staging and I would try again when this one is in. Then if this happens again we shall open a new issue, and link it with ITISFoundation/osparc-issues#558