Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 More than 1 input port containing files can be safely pulled #6286

Merged
merged 11 commits into from
Sep 3, 2024

Conversation

GitHK
Copy link
Contributor

@GitHK GitHK commented Sep 2, 2024

What do these changes do?

This issue was simple to reproduce, but we never had a good report. This was a nightmare for so many people reporting missing files in their inputs.

If input ports contain more than one port with files, these would get downloaded two at a time.
What happened? threading.get_ident() would return a unique id for the given thread. Too bad that this ID can be reassigned. To avoid future issues uuid4 replaced it.

It is now possible to pull 10 ports with each containing a 1GB file without any issue. It now works reliably every time.

Before this, almost always you would have issues when pulling 5 ports with a 1GB file in each port.

Related issue/s

How to test

Dev-ops checklist

@GitHK GitHK added this to the Eisbock milestone Sep 2, 2024
@GitHK GitHK self-assigned this Sep 2, 2024
@GitHK GitHK added t:maintenance Some planned maintenance work a:dynamic-sidecar dynamic-sidecar service labels Sep 2, 2024
Copy link

codecov bot commented Sep 2, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.6%. Comparing base (cafbf96) to head (8b1b968).
Report is 494 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #6286      +/-   ##
=========================================
+ Coverage    84.5%   88.6%    +4.0%     
=========================================
  Files          10    1028    +1018     
  Lines         214   45865   +45651     
  Branches       25     260     +235     
=========================================
+ Hits          181   40667   +40486     
- Misses         23    5140    +5117     
- Partials       10      58      +48     
Flag Coverage Δ
integrationtests 64.8% <90.0%> (?)
unittests 85.9% <100.0%> (+1.4%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
.../simcore_sdk/node_ports_common/data_items_utils.py 100.0% <100.0%> (ø)
...re-sdk/src/simcore_sdk/node_ports_v2/port_utils.py 69.0% <100.0%> (ø)

... and 1036 files with indirect coverage changes

Andrei Neagu added 2 commits September 2, 2024 13:39
@GitHK GitHK marked this pull request as ready for review September 2, 2024 11:57
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try to use pytest mocker spy: https://pytest-mock.readthedocs.io/en/latest/usage.html#spy

This would allow to not have weird constants that are linked to other worldly defined fixtures..

packages/simcore-sdk/tests/conftest.py Outdated Show resolved Hide resolved
@GitHK GitHK requested review from pcrespov and sanderegg September 2, 2024 13:16
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

discussed. waiting for changes. thanks!

Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx.
Could you please add a small note to stress that we are not creating the folder but just generating a random path name?

Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the changes. just a few small things. thanks!

Copy link

sonarqubecloud bot commented Sep 3, 2024

@GitHK GitHK merged commit dbd4cd9 into ITISFoundation:master Sep 3, 2024
57 checks passed
@GitHK GitHK deleted the pr-osparc-inputs-pulling-issue branch September 3, 2024 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:dynamic-sidecar dynamic-sidecar service t:maintenance Some planned maintenance work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Urgent bug - download of multiple inputs fails
6 participants