♻️✨ Comp backend task state reporting fixed

[sanderegg]

What do these changes do?

This PR shall fix the issues where the Running state of a task was shown although the task is not computing, but is taken by a dask-sidecar (Dask scheduler states make no difference between a task being processed or taken by a worker and queued on the worker memory).
Therefore this shall fix:

• when multiple computational services where shown as Running
• and also the Running state is now shown as soon as the computational service signaled it started progressing, therefore linked directly to the worker activity and not only from the dask-scheduler processing fct
• pending <-> waiting_for_cluster state now stable

This was achieved by:

• properly separating Dask task state and RunningState
• since Dask does not know the difference between computing and queued in worker, this can be computed on the dv-2 by checking if progress was already signaled through the Dask Pub/Sub mechanism

Bonus:

• ensure that RPC calls do not return internal exceptions

Related issue/s

• fixes Computational backend: state is reported as Pending even though the computation already started and logs are shown #4517
• fixes UI says that a comp. service is running, while it is not true #4756

How to test

DevOps Checklist

[elisabettai]

Very neat, thanks for the fix, dask-master! 🙃

[codecov]

Codecov Report

Merging #4775 (1e64f5d) into master (4af4b44) will increase coverage by 0.7%.
The diff coverage is 85.8%.

@@           Coverage Diff            @@
##           master   #4775     +/-   ##
========================================
+ Coverage    86.8%   87.5%   +0.7%     
========================================
  Files        1143    1097     -46     
  Lines       47699   45957   -1742     
  Branches     1015     861    -154     
========================================
- Hits        41425   40241   -1184     
+ Misses       6043    5517    -526     
+ Partials      231     199     -32     

Flag	Coverage Δ
integrationtests	65.0% <72.3%> (+0.4%)	⬆️
unittests	85.2% <80.1%> (+0.5%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...ore_service_director_v2/modules/clusters_keeper.py	45.8% <0.0%> (-4.2%)	⬇️
...imcore_service_director_v2/utils/comp_scheduler.py	100.0% <ø> (ø)
...rector_v2/modules/comp_scheduler/base_scheduler.py	89.8% <68.9%> (-1.1%)	⬇️
..._director_v2/modules/db/repositories/comp_tasks.py	96.6% <84.2%> (-1.0%)	⬇️
...rector_v2/modules/comp_scheduler/dask_scheduler.py	92.1% <97.2%> (+1.4%)	⬆️
...ice-library/src/servicelib/rabbitmq/_rpc_router.py	90.6% <100.0%> (-0.3%)	⬇️
...ore_service_director_v2/api/routes/computations.py	91.1% <100.0%> (+20.9%)	⬆️
...-v2/src/simcore_service_director_v2/core/errors.py	78.3% <100.0%> (+0.3%)	⬆️
...mcore_service_director_v2/models/dask_subsystem.py	100.0% <100.0%> (ø)
...simcore_service_director_v2/modules/dask_client.py	93.0% <100.0%> (+0.5%)	⬆️

... and 53 files with indirect coverage changes

[pcrespov]

thx!
Is there a reasonable way to test this bug so it does not happen again?

[sanderegg]

thx! Is there a reasonable way to test this bug so it does not happen again?

@pcrespov you might remember we did talk about simplifying the director-v2 scheduler, and that is when we finally decided to remove one of the signals for sending service states. that is when we broke it actually.

[codeclimate]

Code Climate has analyzed commit 1e64f5d and detected 0 issues on this pull request.

View more on Code Climate.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication