🐛Computational backend: disappearing logs

[sanderegg]

What do these changes do?

• upgrade all libraries in dask-sidecar, osparc-dask-gateway
• upgrade related dask libraries in director-v2
• refactoring in dask-sidecar
• additional tests to pass huge amount of logs to test dask Pub/Sub mechanism

IMPORTANTT NOTE:

• very big issue arises from version 2023.4 from dask. created an issue in dask backlog to get more information see here
• in case this does not get fixed we might have a severe issue.

Related issue/s

• relates to Dask Backend: some logs are missing in the UI #4127
• relates to M1-12 Maintenance and DevOps osparc-issues#675

Highlights on updated libraries (only updated libraries are included)

• #packages before: 36
• #packages after : 37

#	name	before	after	upgrade	count	packages
1	aio-pika	9.0.4	9.0.7		1	dask-sidecar⬆️
2	aiobotocore	2.4.2	2.5.0	minor	1	dask-sidecar⬆️
3	aiormq	6.7.3	6.7.6		1	dask-sidecar⬆️
4	attrs	22.2.0	23.1.0	MAJOR	2	osparc-gateway-server⬆️🧪
5	aws-sam-translator	1.66.0	1.67.0	minor	1	dask-sidecar🧪
6	boto3	1.24.59	1.26.76	minor	1	dask-sidecar🧪
7	botocore	1.27.59	1.29.76	minor	2	dask-sidecar⬆️🧪
8	certifi	2022.12.7	2023.5.7	MAJOR	2	dask-sidecar⬆️🧪
9	coverage	7.2.5	7.2.6		1	dask-sidecar🧪
10	cryptography	39.0.2	40.0.2	MAJOR	1	osparc-gateway-server⬆️
11	dask	2023.3.0	2023.5.0,2023.3.2	minor	6	dask-sidecar⬆️⬆️ director-v2⬆️🧪 osparc-gateway-server🧪🧪
12	distributed	2023.3.0	2023.5.0,2023.3.2	minor	6	dask-sidecar⬆️⬆️ director-v2⬆️🧪 osparc-gateway-server🧪🧪
13	email-validator	1.3.1	2.0.0.post2	MAJOR	2	dask-sidecar⬆️ osparc-gateway-server⬆️
14	faker	18.7.0	18.9.0	minor	2	dask-sidecar🧪 osparc-gateway-server🧪
15	fsspec	2023.3.0	2023.5.0	minor	6	dask-sidecar⬆️⬆️ director-v2⬆️🧪 osparc-gateway-server🧪🧪
16	heapdict	1.0.1	🗑️ removed		6	dask-sidecar⬆️⬆️ director-v2⬆️🧪 osparc-gateway-server🧪🧪
17	moto	4.1.9	4.1.10		1	dask-sidecar🧪
18	numpy	1.24.2	1.24.3		1	dask-sidecar⬆️
19	packaging	23.0	23.1	minor	11	dask-sidecar⬆️⬆️🧪🔧 director-v2⬆️🧪🔧 osparc-gateway-server🧪🔧🧪🧪
20	partd	1.3.0	1.4.0	minor	6	dask-sidecar⬆️⬆️ director-v2⬆️🧪 osparc-gateway-server🧪🧪
21	pillow	9.4.0	9.5.0	minor	1	dask-sidecar⬆️
22	pre-commit	3.3.1	3.3.2		1	dask-sidecar🔧
23	psutil	5.9.4	5.9.5		6	dask-sidecar⬆️⬆️ director-v2⬆️🧪 osparc-gateway-server🧪🧪
24	pydantic	1.10.2	1.10.8		3	dask-sidecar⬆️🧪 osparc-gateway-server⬆️
25	pytest-cov	4.0.0	4.1.0	minor	1	dask-sidecar🧪
26	redis	4.5.4	4.5.5		1	dask-sidecar⬆️
27	s3fs	2023.3.0	2023.5.0	minor	1	dask-sidecar⬆️
28	sqlalchemy	1.4.47	1.4.48		2	osparc-gateway-server⬆️🧪
29	tornado	6.2	6.3.2	minor	6	dask-sidecar⬆️⬆️ director-v2⬆️🧪 osparc-gateway-server🧪🧪
30	typer	0.7.0	0.9.0	minor	1	dask-sidecar⬆️
31	types-pyyaml	6.0.12.9	6.0.12.10		1	dask-sidecar🧪
32	typing-extensions	4.5.0	4.6.0,4.6.1	minor	6	dask-sidecar⬆️🧪🔧 osparc-gateway-server⬆️🧪🔧
33	urllib3	1.26.14	1.26.16		7	dask-sidecar⬆️⬆️🧪 director-v2⬆️🧪 osparc-gateway-server🧪🧪
34	websocket-client	1.5.1	1.5.2		2	dask-sidecar🧪 osparc-gateway-server🧪
35	yarl	1.8.2	1.9.2	minor	4	dask-sidecar⬆️🧪 osparc-gateway-server⬆️🧪
36	zict	2.2.0	3.0.0	MAJOR	6	dask-sidecar⬆️⬆️ director-v2⬆️🧪 osparc-gateway-server🧪🧪

Legend:

• ⬆️ base dependency (only services because packages are floating)
• 🧪 test dependency
• 🔧 tool dependency

How to test

DevOps Checklist

[codecov]

Codecov Report

Merging #4265 (acaef9b) into master (549bc81) will increase coverage by 1.9%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##           master   #4265      +/-   ##
=========================================
+ Coverage    85.6%   87.6%    +1.9%     
=========================================
  Files         957     558     -399     
  Lines       41549   26649   -14900     
  Branches      963     173     -790     
=========================================
- Hits        35596   23359   -12237     
+ Misses       5732    3241    -2491     
+ Partials      221      49     -172     

Flag	Coverage Δ
integrationtests	68.9% <ø> (-0.1%)	⬇️
unittests	84.5% <100.0%> (+2.0%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...car/src/simcore_service_dask_sidecar/file_utils.py	98.1% <ø> (ø)
...ce_dask_sidecar/computational_sidecar/constants.py	100.0% <100.0%> (ø)
...service_dask_sidecar/computational_sidecar/core.py	100.0% <100.0%> (ø)
...dask_sidecar/computational_sidecar/docker_utils.py	94.0% <100.0%> (+2.9%)	⬆️
...car/src/simcore_service_dask_sidecar/dask_utils.py	97.5% <100.0%> (+1.1%)	⬆️

... and 417 files with indirect coverage changes

[codeclimate]

Code Climate has analyzed commit acaef9b and detected 0 issues on this pull request.

View more on Code Climate.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication