Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

⬆️Upgrade dask-based services #4536

Merged

Conversation

sanderegg
Copy link
Member

Highlights on updated libraries (only updated libraries are included)

  • #packages before: 40
  • #packages after : 50
# name before after upgrade count packages
1 aio-pika 9.1.4 9.1.2 🔥 downgrade 2 director-v2⬆️🧪
2 aiobotocore 2.5.0 2.5.2 1 dask-sidecar⬆️
3 aiohttp 3.8.4 3.8.5 6 dask-sidecar⬆️🧪
director-v2⬆️🧪
osparc-gateway-server⬆️🧪
4 attrs 21.4.0 23.1.0 MAJOR 2 dask-sidecar⬆️🧪
5 boto3 1.26.76 1.26.161 1 dask-sidecar🧪
6 botocore 1.29.76 1.29.161 2 dask-sidecar⬆️🧪
7 certifi 2023.5.7 2023.7.22 minor 6 dask-sidecar⬆️🧪
dask-task-models-library🧪
director-v2⬆️🧪
osparc-gateway-server🧪
8 cfn-lint 0.78.1 0.77.10 🔥 downgrade 1 dask-sidecar🧪
9 charset-normalizer 3.1.0 3.2.0 minor 4 dask-sidecar⬆️🧪
osparc-gateway-server⬆️🧪
10 click 8.1.3 8.1.6 9 dask-sidecar⬆️⬆️🧪🔧
director-v2⬆️🧪🔧
osparc-gateway-server🧪🔧
11 cryptography 40.0.2 41.0.2 MAJOR 1 osparc-gateway-server⬆️
12 dask 2023.7.0 2023.7.1 1 dask-task-models-library🧪
13 dill 0.3.6 0.3.7 4 dask-sidecar🔧
dask-task-models-library🔧
director-v2🔧
osparc-gateway-server🔧
14 distributed 2023.7.0 2023.7.1 1 dask-task-models-library🧪
15 dnspython 2.3.0 2.4.0 minor 2 dask-sidecar⬆️
osparc-gateway-server⬆️
16 faker 19.1.0 19.2.0 minor 4 dask-sidecar🧪
dask-task-models-library🧪
director-v2🧪
osparc-gateway-server🧪
17 frozenlist 1.3.3 1.4.0 minor 4 dask-sidecar⬆️🧪
osparc-gateway-server⬆️🧪
18 fsspec 2023.5.0 2023.6.0 minor 5 dask-sidecar⬆️⬆️
director-v2⬆️🧪
osparc-gateway-server🧪
19 identify 2.5.25 2.5.26 4 dask-sidecar🔧
dask-task-models-library🔧
director-v2🔧
osparc-gateway-server🔧
20 importlib-metadata 6.6.0 6.8.0 minor 5 dask-sidecar⬆️⬆️
director-v2⬆️🧪
osparc-gateway-server🧪
21 jsonschema 3.2.0 4.18.4 MAJOR 2 dask-sidecar⬆️🧪
22 markupsafe 2.1.2 2.1.3 6 dask-sidecar⬆️⬆️🧪
director-v2⬆️🧪
osparc-gateway-server🧪
23 numpy 1.24.3 1.25.1 minor 1 dask-sidecar⬆️
24 openapi-schema-validator 0.2.3 0.6.0 minor 1 dask-sidecar🧪
25 openapi-spec-validator 0.4.0 0.6.0 minor 1 dask-sidecar🧪
26 pillow 9.5.0 10.0.0 MAJOR 1 dask-sidecar⬆️
27 pydantic 1.10.8 1.10.11 3 dask-sidecar⬆️🧪
osparc-gateway-server⬆️
28 pyinstrument 4.5.0, 4.4.0 4.5.1 2 dask-sidecar⬆️
director-v2⬆️
29 pyrsistent 0.19.3 🗑️ removed 2 dask-sidecar⬆️🧪
30 redis 4.5.5 4.6.0 minor 1 dask-sidecar⬆️
31 referencing 0.30.0 0.29.3 🔥 downgrade 1 director-v2⬆️
32 respx 0.20.1 0.20.2 1 director-v2🧪
33 ruff 0.0.278 0.0.280 4 dask-sidecar🔧
dask-task-models-library🔧
director-v2🔧
osparc-gateway-server🔧
34 s3fs 2023.5.0 2023.6.0 minor 1 dask-sidecar⬆️
35 sqlalchemy 1.4.48 1.4.49 2 osparc-gateway-server⬆️🧪
36 tblib 1.7.0 2.0.0 MAJOR 5 dask-sidecar⬆️⬆️
director-v2⬆️🧪
osparc-gateway-server🧪
37 types-pyyaml 6.0.12.10 6.0.12.11 1 dask-sidecar🧪
38 typing-extensions 4.6.1, 4.6.0 4.7.1 minor 6 dask-sidecar⬆️🧪🔧
osparc-gateway-server⬆️🧪🔧
39 wheel 0.40.0 0.41.0 minor 4 dask-sidecar🔧
dask-task-models-library🔧
director-v2🔧
osparc-gateway-server🔧
40 zipp 3.15.0 3.16.2 minor 5 dask-sidecar⬆️⬆️
director-v2⬆️🧪
osparc-gateway-server🧪

Legend:

  • ⬆️ base dependency (only services because packages are floating)
  • 🧪 test dependency
  • 🔧 tool dependency

Repo-wide overview of libraries

  • #reqs files parsed: 76
# name versions-base versions-test versions-tool
1 aio-pika 9.1.2 9.1.2
2 aioboto3 9.6.0, 10.4.0 9.6.0, 11.2.0
3 aiobotocore 2.3.0, 2.4.2, 2.5.2 2.3.0, 2.5.0
4 aiocache 0.11.1, 0.12.1 0.12.1
5 aiodebug 2.3.0 2.3.0
6 aiodocker 0.19.1, 0.21.0 0.21.0
7 aiofiles 0.8.0, 22.1.0, 23.1.0 23.1.0
8 aiohttp 3.8.3, 3.8.4, 3.8.5 3.8.3, 3.8.4, 3.8.5
9 aiohttp-jinja2 1.5
10 aiohttp-security 0.4.0
11 aiohttp-session 2.11.0
12 aiohttp-swagger 1.0.16
13 aioitertools 0.10.0, 0.11.0 0.11.0
14 aiopg 1.4.0 1.4.0
15 aioprocessing 2.0.1
16 aioredis 2.0.1
17 aioresponses 0.7.4
18 aiormq 6.7.6 6.7.6
19 aiosignal 1.2.0, 1.3.1 1.2.0, 1.3.1
20 aiosmtplib 1.1.6
21 aiozipkin 1.1.1
22 alembic 1.8.1, 1.11.1 1.8.1, 1.11.1
23 anyio 3.6.2, 3.7.0, 3.7.1 3.6.2, 3.7.0, 3.7.1
24 arrow 1.2.3 1.2.3
25 asgi-lifespan 2.1.0
26 asgiref 3.5.2
27 astroid 2.15.6
28 async-asgi-testclient 1.4.11
29 async-timeout 4.0.2 4.0.2
30 asyncpg 0.27.0, 0.28.0 0.28.0
31 attrs 21.4.0, 23.1.0 21.4.0, 23.1.0
32 aws-sam-translator 1.55.0, 1.71.0
33 aws-xray-sdk 2.12.0
34 bcrypt 3.2.0
35 bidict 0.22.0
36 black 23.7.0
37 blinker 1.6.2
38 blosc 1.11.1
39 bokeh 2.4.3 2.4.3
40 boto3 1.21.21, 1.24.59, 1.24.96 1.21.21, 1.24.59, 1.26.76, 1.26.161, 1.28.7
41 boto3-stubs 1.28.7
42 botocore 1.24.21, 1.27.59, 1.27.96, 1.29.161 1.24.21, 1.27.59, 1.29.76, 1.29.161, 1.31.7
43 botocore-stubs 1.27.17, 1.29.78 1.31.7
44 build 0.10.0
45 bump2version 1.0.1
46 certifi 2022.12.7, 2023.5.7, 2023.7.22 2022.12.7, 2023.5.7, 2023.7.22
47 cffi 1.15.0, 1.15.1 1.15.1
48 cfgv 3.3.1
49 cfn-lint 0.72.0, 0.72.6, 0.77.10, 0.78.1
50 change-case 0.5.2
51 charset-normalizer 2.0.12, 2.1.1, 3.0.1, 3.1.0, 3.2.0 2.0.12, 2.1.1, 3.0.1, 3.1.0, 3.2.0
52 click 8.1.3, 8.1.6 8.1.3, 8.1.6 8.1.3, 8.1.6
53 cloudpickle 2.2.1 2.2.1
54 colorama 0.4.6
55 colorlog 6.7.0 6.7.0
56 commonmark 0.9.1
57 contourpy 1.0.7
58 coverage 7.2.7
59 cryptography 39.0.1, 41.0.1, 41.0.2 41.0.1, 41.0.2
60 cycler 0.11.0
61 cython 0.29.36
62 dask 2023.3.2, 2023.7.1 2023.3.2
63 dask-gateway 2023.1.1 2023.1.1
64 dask-gateway-server 2023.1.1 2023.1.1
65 dateparser 1.1.8
66 debugpy 1.6.7
67 deepdiff 6.3.1
68 dill 0.3.6, 0.3.7
69 distlib 0.3.7
70 distributed 2023.3.2, 2023.7.1 2023.3.2
71 distro 1.5.0
72 dnspython 2.1.0, 2.2.1, 2.3.0, 2.4.0 2.4.0
73 docker 6.0.0, 6.1.3 6.1.3
74 docker-compose 1.29.1
75 dockerpty 0.4.1
76 docopt 0.6.2
77 ecdsa 0.18.0
78 email-validator 1.2.1, 1.3.0, 1.3.1, 2.0.0.post2 2.0.0.post2
79 et-xmlfile 1.1.0
80 exceptiongroup 1.1.1, 1.1.2 1.1.1, 1.1.2
81 execnet 2.0.2
82 expiringdict 1.2.1
83 faker 19.1.0, 19.2.0
84 fakeredis 2.17.0
85 fastapi 0.96.0, 0.98.0, 0.99.1
86 fastapi-pagination 0.10.0, 0.12.5
87 filelock 3.12.2
88 flaky 3.7.0
89 flask 2.1.3, 2.2.5, 2.3.2
90 flask-cors 4.0.0
91 fonttools 4.39.4
92 frozenlist 1.3.0, 1.3.1, 1.3.3, 1.4.0 1.3.0, 1.3.1, 1.3.3, 1.4.0
93 fsspec 2023.6.0 2023.5.0, 2023.6.0
94 graphql-core 3.2.3
95 greenlet 2.0.2 2.0.2
96 gunicorn 20.1.0
97 h11 0.12.0, 0.14.0 0.12.0, 0.14.0
98 h2 4.1.0
99 hpack 4.0.0
100 httmock 1.4.0
101 httpcore 0.15.0, 0.16.3, 0.17.1, 0.17.2, 0.17.3 0.15.0, 0.16.3, 0.17.1, 0.17.2, 0.17.3
102 httptools 0.2.0, 0.5.0, 0.6.0
103 httpx 0.24.0, 0.24.1 0.24.0, 0.24.1
104 hyperframe 6.0.1
105 hypothesis 6.82.0
106 icdiff 2.0.6
107 identify 2.5.25, 2.5.26
108 idna 2.10, 3.3, 3.4 2.10, 3.3, 3.4
109 importlib-metadata 6.8.0 6.6.0, 6.8.0
110 iniconfig 2.0.0 2.0.0
111 inotify 0.2.10
112 isodate 0.6.1
113 isort 5.12.0
114 itsdangerous 1.1.0, 2.1.2 2.1.2
115 jinja-app-loader 1.0.2
116 jinja2 3.1.2 3.1.2 3.1.2
117 jmespath 1.0.0, 1.0.1 1.0.0, 1.0.1
118 jschema-to-python 1.2.3
119 json2html 1.3.0
120 jsondiff 2.0.0 2.0.0
121 jsonpatch 1.33
122 jsonpickle 3.0.1
123 jsonpointer 2.4
124 jsonschema 3.2.0, 4.18.4 3.2.0, 4.18.4
125 jsonschema-spec 0.2.3
126 jsonschema-specifications 2023.7.1 2023.7.1
127 junit-xml 1.9
128 kiwisolver 1.4.4
129 lazy-object-proxy 1.7.1 1.9.0 1.7.1, 1.9.0
130 locket 1.0.0 1.0.0
131 lupa 1.14.1
132 lz4 4.3.2 4.3.2
133 mako 1.2.2, 1.2.4 1.2.2, 1.2.4
134 markdown-it-py 2.2.0, 3.0.0 3.0.0
135 markupsafe 2.1.1, 2.1.3 2.1.1, 2.1.2, 2.1.3 2.1.3
136 matplotlib 3.7.1
137 mccabe 0.7.0
138 mdurl 0.1.2 0.1.2
139 minio 7.0.4
140 more-itertools
141 moto 4.0.1, 4.1.13
142 mpmath 1.3.0
143 msgpack 1.0.3, 1.0.5 1.0.5
144 multidict 6.0.2, 6.0.3, 6.0.4 6.0.2, 6.0.4
145 mypy 1.4.1
146 mypy-extensions 1.0.0 1.0.0
147 networkx 3.1 2.8.8, 3.1
148 nodeenv 1.8.0
149 nose 1.3.7
150 numpy 1.24.3, 1.25.1 1.25.1
151 openapi-core 0.12.0
152 openapi-schema-validator 0.2.3 0.2.3, 0.6.0
153 openapi-spec-validator 0.4.0 0.4.0, 0.6.0
154 openpyxl 3.0.9
155 ordered-set 4.1.0 4.1.0
156 orjson 3.7.2, 3.9.1, 3.9.2
157 packaging 23.0, 23.1 23.0, 23.1 23.0, 23.1
158 pamqp 3.2.1 3.2.1
159 pandas 2.0.1 2.0.3
160 paramiko 2.11.0
161 parse
162 partd 1.4.0 1.4.0
163 passlib 1.7.4
164 pathable 0.4.3
165 pathspec 0.11.1
166 pbr 5.11.1
167 pillow 9.5.0, 10.0.0 10.0.0
168 pint 0.19.2, 0.22 0.22
169 pip-tools 7.1.0
170 platformdirs 3.9.1
171 pluggy 1.2.0 1.2.0
172 pprintpp 0.4.0
173 pre-commit 3.3.3
174 prometheus-api-client 0.5.3
175 prometheus-client 0.14.1
176 psutil 5.9.1, 5.9.5 5.9.5
177 psycopg2-binary 2.9.6 2.9.6
178 ptvsd 4.3.2
179 py-cpuinfo 9.0.0
180 py-partiql-parser 0.3.3
181 pyasn1 0.5.0
182 pycparser 2.20, 2.21 2.21
183 pydantic 1.9.0, 1.10.2, 1.10.7, 1.10.9, 1.10.11 1.10.2, 1.10.11
184 pyftpdlib 1.5.7
185 pygments 2.14.0, 2.15.1 2.15.1
186 pyinstrument 3.4.2, 4.1.1, 4.3.0, 4.4.0, 4.5.0, 4.5.1 4.5.0
187 pyinstrument-cext 0.2.4
188 pyjwt 2.4.0
189 pylint 2.17.4
190 pynacl 1.4.0
191 pyopenssl 23.2.0
192 pyparsing 3.0.9 3.1.0
193 pyproject-hooks 1.0.0
194 pyrsistent 0.18.1, 0.19.2, 0.19.3 0.18.1, 0.19.2, 0.19.3
195 pytest 7.4.0 7.4.0
196 pytest-aiohttp 1.0.4
197 pytest-asyncio 0.21.1
198 pytest-benchmark 4.0.0
199 pytest-cov 4.1.0
200 pytest-docker 2.0.0
201 pytest-icdiff 0.6
202 pytest-instafail 0.5.0
203 pytest-lazy-fixture 0.6.3
204 pytest-localftpserver 1.1.4
205 pytest-mock 3.11.1
206 pytest-runner 6.0.0
207 pytest-sugar 0.9.7
208 pytest-xdist 3.3.1
209 python-dateutil 2.8.2 2.8.2
210 python-dotenv 0.20.0, 0.21.0, 1.0.0 0.21.0, 1.0.0
211 python-engineio 4.3.4
212 python-jose 3.3.0
213 python-magic 0.4.25
214 python-multipart 0.0.5, 0.0.6
215 python-socketio 5.8.0
216 pytz 2022.1, 2023.3 2023.3
217 pyyaml 5.4.1, 6.0.1 6.0.1 5.4.1, 6.0.1
218 redis 4.5.4, 4.5.5, 4.6.0 4.5.4, 4.5.5, 4.6.0
219 referencing 0.29.3, 0.30.0 0.29.3, 0.30.0
220 regex 2023.5.5 2023.6.3
221 requests 2.30.0, 2.31.0 2.30.0, 2.31.0
222 requests-mock 1.11.0
223 responses 0.23.1
224 respx 0.20.1, 0.20.2
225 rfc3339-validator 0.1.4
226 rich 12.6.0, 13.3.5, 13.4.2 13.4.2
227 rpds-py 0.9.2 0.9.2
228 rsa 4.9
229 ruff 0.0.278, 0.0.280
230 s3fs 2023.6.0
231 s3transfer 0.5.2, 0.6.0 0.5.2, 0.6.0, 0.6.1
232 sarif-om 1.0.4
233 semantic-version 2.9.0
234 setproctitle 1.2.3
235 shellingham 1.5.0.post1
236 six 1.15.0, 1.16.0 1.15.0, 1.16.0
237 sniffio 1.2.0, 1.3.0 1.2.0, 1.3.0
238 sortedcontainers 2.4.0 2.4.0
239 sqlalchemy 1.4.47, 1.4.48, 1.4.49 1.4.47, 1.4.48, 1.4.49
240 sqlalchemy2-stubs 0.0.2a35
241 sshpubkeys 3.3.1
242 starlette 0.27.0
243 strict-rfc3339 0.7
244 sympy 1.12
245 tblib 2.0.0 1.7.0, 2.0.0
246 tenacity 8.0.1, 8.1.0, 8.2.1, 8.2.2 8.0.1, 8.2.2
247 termcolor 2.3.0
248 texttable 1.6.3
249 tomli 2.0.1 2.0.1 2.0.1
250 tomlkit 0.11.8
251 toolz 0.12.0 0.12.0
252 tornado 6.3.2 6.3.2
253 tqdm 4.64.0, 4.64.1, 4.65.0 4.65.0
254 traitlets 5.9.0 5.9.0
255 twilio 7.12.0
256 typer 0.4.1, 0.6.1, 0.7.0, 0.9.0 0.9.0 0.9.0
257 types-aiobotocore 2.3.3, 2.4.2.post1
258 types-aiobotocore-ec2 2.4.2
259 types-aiobotocore-s3 2.3.3
260 types-aiofiles 23.1.0.4
261 types-awscrt 0.16.10 0.16.26
262 types-boto3 1.0.2
263 types-pkg-resources 0.1.3
264 types-pyyaml 6.0.12.10, 6.0.12.11
265 types-s3transfer 0.6.1
266 typing-extensions 4.3.0, 4.4.0, 4.5.0, 4.6.3, 4.7.1 4.3.0, 4.4.0, 4.5.0, 4.6.3, 4.7.1 4.3.0, 4.4.0, 4.5.0, 4.6.3, 4.7.1
267 tzdata 2023.3 2023.3
268 tzlocal 5.0.1
269 ujson 5.5.0, 5.8.0
270 urllib3 1.26.9, 1.26.11, 1.26.12, 1.26.14, 1.26.16, 2.0.2, 2.0.4 1.26.9, 1.26.11, 1.26.12, 1.26.14, 1.26.16, 2.0.2, 2.0.4
271 uvicorn 0.15.0, 0.17.0, 0.19.0, 0.20.0, 0.22.0, 0.23.1
272 uvloop 0.16.0, 0.17.0
273 virtualenv 20.24.0, 20.24.1
274 watchdog 2.1.5 3.0.0
275 watchfiles 0.18.0, 0.18.1, 0.19.0
276 watchgod 0.8.2
277 websocket-client 0.59.0, 1.6.1 0.59.0, 1.6.1
278 websockets 10.1, 10.3, 10.4, 11.0.3 11.0.3
279 werkzeug 2.1.2, 2.2.2 2.1.2, 2.2.2, 2.3.6
280 wheel 0.40.0, 0.41.0
281 wrapt 1.14.1, 1.15.0 1.14.1, 1.15.0 1.14.1, 1.15.0
282 xmltodict 0.13.0
283 yarl 1.5.1, 1.9.2 1.5.1, 1.9.2
284 zict 3.0.0 3.0.0
285 zipp 3.16.2 3.15.0, 3.16.2

@sanderegg sanderegg added a:director-v2 issue related with the director-v2 service a:dask-service Any of the dask services: dask-scheduler/sidecar or worker a:osparc-gateway-server labels Jul 24, 2023
@sanderegg sanderegg added this to the Sundae milestone Jul 24, 2023
@sanderegg sanderegg requested review from pcrespov and GitHK as code owners July 24, 2023 09:04
@sanderegg sanderegg self-assigned this Jul 24, 2023
@codecov
Copy link

codecov bot commented Jul 24, 2023

Codecov Report

Merging #4536 (29fa3d6) into master (38739ff) will increase coverage by 0.5%.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #4536     +/-   ##
========================================
+ Coverage    86.6%   87.1%   +0.5%     
========================================
  Files        1009     753    -256     
  Lines       43182   34078   -9104     
  Branches      998     194    -804     
========================================
- Hits        37419   29706   -7713     
+ Misses       5533    4320   -1213     
+ Partials      230      52    -178     
Flag Coverage Δ
integrationtests 68.3% <ø> (-0.2%) ⬇️
unittests 84.0% <ø> (-0.2%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 260 files with indirect coverage changes

@sanderegg sanderegg force-pushed the maintenance/upgrade-gateway-server branch from 5733a06 to d4cef10 Compare July 24, 2023 11:42
@sanderegg sanderegg force-pushed the maintenance/upgrade-gateway-server branch from d4cef10 to 7ff7ee3 Compare July 24, 2023 12:06
@sanderegg sanderegg force-pushed the maintenance/upgrade-gateway-server branch from 7ff7ee3 to 29fa3d6 Compare July 25, 2023 06:48
@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@codeclimate
Copy link

codeclimate bot commented Jul 25, 2023

Code Climate has analyzed commit 29fa3d6 and detected 0 issues on this pull request.

View more on Code Climate.

@sanderegg sanderegg merged commit 2d2c040 into ITISFoundation:master Jul 25, 2023
@sanderegg sanderegg deleted the maintenance/upgrade-gateway-server branch July 25, 2023 07:42
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Sep 22, 2023
50 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:dask-service Any of the dask services: dask-scheduler/sidecar or worker a:director-v2 issue related with the director-v2 service a:osparc-gateway-server
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants