🐛 Add project name search parameter for project listing for the API server usecase 🚨 #7066
+63
−24
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #7066 +/- ##
==========================================
- Coverage 88.30% 83.67% -4.63%
==========================================
Files 1484 752 -732
Lines 58987 35336 -23651
Branches 1299 262 -1037
==========================================
- Hits 52088 29569 -22519
+ Misses 6697 5707 -990
+ Partials 202 60 -142
Continue to review full report in Codecov by Sentry.
|
matusdrobuliak66
added
a:webserver
matusdrobuliak66
requested review from
pcrespov,
sanderegg and
GitHK
as code owners
matusdrobuliak66
requested review from
giancarloromeo,
odeimaiz and
bisgaard-itis
matusdrobuliak66
changed the title
sanderegg
approved these changes
Jan 21, 2025
services/web/server/src/simcore_service_webserver/projects/_crud_api_read.py
Outdated
Show resolved
Hide resolved
bisgaard-itis
approved these changes
Jan 22, 2025
GitHK
approved these changes
Jan 22, 2025
…:matusdrobuliak66/osparc-simcore into bugfix/listing-of-projects-in-api-server
pcrespov
approved these changes
Jan 22, 2025
services/api-server/src/simcore_service_api_server/services_http/webserver.py
Show resolved
Hide resolved
services/api-server/src/simcore_service_api_server/services_http/webserver.py
Show resolved
Hide resolved
|
matusdrobuliak66
merged commit e2aeff5
into
ITISFoundation:master
89 of 93 checks passed
pcrespov
changed the title
15 tasks
matusdrobuliak66
added a commit
that referenced
this pull request
Jan 24, 2025
This was referenced Jan 28, 2025
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
On osparc.io, we observed overloading of the database.
Currently, our database has only 2 CPUs. This graph shows that, on average, we would need 70 CPUs to handle the traffic.
The issue is primarily when api server calls "list projects" query, which involves multiple column searches, especially for the end-to-end testing user (who has 1,800 hidden projects).
Initial analysis showed that the query takes 93 seconds to execute. Since the API server does not require multi-column search, I performed some tests by removing unnecessary search fields and leaving only the project name. This reduced the query time to 150 ms. Furthermore, changing ILIKE to LIKE, which is even lighter, reduced it further to 73 ms.
This is why I introduced a new filter field specifically for the API server, projects_name_search. This approach can also be easily hotfixed if needed.
After further investigation, it turned out that the issue is not directly related to multi-column search itself but rather to how the database engine plans the execution. For some reason, when multiple joins and multiple text ILIKE filters are involved, the engine appears to get confused and does not create an optimal execution plan. For example, it first joins the projects table with the project_tags table (30,000 projects × 4,000 project tags ≈ 120,000,000 rows) and only afterward performs the multi-column search.
By simplifying the query for the API server use case, the engine started to create a more optimal execution plan.
Other steps that should be taken:
@cancel_on_disconnectto each route.Related issue/s
How to test
Dev-ops checklist