Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FREEZE_DaJia1 #1733

Merged
merged 80 commits into from
Aug 21, 2020
Merged

FREEZE_DaJia1 #1733

merged 80 commits into from
Aug 21, 2020

Conversation

sanderegg
Copy link
Member

sanderegg and others added 30 commits June 15, 2020 23:06
@sanderegg
add traefik endpoint to api-gateway (#1555)
03b1525 
added mockserver to simulate apiserver service
fix e2e test: check for api-gateway
@sanderegg
added simcore_webserver_service in pytest simcore package (#1563)
680eb59 
ensure the database is filled up
@pcrespov
Is1269/api-server upgrade (#1475)
821c8e6 
- new api-server refactored
- api-server connected to simcore-stack
@sanderegg
Is/add notebook migration script (#1565)
16c30b9 
[enhancement] add way to import database from a folder
[maintenance] script to upgrade projects that uses a notebook 2.13.0 to use a 2.13.1 notebook
@odeimaiz
Manage groups, Share studies (#1512)
b6a2568 
Backend:
- adds groups endpoint to list, create, get, modify, delete groups through /groups endpoint
- a user in a group has read/write/delete access rights (e.g. member,manager,administrator)
- add/list/modify/delete users in groups through /groups/{gid}/users endpoint
- allow sharing of projects using access_rights defining for each group id the specific read/write/delete rights
- migrates postgres DB accordingly (migrates old access_rights syntax for projects, deprecates user_to_projects table, adds groups thumbnails, )

Frontend:
- New page in Preferences for managing/displaying Organizations and its members
- Testers can create Organizations
- Organization Managers can invite (actually add) osparc existing members by providing their - emails
- Organization Managers can remove members
- Organization Managers can promote members to Manager
- Organization SuperManagers can edit organization details
- Study owners can share the same instance of a study with Organizations and/or Organization Members (Collaborators)
- Study owners can make other collaborators Owner
- Study owners can remove collaborators

Bonus
Backend:
- adds a decorator for checking role permissions
- docs of how to use sc-pg migration capabilities
Frontend:
- Can edit study details also from inside study
- Allow dropping files with no extension
- Show hand cursor over dragable NodeUI ports
- Show move cursor over NodeUI capationbar
- Fix multiFilePicker bug
- Switch button for changing theme
@sanderegg
maintenance fix codecov reports (#1568)
3fb36d0 
* add codecov.yml file
* set up codecov to report for api/packages/services
@dependabot-preview
Bump faker from 4.1.0 to 4.1.1 in /packages/postgres-database (#1573)
bdfcd08 
Bumps [faker](https://github.com/joke2k/faker) from 4.1.0 to 4.1.1.
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.rst)
- [Commits](joke2k/faker@v4.1.0...v4.1.1)

Signed-off-by: dependabot-preview[bot] <[email protected]>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
@pcrespov
Is1570/study fails 500 (#1572)
1b7a47e 
Fixes #1570

* increases verbose in gc errors

* Turned gather errors into warnings

* Fixing issue when valid token with invalid user

* Authz cache cleared when any user is deleted

* Tests fix emulating a valid cookie invalid user

* Enhances error handling
@pcrespov
Maintenance/upgrades and tooling (#1546)
294611d 
* script to create invitations

* ujson 2.0.3 -> 3.0.0 in ALL packages

* Upgrades webserver

* Upgrades packages

* Autoformats web-server
@GitHK
Adds support for GPU scheduling of computational services (#1553)
fb9870f 
* added sidecar_gpu debug profile to launch.json

* added a new API to the director
it is now possible to get service extras
containing additional information which hidden from the user

* sidecar now supports GPU services
- it can now schedule a container requesting GPU resources (VRAM)
- sidecard has 2 start modes CPU and GPU
- for development a GPU sidecar was added

* updating comp_task now adds resoruce requirements
- uses the director API to determine if node requires GPU
- the requires_gpu field is used by the sidecar

* added call to get_service_extras

* fixes some pylint warnings

* this should redirect errors to strout

* disabaled warning

* added some logging to detect why CI fails

* added more debugging for CI

* refactored the way cgroups is parsed

* fixed pylint and message

* removed sidecar_gpu container used for development

* sidecar_gpu removed from the wrong list

* node_extras can be None an it is now accounted for

* correeclty removes the sidecar_gpu now

* sidecar_gpu is not present in all deployments
removing it from both lists as it is only required in development

* added tests for the gpu and non gpu modes
fixed an issue where the process would crash

* fixed pylint

* refactored tests
- names are more readable
- removed some typos

* this is a dictionary

* added comment to rememeber how to run in dev

* updated service description and openapi spec

* moved generate_service_extras

* moved env var resolution to an appropriate place

* added annotations and fixed typos

* added  annotations and less broad exception

* cleanedup implementation

* added more specific exception

* corrected wrong import path

* the sidecar_gpu is now properly started locally
tests where reverted

* fixed pylint caught error

* changing import schema
refactored tests to reflect changes

* removed unused import

* moved sidecar_gpu to docker-compose.devel.yml

* keeping track of these exludes

* replaced with cleaner implementation

* cleanedup and fixed boot modes for sidecar
if will now properly start as expected on all deployments

* fixing import and moved comment

* fixed an issue causing issues starting in dev mode

* api server ninja'd 3006 port moving to 3007

Co-authored-by: Andrei Neagu <[email protected]>
@pcrespov
Maintenance/cleanup api server (#1578)
a329aea 
During the design of this first version, there were many variants of the design that remained in the code. This PR cleans up all the deprecated code before extending it with new features. Some relevant changes:

- Pruned and upgraded requirements
- Removed trials sandbox folder
- Removed unused models, db repos and routes
- Left only basic authentication
- Added client wrapper for webserver sessions (see osparc-simcore/services/api-server/src/simcore_service_api_server/services/webserver.py)
- Reverted logger from loguru to default python logger
- documented dev: services/api-server/README.md
@pcrespov
Cleanup catalog service (#1582)
95df8da 
Cleanup of catalog service before adding more functionality

* Major refactoring
- api/dependencies: dependencies injected in handlers
- api/routes: endpoint handlers and routing
- api/root.py: join all openapi-specs into a single router
- models: domain (orm and business logic models)  and schema (i/o schemas for openapi)  models
- db: database tables (i.e. sa schemas for tables) and repositories (crud layer between handlers and db calls)
- core: init application and settings (parses values by args, environs, .env or default, in this order)
- services: modules/plugins with logic for the app
- __main__,__version__ : main entrypoint and version
@sanderegg
Bugfix/apiserver does not need sslheaders (#1564)
6b02813 
* no need for sslheader in apiserver
* sslheader middleware does not like hyphens
* declare swarm stack name in e2e testing
* wait for services now checks the latest task
* discover name of postgres container
@sanderegg
[feature] UI Fine grained access - project locking and notification
681fc9c 
* write permissions needed to remove a user not delete permission
* added test for opening a shared project 2 times
* access_rights renamed to accessRights
*added test for groups access rights 
* adding state endpoint
* mypy
* now check project is locked
* user is automatically enters room upon successful login
* project state now returns the value + owner of the lock if any
@odeimaiz
Homogenize studies and services (#1569)
bb0788f 
- "Services" tab renamed to "Discover". Under this tab, all the available resources (templates and services) are listed.
- To start a new study:
  - If a templates is selected, a study containing that pipeline will be initialized (same as before)
  - If a service is selected, a study containing that single node will be initialized (new feature). Since this will be a single node study, the service/app will be automatically maximized. Providing a more app-like user experience.

In addition, the content of the Discover tab is also shipped by the frontend compiler with its own customized navigation bar as standalone application that could be served as the Discover/Explore website.

- Plotly removed is not used
- Removed unused images
@odeimaiz
Shared project concurrency (frontend) (#1591)
cb0094b 
- Study cards show lock and tooltip when study is being used somewhere else
- Optimized backend calls for getting Orgs and orgMembers
@pcrespov
Excudes e2e stage from include until fixed (#1595)
6f50e8e
@sanderegg
reduce cardinality of metrics (#1593)
c025875 
* remove node uuid and project uuid from metrics labels to reduce memory footprint
@sanderegg
fix deploy not needing e2e testing since it is disabled
71553ca
@odeimaiz
Feature/fix e2e 2 (#1600)
3adbfa4 
* templates are now in a different tab
* starting a template bug fixed
@odeimaiz
[bugfix] Invalidate cache before starting a study (#1602)
47b24c6 
* Do not allow negative node positions
* Before starting a study, make sure the latest version is fetched
@sanderegg
fix testing if node has gpu support (#1604)
2594bfe 
* fix testing if node has gpu support
@dependabot-preview @pcrespov
Bump docker from 4.2.1 to 4.2.2 in /packages/postgres-database (#1605)
32276c9 
Bumps [docker](https://github.com/docker/docker-py) from 4.2.1 to 4.2.2.
- [Release notes](https://github.com/docker/docker-py/releases)
- [Commits](docker/docker-py@4.2.1...4.2.2)

Signed-off-by: dependabot-preview[bot] <[email protected]>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Pedro Crespo <[email protected]>
@sanderegg
Bugfix/allow reading groups anonymous user (#1615)
153e9b2 
* add groups.read secutiry role
* test that groups right are correctly setup
@sanderegg
Bugfix/concurent opening projects (#1598)
8da6422 
adds usage of redis lock to prevent opening the same study at the same time
@pcrespov
Maintenance/upgrades test tools (#1628)
9ec9d21 
* updates testing libs

* Updated packages test reqs (not base.in reqs)

* Updates web-server test reqs

* Updates storage test reqs

* Upgrades api-server tests reqs

* Upgrades catalog tests reqs

* Updates director test reqs

* Updates sidecar tests reqs

* Moved templates under .vscode with extension
@odeimaiz
filter studies by name before deleting them (#1629)
0f26bc2
@odeimaiz
Improve e2e (#1631)
e42cc0b 
Improve filtering for pre-deletion
@ignapas
Some enhancements and bug fixes (#1608)
18fa7aa 
* Fixed service submission email subject
* Some window styles made uniform.
* Changed some layouts.
* Added standard osparc window component with clickAway closing support.
* Added faded context for modal windows
* Worked arround edit study bug
@pcrespov
Fixes on publish studies handling (#1632)
a58c578 
* Increases pool size in storage and webserver services

* Fixes typo

* Fixes typo in webclient

* renamed variables
pcrespov and others added 18 commits August 17, 2020 17:30
@pcrespov
Maintenance: upgrades requirements of packages and web-server (#1696)
5794a0e 
* Adds info con constraint for isort

* Block upgrade of package and test reqs

* Upgrades deprecated positional argument

Fixes pytest.PytestDeprecationWarning: Passing arguments to pytest.fixture() as positional arguments is deprecated - pass them as a keyword argument instead

* Upgrade to latest pip 20.2.2

* No need for code coverage while testing api specs

* upgrades webserver

* Adds ERD for all pg models

* Adds check to avoid wrong py version

* Fixes reqs after master merge

* Fixes undefined variables

Raised UnboundLocalError: local variable 'log_type' referenced before assignment
@pcrespov
Upgrades api-server dependencies (#1707)
82902f1 
* Upgrades api-server dependencies
Fixes vulnerabilities

* Fixes changes in fastapi API
@sanderegg
services published are to be executable by everyone (#1704)
673d10e 
* templates published are to be executable by everyone
@sanderegg
Split webserver tests (#1711)
af8a29c 
split webserver testing in 4 jobs
@sanderegg
Fix access rights not set correctly for templates (#1712)
3625ef2 
* silence warning for expected file-picker
* fix switching off task
* fixed typo
* add group-nodes to the list of exceptions
@sanderegg
fix access to dictionary
4f0d923
@sanderegg
fix wrong syntax
586f9de
@odeimaiz
Catalog service access (frontend) (#1668)
5b58fb2 
- Service's metadata can be modified by the owner using the frontend
- Service's permissions can be modified by the owner using the frontend
- Template's metadata can be modified
- Bug fix: do not send duplicated services @sanderegg

- Services in a Study can be listed
- Before starting study
  - Get latest version
  - Check if services are accessible
- Refactored ServiceDetails and ServiceStarter
- All the "popUpInWindow" and "createWindow" moved to osparc.Window class
- Copy logs to clipboard
@odeimaiz
Branding oSparc (#1713)
1402dc2 
- Link to Z43-manual
- Brings more visibility to the current . It also adds the "Z43" string if URL contains "speag".
(z43) + platform name in:
  - Tab name
  - Login page
  - Navigation bar
@GitHK
upgrading to ubuntu-20.04 (#1715)
264cfd2 
added ubuntu-20.04 for system-test-swarm-deploy

Co-authored-by: Andrei Neagu <[email protected]>
@GitHK
Computational services can determine on which queue they were schedul…
b759426 
…ed (#1718)

* boot mode was added
a var scheduled mode is injected as env var in the container

* exeption is now logged with tracebeck

* adjusted tests to work as expected

Co-authored-by: Andrei Neagu <[email protected]>
@GitHK
adds more information when checking if node has gpu support (#1725)
e8ebcfc 
* aiodocker is now closed
- prints more info when sarting
- prints the reason why the GPU mode was not found

* remove comment

Co-authored-by: Andrei Neagu <[email protected]>
@pcrespov
Serve group classifiers (#1719)
768dfcd 
Serve group classifiers
@odeimaiz
Serve group classifiers (Frontend) (#1722)
adc6ad8 
* Adds z43:: prefix in statics dev

* adds classifiers table

* Autoformat pg package

* Adds group_classifiers to webserver models facade

* Minor update of db dependencies

* Adds new classifiers entry to API

* Adds groups/{gid}/classfiers entry

* cleanup

* Migration step that adds classifiers table

* Updated ERD view

* Minor doc

* minor refactoring
@sanderegg
Is1717/fine tune default access rights (#1721)
1825744 
any service before 19.08.2020 is visible by everyone
any service after 19.08.2020 will follow "full privacy" rationale (only the owner sees the service)
@GitHK
using unique identifiers for this name (#1727)
184ae08 
Co-authored-by: Andrei Neagu <[email protected]>
@GitHK
Redis cfg fixed for all sidecar deployments (#1730)
334e4c6 
* redis works with all deployments

* codeclimate trigger

* reverting

Co-authored-by: Andrei Neagu <[email protected]>
@sanderegg
owner was not added to rights (#1732)
22c5137
@sanderegg sanderegg self-assigned this Aug 21, 2020
@sanderegg sanderegg added this to the Da Jia milestone Aug 21, 2020
@codecov
Copy link

codecov bot commented Aug 21, 2020 •
edited
Loading

Codecov Report

Merging #1733 into staging will decrease coverage by 1.1%.
The diff coverage is 49.3%.

Impacted file tree graph

@@            Coverage Diff            @@
##           staging   #1733     +/-   ##
=========================================
- Coverage     73.7%   72.5%   -1.2%     
=========================================
  Files          278     279      +1     
  Lines        10918   11268    +350     
  Branches      1179    1238     +59     
=========================================
+ Hits          8048    8174    +126     
- Misses        2527    2757    +230     
+ Partials       343     337      -6
Flag Coverage Δ
#integrationtests 57.1% <32.4%> (+0.4%) ⬆️
#unittests 66.4% <47.2%> (-1.0%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...erver/src/simcore_service_webserver/application.py 94.9% <ø> (-0.2%) ⬇️
...rc/simcore_service_webserver/application_config.py 100.0% <ø> (ø)
...eb/server/src/simcore_service_webserver/catalog.py 77.1% <ø> (-2.5%) ⬇️
...r/src/simcore_service_webserver/computation_api.py 65.6% <ø> (+0.2%) ⬆️
...webserver/computation_comp_tasks_listening_task.py 56.2% <ø> (-7.0%) ⬇️
...simcore_service_webserver/computation_subscribe.py 91.6% <ø> (ø)
.../server/src/simcore_service_webserver/db_models.py 100.0% <ø> (ø)
...imcore_service_webserver/diagnostics_monitoring.py 92.3% <ø> (ø)
...src/simcore_service_webserver/director/__init__.py 84.6% <ø> (-1.1%) ⬇️
...server/src/simcore_service_webserver/groups_api.py 89.0% <ø> (-2.7%) ⬇️
... and 80 more

pcrespov
pcrespov approved these changes Aug 21, 2020
View reviewed changes
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, but please do not squash :-)

@sanderegg sanderegg merged commit f2dae07 into staging Aug 21, 2020
@sanderegg sanderegg deleted the FREEZE_DaJia1 branch August 21, 2020 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@odeimaiz odeimaiz odeimaiz approved these changes

@pcrespov pcrespov pcrespov approved these changes

@mguidon mguidon Awaiting requested review from mguidon

Assignees

@sanderegg sanderegg

Labels
None yet
Projects
None yet
Milestone
Da Jia
Development

Successfully merging this pull request may close these issues.
6 participants
@sanderegg @pcrespov @odeimaiz @GitHK @ignapas @mguidon