Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FREEZE_DaJia1 #1733

Merged
merged 80 commits into from
Aug 21, 2020
Merged
Changes from 1 commit
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
03b1525
add traefik endpoint to api-gateway (#1555)
sanderegg Jun 15, 2020
680eb59
added simcore_webserver_service in pytest simcore package (#1563)
sanderegg Jun 17, 2020
821c8e6
Is1269/api-server upgrade (#1475)
pcrespov Jun 17, 2020
16c30b9
Is/add notebook migration script (#1565)
sanderegg Jun 19, 2020
b6a2568
Manage groups, Share studies (#1512)
odeimaiz Jun 19, 2020
3fb36d0
maintenance fix codecov reports (#1568)
sanderegg Jun 22, 2020
bdfcd08
Bump faker from 4.1.0 to 4.1.1 in /packages/postgres-database (#1573)
dependabot-preview[bot] Jun 22, 2020
1b7a47e
Is1570/study fails 500 (#1572)
pcrespov Jun 22, 2020
294611d
Maintenance/upgrades and tooling (#1546)
pcrespov Jun 22, 2020
fb9870f
Adds support for GPU scheduling of computational services (#1553)
GitHK Jun 23, 2020
a329aea
Maintenance/cleanup api server (#1578)
pcrespov Jun 24, 2020
95df8da
Cleanup catalog service (#1582)
pcrespov Jun 26, 2020
6b02813
Bugfix/apiserver does not need sslheaders (#1564)
sanderegg Jun 30, 2020
681fc9c
[feature] UI Fine grained access - project locking and notification
sanderegg Jun 30, 2020
bb0788f
Homogenize studies and services (#1569)
odeimaiz Jul 2, 2020
cb0094b
Shared project concurrency (frontend) (#1591)
odeimaiz Jul 2, 2020
6f50e8e
Excudes e2e stage from include until fixed (#1595)
pcrespov Jul 2, 2020
c025875
reduce cardinality of metrics (#1593)
sanderegg Jul 3, 2020
71553ca
fix deploy not needing e2e testing since it is disabled
sanderegg Jul 3, 2020
3adbfa4
Feature/fix e2e 2 (#1600)
odeimaiz Jul 3, 2020
47b24c6
[bugfix] Invalidate cache before starting a study (#1602)
odeimaiz Jul 3, 2020
2594bfe
fix testing if node has gpu support (#1604)
sanderegg Jul 7, 2020
32276c9
Bump docker from 4.2.1 to 4.2.2 in /packages/postgres-database (#1605)
dependabot-preview[bot] Jul 7, 2020
153e9b2
Bugfix/allow reading groups anonymous user (#1615)
sanderegg Jul 8, 2020
8da6422
Bugfix/concurent opening projects (#1598)
sanderegg Jul 8, 2020
9ec9d21
Maintenance/upgrades test tools (#1628)
pcrespov Jul 13, 2020
0f26bc2
filter studies by name before deleting them (#1629)
odeimaiz Jul 13, 2020
e42cc0b
Improve e2e (#1631)
odeimaiz Jul 14, 2020
18fa7aa
Some enhancements and bug fixes (#1608)
ignapas Jul 14, 2020
a58c578
Fixes on publish studies handling (#1632)
pcrespov Jul 14, 2020
45add3b
Is1585/cleanup storage (#1586)
pcrespov Jul 16, 2020
7c17e52
Bump lodash from 4.17.15 to 4.17.19 (#1639)
dependabot[bot] Jul 17, 2020
df488dd
Fixes default (#1640)
pcrespov Jul 17, 2020
91d9482
2 bugs fixed + Some improvements (#1634)
odeimaiz Jul 17, 2020
f801fdd
is1594 fix and re-activate e2e testing (#1620)
pcrespov Jul 20, 2020
4159b89
Platform stability: (#1645)
odeimaiz Jul 20, 2020
78631d2
Theming (#1656)
ignapas Jul 29, 2020
a12817d
Bump aiozipkin from 0.6.0 to 0.7.0 in /packages/service-library (#1642)
dependabot-preview[bot] Aug 3, 2020
f0e11ca
Improving storage performance (#1659)
GitHK Aug 3, 2020
eba4c8b
Bump pytest-docker from 0.7.2 to 0.8.0 in /packages/service-library (…
dependabot-preview[bot] Aug 3, 2020
d0791e7
Bump ujson from 3.0.0 to 3.1.0 in /packages/service-library (#1664)
dependabot-preview[bot] Aug 4, 2020
00c4f65
Bump yarl from 1.4.2 to 1.5.1 in /packages/postgres-database (#1665)
dependabot-preview[bot] Aug 4, 2020
34919ab
UI/UX improvements (#1657)
odeimaiz Aug 4, 2020
d65a3b9
Bump pytest-docker from 0.7.2 to 0.8.0 in /packages/postgres-database…
dependabot-preview[bot] Aug 6, 2020
37e31ef
Bump coverage from 5.2 to 5.2.1 in /packages/postgres-database (#1653)
dependabot-preview[bot] Aug 6, 2020
294b6ac
Simple Sweeper (#1641)
odeimaiz Aug 7, 2020
259e790
typo in git refs/heads (#1679) (#1680)
sanderegg Aug 10, 2020
b524b5e
pip missing dependencies (#1687)
sanderegg Aug 10, 2020
b434742
Bump coverage from 5.2 to 5.2.1 in /packages/service-library (#1650)
dependabot-preview[bot] Aug 11, 2020
deed000
adds pull retry to help with failing registry (#1688)
GitHK Aug 11, 2020
e23b47d
e2e: JupyterNB + JupyterLab (#1689)
odeimaiz Aug 11, 2020
abfced4
Making services more debuggable (#1690)
GitHK Aug 11, 2020
43f358c
Adds MPI scheduling support (#1673)
GitHK Aug 12, 2020
8ff27cb
Fixes some error with the services (#1692)
GitHK Aug 12, 2020
0e0f7f4
Mounts and envs (#1674)
mguidon Aug 13, 2020
95772b7
Classifiers (#1670)
odeimaiz Aug 13, 2020
2b72867
Extend jupyters e2e (#1693)
odeimaiz Aug 13, 2020
f4f2cbc
Is/catalog service access (backend) (#1589)
sanderegg Aug 17, 2020
11b0f74
FIX: catching exceptions does not work outside asyncio layer (#1694)
GitHK Aug 17, 2020
39447f8
Garbage Collector removes manually marked GUEST users (#1683)
GitHK Aug 17, 2020
5794a0e
Maintenance: upgrades requirements of packages and web-server (#1696)
pcrespov Aug 17, 2020
82902f1
Upgrades api-server dependencies (#1707)
pcrespov Aug 18, 2020
673d10e
services published are to be executable by everyone (#1704)
sanderegg Aug 18, 2020
af8a29c
Split webserver tests (#1711)
sanderegg Aug 18, 2020
3625ef2
Fix access rights not set correctly for templates (#1712)
sanderegg Aug 18, 2020
4f0d923
fix access to dictionary
sanderegg Aug 18, 2020
586f9de
fix wrong syntax
sanderegg Aug 19, 2020
5b58fb2
Catalog service access (frontend) (#1668)
odeimaiz Aug 19, 2020
1402dc2
Branding oSparc (#1713)
odeimaiz Aug 19, 2020
264cfd2
upgrading to ubuntu-20.04 (#1715)
GitHK Aug 19, 2020
b759426
Computational services can determine on which queue they were schedul…
GitHK Aug 20, 2020
e8ebcfc
adds more information when checking if node has gpu support (#1725)
GitHK Aug 20, 2020
768dfcd
Serve group classifiers (#1719)
pcrespov Aug 20, 2020
adc6ad8
Serve group classifiers (Frontend) (#1722)
odeimaiz Aug 20, 2020
1825744
Is1717/fine tune default access rights (#1721)
sanderegg Aug 21, 2020
184ae08
using unique identifiers for this name (#1727)
GitHK Aug 21, 2020
334e4c6
Redis cfg fixed for all sidecar deployments (#1730)
GitHK Aug 21, 2020
22c5137
owner was not added to rights (#1732)
sanderegg Aug 21, 2020
0207fdb
Merge branch 'staging' into FREEZE_DaJia1
sanderegg Aug 21, 2020
fb49601
fix bad merge
sanderegg Aug 21, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Is1269/api-server upgrade (#1475)
- new api-server refactored
- api-server connected to simcore-stack
pcrespov authored Jun 17, 2020

Unverified

No user is associated with the committer email.
commit 821c8e61a5d23e4480781db669b19217c8ad220f
1 change: 1 addition & 0 deletions .codeclimate.yml
Original file line number Diff line number Diff line change
@@ -78,3 +78,4 @@ exclude_patterns:
- "**/migrations/"
- "**/*.js"
- "**/pytest-simcore/"
- "**/sandbox/"
1 change: 1 addition & 0 deletions .env-devel
Original file line number Diff line number Diff line change
@@ -47,6 +47,7 @@ TRACING_ZIPKIN_ENDPOINT=http://jaeger:9411

TRAEFIK_SIMCORE_ZONE=internal_simcore_stack

WEBSERVER_HOST=webserver
WEBSERVER_LOGIN_REGISTRATION_CONFIRMATION_REQUIRED=0
WEBSERVER_LOGIN_REGISTRATION_INVITATION_REQUIRED=0
# python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key())"
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@ Makefile @pcrespov, @sanderegg
/scripts/demo @odeimaiz, @pcrespov
/scripts/json-schema-to-openapi-schema @sanderegg
/scripts/template-projects @odeimaiz, @pcrespov
/services/api-gateway @pcrespov
/services/api-server @pcrespov
/services/catalog @pcrespov
/services/sidecar @pcrespov, @mguidon
/services/web/client @odeimaiz, @oetiker, @ignapas
18 changes: 9 additions & 9 deletions .github/workflows/ci-testing-deploy.yml
Original file line number Diff line number Diff line change
@@ -56,8 +56,8 @@ jobs:
- name: test
run: ./ci/github/unit-testing/api.bash test

unit-test-api-gateway:
name: Unit-testing api-gateway
unit-test-api-server:
name: Unit-testing api-server
runs-on: ${{ matrix.os }}
strategy:
matrix:
@@ -86,22 +86,22 @@ jobs:
restore-keys: |
${{ runner.os }}-pip-
- name: install
run: ./ci/github/unit-testing/api-gateway.bash install
run: ./ci/github/unit-testing/api-server.bash install
- name: test
run: ./ci/github/unit-testing/api-gateway.bash test
run: ./ci/github/unit-testing/api-server.bash test
- uses: codecov/codecov-action@v1
with:
flags: unittests #optional
- name: prepare codeclimate coverage file
run: |
curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter
chmod +x ./cc-test-reporter
./cc-test-reporter format-coverage -t coverage.py -o codeclimate.unit_api_gateway_coverage.json coverage.xml
./cc-test-reporter format-coverage -t coverage.py -o codeclimate.unit_api_server_coverage.json coverage.xml
- name: upload codeclimate coverage
uses: actions/upload-artifact@v1
with:
name: unit_api_gateway_coverage
path: codeclimate.unit_api_gateway_coverage.json
name: unit_api_server_coverage
path: codeclimate.unit_api_server_coverage.json

unit-test-catalog:
name: Unit-testing catalog
@@ -838,7 +838,7 @@ jobs:
- uses: actions/checkout@v2
- uses: actions/download-artifact@v1
with:
name: unit_api_gateway_coverage
name: unit_api_server_coverage
- uses: actions/download-artifact@v1
with:
name: unit_catalog_coverage
@@ -873,7 +873,7 @@ jobs:
run: |
mkdir all_coverages
cp \
unit_api_gateway_coverage/*.json \
unit_api_server_coverage/*.json \
unit_catalog_coverage/*.json \
unit_director_coverage/*.json \
unit_sidecar_coverage/*.json \
2 changes: 1 addition & 1 deletion .pylintrc
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code
extension-pkg-whitelist=
extension-pkg-whitelist=pydantic

# Add files or directories to the blacklist. They should be base names, not
# paths.
16 changes: 8 additions & 8 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -107,9 +107,9 @@ jobs:
after_failure:
- unbuffer bash ci/travis/unit-testing/webserver.bash after_failure

# test python, api-gateway ----------------------------------------------------------------------
# test python, api-server ----------------------------------------------------------------------
- stage: build / unit-testing
name: api-gateway
name: api-server
language: python
python:
- "3.6"
@@ -118,17 +118,17 @@ jobs:
- docker
cache: pip
before_install:
- sudo bash ci/travis/unit-testing/api-gateway.bash before_install
- sudo bash ci/travis/unit-testing/api-server.bash before_install
install:
- unbuffer bash ci/travis/unit-testing/api-gateway.bash install
- unbuffer bash ci/travis/unit-testing/api-server.bash install
before_script:
- unbuffer bash ci/travis/unit-testing/api-gateway.bash before_script
- unbuffer bash ci/travis/unit-testing/api-server.bash before_script
script:
- unbuffer bash ci/travis/unit-testing/api-gateway.bash script
- unbuffer bash ci/travis/unit-testing/api-server.bash script
after_success:
- unbuffer bash ci/travis/unit-testing/api-gateway.bash after_success
- unbuffer bash ci/travis/unit-testing/api-server.bash after_success
after_failure:
- unbuffer bash ci/travis/unit-testing/api-gateway.bash after_failure
- unbuffer bash ci/travis/unit-testing/api-server.bash after_failure

# test python, catalog ----------------------------------------------------------------------
- stage: build / unit-testing
17 changes: 10 additions & 7 deletions Makefile
Original file line number Diff line number Diff line change
@@ -29,7 +29,7 @@ $(if $(IS_WIN),$(error Windows is not supported in all recipes. Use WSL instead.
# TODO: read from docker-compose file instead $(shell find $(CURDIR)/services -type f -name 'Dockerfile')
# or $(notdir $(subst /Dockerfile,,$(wildcard services/*/Dockerfile))) ...
SERVICES_LIST := \
api-gateway \
api-server \
catalog \
director \
sidecar \
@@ -46,10 +46,12 @@ export VCS_STATUS_CLIENT:= $(if $(shell git status -s),'modified/untracked','cle
export BUILD_DATE := $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")

# api-versions
export CATALOG_API_VERSION := $(shell cat $(CURDIR)/services/catalog/VERSION)
export DIRECTOR_API_VERSION := $(shell cat $(CURDIR)/services/director/VERSION)
export STORAGE_API_VERSION := $(shell cat $(CURDIR)/services/storage/VERSION)
export WEBSERVER_API_VERSION:= $(shell cat $(CURDIR)/services/web/server/VERSION)
export API_SERVER_API_VERSION := $(shell cat $(CURDIR)/services/api-server/VERSION)
export CATALOG_API_VERSION := $(shell cat $(CURDIR)/services/catalog/VERSION)
export DIRECTOR_API_VERSION := $(shell cat $(CURDIR)/services/director/VERSION)
export STORAGE_API_VERSION := $(shell cat $(CURDIR)/services/storage/VERSION)
export WEBSERVER_API_VERSION := $(shell cat $(CURDIR)/services/web/server/VERSION)


# swarm stacks
export SWARM_STACK_NAME ?= simcore
@@ -189,7 +191,7 @@ endif


up-devel: .stack-simcore-development.yml .init-swarm $(CLIENT_WEB_OUTPUT) ## Deploys local development stack, qx-compile+watch and ops stack (pass 'make ops_disabled=1 up-...' to disable)
# Start compile+watch front-end container [front-end]
# Start compile+watch front-end container [front-end]
$(MAKE_C) services/web/client down compile-dev flags=--watch
# Deploy stack $(SWARM_STACK_NAME) [back-end]
@docker stack deploy -c $< $(SWARM_STACK_NAME)
@@ -303,7 +305,8 @@ pylint: ## Runs python linter framework's wide
/bin/bash -c "pylint --jobs=0 --rcfile=.pylintrc $(strip $(shell find services packages -iname '*.py' \
-not -path "*egg*" \
-not -path "*migration*" \
-not -path "*contrib*" \
-not -path "*datcore.py" \
-not -path "*sandbox*" \
-not -path "*-sdk/python*" \
-not -path "*generated_code*" \
-not -path "*datcore.py" \
Original file line number Diff line number Diff line change
@@ -5,14 +5,14 @@ IFS=$'\n\t'

install() {
bash ci/helpers/ensure_python_pip.bash
pushd services/api-gateway; pip3 install -r requirements/ci.txt; popd
pushd services/api-server; pip3 install -r requirements/ci.txt; popd
pip list --verbose
}

test() {
pytest --cov=simcore_service_api_gateway --durations=10 --cov-append \
pytest --cov=simcore_service_api_server --durations=10 --cov-append \
--color=yes --cov-report=term-missing --cov-report=xml \
-v -m "not travis" services/api-gateway/tests/unit
-v -m "not travis" services/api-server/tests/unit
}

# Check if the function exists (bash specific)
3 changes: 2 additions & 1 deletion ci/helpers/install_pylint.bash
Original file line number Diff line number Diff line change
@@ -14,7 +14,8 @@ pip3 install "$PYLINT_VERSION"
# Minimal packages to pass linter
pip install \
celery\
docker
docker\
pyjwt


echo "INFO:" "$(pylint --version)" "@" "$(command -v pylint)"
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@
set -euo pipefail
IFS=$'\n\t'

FOLDER_CHECKS=(api/ api-gateway packages/ .travis.yml)
FOLDER_CHECKS=(api/ api-server packages/ .travis.yml)

before_install() {
if bash ci/travis/helpers/test-for-changes.bash "${FOLDER_CHECKS[@]}";
@@ -18,7 +18,7 @@ install() {
if bash ci/travis/helpers/test-for-changes.bash "${FOLDER_CHECKS[@]}";
then
bash ci/helpers/ensure_python_pip.bash
pushd services/api-gateway; pip3 install -r requirements/ci.txt; popd
pushd services/api-server; pip3 install -r requirements/ci.txt; popd
fi
}

@@ -32,11 +32,11 @@ before_script() {
script() {
if bash ci/travis/helpers/test-for-changes.bash "${FOLDER_CHECKS[@]}";
then
pytest --cov=simcore_service_api_gateway --durations=10 --cov-append \
pytest --cov=simcore_service_api_server --durations=10 --cov-append \
--color=yes --cov-report=term-missing --cov-report=xml \
-v -m "not travis" services/api-gateway/tests/unit
-v -m "not travis" services/api-server/tests/unit
else
echo "No changes detected. Skipping unit-testing of api-gateway."
echo "No changes detected. Skipping unit-testing of api-server."
fi
}

Original file line number Diff line number Diff line change
@@ -11,6 +11,7 @@
from copy import deepcopy
from logging.config import fileConfig
from pathlib import Path
from typing import Dict

import alembic.command
import click
@@ -116,7 +117,7 @@ def main():
@click.option("--host")
@click.option("--port", type=int)
@click.option("--database", "-d")
def discover(**cli_inputs):
def discover(**cli_inputs) -> Dict:
""" Discovers databases and caches configs in ~/.simcore_postgres_database.json (except if --no-cache)"""
# NOTE: Do not add defaults to user, password so we get a chance to ping urls
# TODO: if multiple candidates online, then query user to select
@@ -173,7 +174,7 @@ def _test_swarm():
fg="green",
)

return
return cfg

except Exception as err:
inline_msg = str(err).replace("\n", ". ")
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
""" API keys to access public gateway
""" API keys to access public API


These keys grant the client authorization to the API resources
12 changes: 5 additions & 7 deletions packages/postgres-database/tests/conftest.py
Original file line number Diff line number Diff line change
@@ -3,11 +3,12 @@
# pylint:disable=unused-argument
# pylint:disable=redefined-outer-name

import pytest
import yaml
from typing import Callable, Coroutine, Union

import sqlalchemy as sa
import aiopg.sa
import pytest
import sqlalchemy as sa
import yaml


@pytest.fixture(scope="session")
@@ -32,11 +33,8 @@ def postgres_service(docker_services, docker_ip, docker_compose_file) -> str:
return dsn


from typing import Union, Coroutine, Callable


@pytest.fixture
def make_engine(postgres_service):
def make_engine(postgres_service: str) -> Callable:
dsn = postgres_service

def maker(is_async=True) -> Union[Coroutine, Callable]:
52 changes: 52 additions & 0 deletions packages/postgres-database/tests/test_delete_projects_and_users.py
Original file line number Diff line number Diff line change
@@ -85,6 +85,58 @@ async def start():
return loop.run_until_complete(start())


@pytest.mark.skip(reason="sandbox for dev purposes")
async def test_insert_user(engine):
async with engine.acquire() as conn:

# execute + scalar
res: ResultProxy = await conn.execute(
users.insert().values(**random_user(name="FOO"))
)
assert res.returns_rows
assert res.rowcount == 1
assert res.keys() == ("id",)

user_id = await res.scalar()
assert isinstance(user_id, int)
assert user_id > 0

# only scalar
user2_id: int = await conn.scalar(
users.insert().values(**random_user(name="BAR"))
)
assert isinstance(user2_id, int)
assert user2_id == user_id + 1

# query result
res: ResultProxy = await conn.execute(
users.select().where(users.c.id == user2_id)
)
assert res.returns_rows
assert res.rowcount == 1
assert len(res.keys()) > 1

# DIFFERENT betwen .first() and fetchone()

user2: RowProxy = await res.first()
# Fetch the first row and then close the result set unconditionally.
assert res.closed

res: ResultProxy = await conn.execute(
users.select().where(users.c.id == user2_id)
)
user2a: RowProxy = await res.fetchone()
# If rows are present, the cursor remains open after this is called.
assert not res.closed
assert user2 == user2a

user2b: RowProxy = await res.fetchone()
# If no more rows, the cursor is automatically closed and None is returned
assert user2b is None
assert res.closed



async def test_count_users(engine):
async with engine.acquire() as conn:
users_count = await conn.scalar(users.count())
3 changes: 2 additions & 1 deletion packages/postgres-database/tests/test_groups.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# pylint: disable=E1120
# pylint: disable=no-name-in-module
# pylint: disable=no-value-for-parameter

import faker
import pytest
Original file line number Diff line number Diff line change
@@ -48,5 +48,5 @@ async def wait_till_traefik_responsive(api_endpoint: URL):
assert "service" in proxied_service
if "webserver" in proxied_service["service"]:
assert proxied_service["status"] == "enabled"
elif "api-gateway" in proxied_service["service"]:
elif "api-server" in proxied_service["service"]:
assert proxied_service["status"] == "enabled"
4 changes: 3 additions & 1 deletion scripts/common.Makefile
Original file line number Diff line number Diff line change
@@ -92,7 +92,9 @@ info: ## displays basic info


.PHONY: autoformat
autoformat: ## runs black python formatter on this service's code
autoformat: ## runs black python formatter on this service's code. Use AFTER make install-*
# sort imports
@python3 -m isort --atomic -rc $(CURDIR)
# auto formatting with black
@python3 -m black --verbose \
--exclude "/(\.eggs|\.git|\.hg|\.mypy_cache|\.nox|\.tox|\.venv|\.svn|_build|buck-out|build|dist|migration|client-sdk|generated_code)/" \
Loading