Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: curation API differentiates unpublished collections for revisions #3912

Merged

Conversation

ebezzi
Copy link
Member

@ebezzi ebezzi commented Jan 11, 2023

For an unpublished collection that is not a revision, Curation API should return the collection_url pointing to the permalink instead of the private version. This is a requirement for the curation workflow.

Bento007 and others added 2 commits January 10, 2023 16:34
- add code coverage for python unit tests
- run coverage for every PR and upload report to Codecov
- add make recipes to generate coverage report
- add allure test report for unit tests
- update coverage version
@codecov
Copy link

codecov bot commented Jan 11, 2023

Codecov Report

Merging #3912 (0eadad1) into staging (b4e5fb7) will decrease coverage by 22.47%.
The diff coverage is 59.02%.

❗ Current head 0eadad1 differs from pull request most recent head a7262c1. Consider uploading reports for the commit a7262c1 to get more accurate results

@@             Coverage Diff              @@
##           staging    #3912       +/-   ##
============================================
- Coverage    87.36%   64.88%   -22.48%     
============================================
  Files          141       48       -93     
  Lines         9138     3417     -5721     
============================================
- Hits          7983     2217     -5766     
- Misses        1155     1200       +45     
Flag Coverage Δ
backend ?
python ?
unitTest ?
unittests 64.88% <59.02%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
backend/common/utils/semvar_utils.py 87.50% <ø> (ø)
backend/common/utils/type_conversion_utils.py 46.83% <ø> (ø)
...ckend/portal/pipeline/processing/h5ad_data_file.py 89.36% <ø> (ø)
backend/common/utils/secret_config.py 51.28% <12.50%> (ø)
backend/layers/persistence/persistence.py 18.88% <18.88%> (ø)
backend/layers/thirdparty/crossref_provider.py 24.46% <24.46%> (ø)
backend/common/corpora_config.py 40.00% <25.00%> (ø)
backend/common/utils/aws.py 31.42% <33.33%> (ø)
backend/common/corpora_orm.py 67.84% <39.13%> (ø)
backend/layers/processing/downloader.py 44.92% <44.92%> (ø)
... and 152 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@ebezzi ebezzi merged commit bf8908a into staging Jan 11, 2023
@ebezzi ebezzi deleted the ebezzi/curation-api-unpublished-collection-collection-url branch January 11, 2023 16:38
nayib-jose-gloria added a commit that referenced this pull request Jan 23, 2023
* feat(MUIv5): Upgrade SDS and MUI (#3734)

* feat(MUIv5): Upgrade SDS and MUI

* update ButtonIcon

* fix Filter

* upgrade sds

* wip fix highlight

* replace Blueprint HTMLTable

* fix QuickSelect style

* remove lint ignore

* fix QuickSelect style

* fix: Fix the re-tagging issue due to invalid registryId (#3781)

Fixes the following error in Docker re-tag:

```
[ERROR]: error getting Image: operation error ECR: BatchGetImage, https response error StatusCode: 400, RequestID: f00322e8-7a33-4e28-a07f-93f2bdaa25bb, InvalidParameterException: Invalid parameter at 'registryId' failed to satisfy constraint: 'must satisfy regular expression [0-9]{12}'
```

* feat: portal redesign (#3485)

* prep

* more stuff

* missing line

* changes

* bunch of changes

* interface and tests

* Update backend/layers/common/entities.py

Co-authored-by: Andrew Tolopko <[email protected]>

* temp stuff

* refactor, add documentation

* more refactor

* temp

* typo

* Update backend/layers/persistence/persistence.py

Co-authored-by: Andrew Tolopko <[email protected]>

* more tests

* Update backend/layers/persistence/persistence.py

Co-authored-by: Andrew Tolopko <[email protected]>

* Update backend/layers/persistence/persistence.py

Co-authored-by: Andrew Tolopko <[email protected]>

* more tests

* refactor

* Update backend/layers/persistence/persistence.py

Co-authored-by: Andrew Tolopko <[email protected]>

* more tests

* even more stuff

* Changes

* PR comments

* PR suggestions, as code changes

* changes

* Comments

* Remove owner from collection metadata

* rename

* more work

* Remove all authorization from the business logic layer

* Assertions for metadata validation errorS

* More and more stuff

* convert everything to id classes

* First pass

* more files

* more stuff

* TestCreateCollection ✅

* TestGetCollectionVersion ✅

* TestGetAllCollections ✅

* Add one more test

* chore: business layer tests for update collection (#3524)

* chore: business layer tests for update collection

* Update tests/unit/backend/layers/business/test_business.py

Co-authored-by: Daniel Hegeman <[email protected]>

* remove old test

Co-authored-by: Daniel Hegeman <[email protected]>

* signature

* Overhaul dataset processing status

* missing files

* fix enum mixin

* comments

* few fixes

* TestUpdateCollectionDatasets ✅

* TestGetDataset ✅

* TestUpdateDataset ✅

* TestCollectionOperations ✅

* Link validation

* Move business interface to another file

* initial dump

* missing files

* more tests

* stuff

* missing files

* more stuff

* chore: add DatasetMetadata missing fields (#3572)

* changes

* Add name, x_approximate_distribution

* baby steps

* test__get_collection__ok ✅

* many more tests ✅

* small assertion

* Link names can be optional

* stuff

* fix bug + enforce immutability in several places

* many more tests ✅

* moving to datasets

* chore: deepcopy in persistence_mock (#3578)

* deepcopy in persistence_mock

* Added comment

* Add DatasetArtifactId

* more tests

* start

* functions

* baby steps

* ingest_dataset now also returns canonical dataset_id

* test__get_all_datasets_for_index_with_ontology_expansion ✅

* missing files

* test__get_dataset_assets ✅

* rework base class

* revision tests - first pass

* stuff

* TestRevision ✅

* TestDeleteRevision ✅

* Fix a bunch of pending tests

* chore: add dataset_version -> collection_id link

* temp

* chore: add methods to get versions from a canonical collection_id

* one more test

* upload tests

* more tests ✅

* upload link tests ✅

* Publish collection ✅

* chore: add dataset_version -> collection_id link (#3585)

* chore: collection versions can only be created one at a time (#3582)

* chore: collection versions can only be created one at a time

* add a case for non existing collections

* chore: add methods to get versions from a canonical collection_id (#3586)

* last minute changes

* initial stuff

* more stuff

* several advancements

* stuff

* more stuff

* fix: Migration task fails to start (#3588)

* Fix commits

* stuff

* stuff

* temp change

* unit tests attempt

* more unit tests

* more unit tests

* happy path

* s3 provider

* more implementations

* stuff

* Changes

* add mock

* stuff

* typo

* typo

* meta endpoint

* conflict

* remove outdated stuff

* Remove old test

* one more test

* fixes

* stuff

* tmp remove curator_name

* Add field

* one fix

* more stuff

* overhaul

* Rename class

* rename published_at -> originally_published_at

* feat: explicit canonical collection (#3622)

* stuff

* tmp remove curator_name

* Add field

* one fix

* more stuff

* overhaul

* Rename class

* rename published_at -> originally_published_at

* timestamps

* timestamps 2

* feat: persistence layer

Signed-off-by: nayib-jose-gloria <[email protected]>

* feat: add persistence implementation (first draft)

Signed-off-by: nayib-jose-gloria <[email protected]>

* feat: add finalization of canonical dataset on collection version publish

Signed-off-by: nayib-jose-gloria <[email protected]>

* feat: redesign canonical dataset (#3627)

* feat: canonical dataset

* published_at in API

* fix tests

* partial changes

* add set-up to create schema from persistence orm

Signed-off-by: nayib-jose-gloria <[email protected]>

* fixes to the ORM

* fix method call

* fix: use DatabaseProvider for test_business and update persistence layer accordingly

Signed-off-by: nayib-jose-gloria <[email protected]>

* stuff

* stuff

* fix TestCollectionOperations tests

Signed-off-by: nayib-jose-gloria <[email protected]>

* persistence layer changes (#3650)

* checkpoint

* more stuff

* one more test

* stuff and more stuff

* final

* tests

* changes

* pr feedback

Signed-off-by: nayib-jose-gloria <[email protected]>

* Various bugs&fixes

* fixes

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix tests and pr feedback

Signed-off-by: nayib-jose-gloria <[email protected]>

* feat: processing layer (#3603)

* Fixes for API tests

* get_collection_index

* Missing files

* bunch of changes

* properly set revision_of

* failure

* failure layer

* more merge

* feat: migration script + some fixes (#3681)

* first pass

* second pass

* stuff

* stuff

* more stuff

* bunch of changes

* script

* parametrize db name

* docs

* more debug stuff

* remove debug stuff

* black and flake8

Signed-off-by: Trent Smith <[email protected]>

* fix: publisher metadata correct type (#3699)

* fix: publisher metadata correct type

* fix: publisher metadata correct type

* typo

* fix: allows business tests to be run by both persistence mock and as an integration test with postgres docker container

Signed-off-by: nayib-jose-gloria <[email protected]>

* lint

Signed-off-by: nayib-jose-gloria <[email protected]>

* feat: revise at for curation api (#3675)

Add revised_at and validation message to backend.layers.common.entity

- change UserInfo.user_id to a property. Simplifies accessing the variable.
- use pydantic.datalclass. This gives us type checking for free.
- Moved errors in BusinessException to be an instance variable. Making it is class variable will have unintended consequences.
- Fixed pointer in persistence_mock.py
- Populating revised_at in persistence_mock.py

* fix merge conflict

Signed-off-by: Trent Smith <[email protected]>

* fix: get portal API tests working with persistence mock

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix: get portal API tests working with database provider implementation

Signed-off-by: nayib-jose-gloria <[email protected]>

* refactor: simplify session management + include option to run tests as unit tests or integration tests

Signed-off-by: nayib-jose-gloria <[email protected]>

* pr feedback

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix: processing tests (#3715)

* fix: strip -> removesuffix (#3716)

* chore: redesign run tests in GHA (#3694)

* chore: redesign run tests in GHA

* pip

* switch to docker

* remove import

* all tests

* Lint

* overhaul test commands

* fix workflow

* feat: add canonical collection tombstoning to portal redesign (#3740)

* linting

Signed-off-by: nayib-jose-gloria <[email protected]>

* feat: support canonical collection tombstoning

Signed-off-by: nayib-jose-gloria <[email protected]>

* linting + pr feedback

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>

* feat: set validation message (#3735)

* feat: set validation message

* Lint

* chore: redesign miscellaneous fixes (#3739)

* stuff

* lint

* fix test

* add published_at to ignored fields for revision diff

* add crossref provider

* PR comments & lint

* move explorer_url

* missing files

* chore: restore processing tests (#3736)

* not quite there yet

* stuff

* missing files

* fix

* lint

* chore: restore cloudfront invalidation (#3761)

* chore: restore cloudfront invalidation

* rename

* rename file

* missing files

* chore: curator name (#3780)

* feat: translate curation api for redesign (#3690)

* fix typo

* resolve conflicts

Signed-off-by: Trent Smith <[email protected]>

* curation api WIP

Signed-off-by: Trent Smith <[email protected]>

* curation_api

Signed-off-by: Trent Smith <[email protected]>

* curation_api

Signed-off-by: Trent Smith <[email protected]>

* consolidate tests

Signed-off-by: Trent Smith <[email protected]>

* resolve DOIs

Signed-off-by: Trent Smith <[email protected]>

* Fix s3_upload_credentials.py

Signed-off-by: Trent Smith <[email protected]>

* Fix some get collection id tests

Signed-off-by: Trent Smith <[email protected]>

* Fix post Revisions

Fix some exiting tests after merging from target branch.

Signed-off-by: Trent Smith <[email protected]>

* Fix Get Collections public test

Signed-off-by: Trent Smith <[email protected]>

* Fix get collections tests

Signed-off-by: Trent Smith <[email protected]>

* Fix get collection tests

Signed-off-by: Trent Smith <[email protected]>

* Fix get collection tests

Signed-off-by: Trent Smith <[email protected]>

* feat: curation API for datasets (#3708)

* init

* stuff

* baby steps

* new stuff

* end

* more stuff

* Remove ApiCommon

* Update backend/portal/api/curation/v1/curation/collections/collection_id/datasets/dataset_id/actions.py

Co-authored-by: Trent Smith <[email protected]>

* Update backend/portal/api/curation/v1/curation/collections/collection_id/datasets/dataset_id/actions.py

Co-authored-by: Trent Smith <[email protected]>

* PR comments

Co-authored-by: Trent Smith <[email protected]>

* fixing patch tests

Signed-off-by: Trent Smith <[email protected]>

* consolidating tests

Signed-off-by: Trent Smith <[email protected]>

* fixing get portal api index

Signed-off-by: Trent Smith <[email protected]>

* Update tests/unit/backend/layers/business/test_business.py

* Update backend/layers/business/business.py

* fix typos

Signed-off-by: Trent Smith <[email protected]>

Signed-off-by: Trent Smith <[email protected]>
Co-authored-by: Emanuele Bezzi <[email protected]>

* chore: curator API fix tests (#3782)

* fix typo

* resolve conflicts

Signed-off-by: Trent Smith <[email protected]>

* curation api WIP

Signed-off-by: Trent Smith <[email protected]>

* curation_api

Signed-off-by: Trent Smith <[email protected]>

* curation_api

Signed-off-by: Trent Smith <[email protected]>

* consolidate tests

Signed-off-by: Trent Smith <[email protected]>

* resolve DOIs

Signed-off-by: Trent Smith <[email protected]>

* Fix s3_upload_credentials.py

Signed-off-by: Trent Smith <[email protected]>

* Fix some get collection id tests

Signed-off-by: Trent Smith <[email protected]>

* Fix post Revisions

Fix some exiting tests after merging from target branch.

Signed-off-by: Trent Smith <[email protected]>

* Fix Get Collections public test

Signed-off-by: Trent Smith <[email protected]>

* Fix get collections tests

Signed-off-by: Trent Smith <[email protected]>

* Fix get collection tests

Signed-off-by: Trent Smith <[email protected]>

* Fix get collection tests

Signed-off-by: Trent Smith <[email protected]>

* feat: curation API for datasets (#3708)

* init

* stuff

* baby steps

* new stuff

* end

* more stuff

* Remove ApiCommon

* Update backend/portal/api/curation/v1/curation/collections/collection_id/datasets/dataset_id/actions.py

Co-authored-by: Trent Smith <[email protected]>

* Update backend/portal/api/curation/v1/curation/collections/collection_id/datasets/dataset_id/actions.py

Co-authored-by: Trent Smith <[email protected]>

* PR comments

Co-authored-by: Trent Smith <[email protected]>

* fixing patch tests

Signed-off-by: Trent Smith <[email protected]>

* consolidating tests

Signed-off-by: Trent Smith <[email protected]>

* fixing get portal api index

Signed-off-by: Trent Smith <[email protected]>

* Update tests/unit/backend/layers/business/test_business.py

* Update backend/layers/business/business.py

* fix typos

Signed-off-by: Trent Smith <[email protected]>

* init

* rollback

* checkpoint

* checkpoint

* checkpoint

* checkpoint

* more tests

* Asset tests

* more test fixes

* more test fixes

* comment

* Linter

* fix

* linter

* fixes

* improve tests

* tests

* remove monkeypatch

* Update backend/portal/api/curation/v1/curation/collections/actions.py

Co-authored-by: Trent Smith <[email protected]>

Signed-off-by: Trent Smith <[email protected]>
Co-authored-by: Trent Smith <[email protected]>
Co-authored-by: Trent Smith <[email protected]>

* chore: add curator_name to migration script (#3791)

* chore: redesign last minute fixes (#3792)

* fixes

* more fixes

* lint

* feat: add integration test mode as a gha test step + makefile command (#3788)

* feat: add integration test mode as a gha test step + makefile command

Signed-off-by: nayib-jose-gloria <[email protected]>

* pin dataclasses-json version in requirements + reintroduce step to rebuild backend container on changed files/dockerfiles

Signed-off-by: nayib-jose-gloria <[email protected]>

* add pydantic to processing requirements, rebuild containers with no cache

Signed-off-by: nayib-jose-gloria <[email protected]>

* remove --no-cache flag from rebuild commands

Signed-off-by: nayib-jose-gloria <[email protected]>

* remove pydantic to pass tests

Signed-off-by: nayib-jose-gloria <[email protected]>

* re-add dataclasses import from std lib

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>
Co-authored-by: Emanuele Bezzi <[email protected]>

* chore: last minute fixes #2 (#3794)

* update ontology test

Signed-off-by: nayib-jose-gloria <[email protected]>
Signed-off-by: Trent Smith <[email protected]>
Co-authored-by: Andrew Tolopko <[email protected]>
Co-authored-by: Andrew Tolopko <[email protected]>
Co-authored-by: Daniel Hegeman <[email protected]>
Co-authored-by: Alex Lokshin <[email protected]>
Co-authored-by: nayib-jose-gloria <[email protected]>
Co-authored-by: Trent Smith <[email protected]>
Co-authored-by: Nayib Gloria <[email protected]>
Co-authored-by: Trent Smith <[email protected]>

* fix: missing 'e' in explorer_url + dataset id in meta endpoint (#3795)

* fix: missing 'e' in explorer_url

* more

* feat: dynamic sitemap and robots.txt file (#3753)

Co-authored-by: Timmy Huang <[email protected]>

* fix: version_id in meta endpoint (#3809)

* fix: version_id in meta endpoint

* fix tests

* fix: support for collection partial updates (#3812)

* fix: support for collection partial updates

* Lint

* fix e2e after MUIv5 upgrade (#3813)

Co-authored-by: atarashansky <[email protected]>

* fix: y-axis scrolling jump on heatmap download (#3691)

* fix: y-axis scrolling jump on heatmap download

* restoring scrollTop position after create image and before download

* feat: adding tint screen on download load

* fix: functional tests portal redesign compatibility (#3815)

* fix: failing functional tests related to portal redesign

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix: failing functional tests related to portal redesign

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix: lint

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix: grab id from DatasetId

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix: rollback container-functionaltest change

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>

* test(wmg): Fix Source Data (#3816)

* test(wmg): Fix Data Source part 2 (#3817)

* fix: only add original_ids on unpublished revisions of published datasets (#3820)

* fix: only add original_ids on unpublished revisions of published datasets

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix: fix and add portal API tests + update test method return type hints

Signed-off-by: nayib-jose-gloria <[email protected]>

* lint

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix: bad CrossrefProvider imports + reintroduce auth0 retry (#3819)

* fix: bad CrossrefProvider imports

* fix tests

* lint

* last minute fix

* fix: for portal API dataset status and delete endpoints, fetch latest collection version (#3822)

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>

* chore: fix test snapshot (#3818)

Co-authored-by: atarashansky <[email protected]>
Co-authored-by: Daniel Hegeman <[email protected]>

* chore: support INITIALIZED dataset status (#3824)

* chore: support INITIALIZED dataset status

* try to fix tests

* feat: PATCH collection curator API redesign (#3814)

* init

* more tests

* end

* PR comments

* fix import

* fix: remove original_id and fix post-deploy tests (#3826)

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix: test_api_key_crud functional test (#3828)

* fix: remove original_id and fix post-deploy tests

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix: test_api_key_crud functional test

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>

* chore: redesign dataset submissions lambda (#3827)

* init

* more

* fix tests

* lint

* mock config

* Bunch of fixes

* more changes

* Missing dependencies

* fix: redesign submissions lambda fix 1 (#3832)

* fix: redesign submissions lambda fix 1

* lint

* fix: redesign submissions lambda fix 2 (#3833)

* fix: redesign submissions lambda fix 2

* missing providers

* chore: double lookup for GET dataset in curation API (#3834)

* chore: double lookup for GET dataset in curation API

* linter

* add comment

* fix: send slack alerts on failing post-deploy tests (#3821)

* fix: send slack alerts on failing post-deploy tests

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix: use workflow status GHA action step to check overall workflow status at the end of workflow

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix workflow overall failure check

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>

* feat: marker genes unavailable for cell types <25 (#3825)

* feat: marker genes unavailable for cell types <25

Co-authored-by: atarashansky <[email protected]>

* chore: reintroduce wmg unit tests + migrate to redesign (#3837)

* chore: reintroduce wmg unit tests to a separate action
* step 1: fix get_dataset_s3_uris
* add layers to dockerfile
* 2nd endpoint
* remove raw h5ad

* chore: add filter for local.h5ad to wmg extract function (#3843)

* chore: reintroduce wmg unit tests to a separate action

* rename

* step 1: fix get_dataset_s3_uris

* add layers to dockerfile

* dep

* Fix

* checkpoint

* test #1

* 2nd endpoint

* lint

* remove raw h5ad

* fix

* fix: add local.h5ad trailing

* Linter

* fix(fmg): fix ordering and remove p-value (#3831)

* fix: mainting top-n ordering for marker genes

* accept new response in frontend

* remove p-value from chart

* add back p-value to api yml

* fmt + lint

* update tests

* p-value back in example

* fix remaining tests

* blacked and fixed unit tests

* blacked unit tests

* fixed formatting

Co-authored-by: atarashansky <[email protected]>
Co-authored-by: Seve Badajoz <[email protected]>
Co-authored-by: atarashansky <[email protected]>
Co-authored-by: atarashansky <[email protected]>

* feat: Exclude Blood from FMG (#3839)

* feat: Exclude Blood from FMG

* change effect size to marker score (#3855)

Co-authored-by: atarashansky <[email protected]>

* chore(deps): bump json5 from 1.0.1 to 1.0.2 in /frontend (#3842)

Bumps [json5](https://github.com/json5/json5) from 1.0.1 to 1.0.2.
- [Release notes](https://github.com/json5/json5/releases)
- [Changelog](https://github.com/json5/json5/blob/main/CHANGELOG.md)
- [Commits](json5/json5@v1.0.1...v1.0.2)

---
updated-dependencies:
- dependency-name: json5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Timmy Huang <[email protected]>

* docs: add discover api data model img for cell census schema docs reference (#3844)

* chore: RAW_H5AD type applied in migration + processing (#3862)

* chore: RAW_H5AD type applied in migration + processing

* Lint

Co-authored-by: Nayib Gloria <[email protected]>

* feat(WMG+FMG): improve GE layout, rebuild y axis chart, and fix inconsistent gene spacing (#3811)

* redo GE layout and rebuild y axis chart

* remove vestigial variables

* update

* checkpoint

* update

Co-authored-by: atarashansky <[email protected]>
Co-authored-by: atarashansky <[email protected]>

* update tests (#3868)

Co-authored-by: atarashansky <[email protected]>

* fix(portal redesign): handle raw h5ads at api layer in portal and curation API (#3873)

* fix(portal redesign): handle raw h5ads at api layer in portal and curation API

Signed-off-by: nayib-jose-gloria <[email protected]>

* remove unused import

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix(wmg): don't send requests if only genes or tissues have been removed (#3859)

* fix(wmg): don't send requests if only genes or tissues have been removed

* lint

Co-authored-by: Seve Badajoz <[email protected]>

* feat(fmg): shift marker genes to front if already selected when adding (#3874)

* branch init

* feat(fmg): shift marker genes to front if already selected when adding

Co-authored-by: Seve Badajoz <[email protected]>

* fix: downloaded image being cut off (#3869)

* fix(wmg): y axis is now properly sticky (#3877)

* fix y axis stickiness when scrolling horizontally

* add comment

* remove inline container

Co-authored-by: atarashansky <[email protected]>

* fix(wmg): image download (#3880)

* chore(wmg): rewrite x axis using react (#3879)

* rewrite x axis as react

* update

* update

* prettier

* remove dup prop

Co-authored-by: atarashansky <[email protected]>

* fix(WMG): remove step 2 when genes are added and heatmap is loading (#3763)

* branch init

* remove isLoading check for gene step removal

Co-authored-by: atarashansky <[email protected]>

* fix(wmg): Select Tissues/Genes dropdown not triggering onClick event (#3881)

* chore(fmg): fmg documentation (#3870)

* fmg documentation

* update deps

* update deps

Co-authored-by: atarashansky <[email protected]>

* feat: FMG tooltip doc link (#3852)

* feat: FMG tooltip doc link

* adding doc link

* Update routes.ts

Co-authored-by: atarashansky <[email protected]>

* fix(wmg): echarts init (#3884)

* change link (#3887)

Co-authored-by: atarashansky <[email protected]>

* fix e2e after x axis rewrite (#3888)

Co-authored-by: atarashansky <[email protected]>

* chore: only return dataset_deployments on CXG completion (#3851)

* chore: only return dataset_deployments on CXG completion

* fix test + lint

* empty list instead of None

* chore: translates the portal API module from class-based to function-based (#3878)

* init

* chore: migrate portal API to be function based instead of class based

* linter

* rename router.py -> providers.py

* Add missing files

* chore: update alembic migration to use new database (#3841)

- Upgrade SQLAlchemy for the processing container so it matches the other containers
- Upgrade Alembic for local development.
- factor out "persistance_schema" into a constant to avoid naming issues.
- Update alembic to support both the new and legacy ORM for now. The legacy ORM can be removed in a future migration.
- Add an Alembic migration for the redesign database changes. This is will not affect the legacy database.
- Modify `migrate_redesign_write` so it no longer create the new database tables. This should be handled by the automated migration processes in GHA.
- Fixe the legacy ORM to match the actual database.
- Update `create_db` to use the new ORM.
- updated sqlalchemy in the processing container to be current with other containers.
- Add Make recipe `db/check` to make it easier to detect if the ORM is different from the database. 
- Add a try..except block in `DatabaseProvider._drop_schema` so prevent breaking if the schema does not exist.
- Add support for creating both the legacy and new database.
- removing CORPORA_LOCAL_DEV from setup_dev_data.sh.

* fix(wmg): still check for equality on non-checked filters when checking to clobber queries (#3898)

Co-authored-by: Seve Badajoz <[email protected]>

* update (#3886)

Co-authored-by: atarashansky <[email protected]>

* update (#3901)

Co-authored-by: atarashansky <[email protected]>

* chore: update marker gene unit test  (#3902)

* update test

* empty

Co-authored-by: atarashansky <[email protected]>

* update (#3885)

Co-authored-by: atarashansky <[email protected]>

* refactor: persistence layer clean-up (#3835)

* refactor: persistence layer clean-up

Signed-off-by: nayib-jose-gloria <[email protected]>

* lint

Signed-off-by: nayib-jose-gloria <[email protected]>

* update missing id changes

Signed-off-by: nayib-jose-gloria <[email protected]>

* lint fix

Signed-off-by: nayib-jose-gloria <[email protected]>

* more missing id's

Signed-off-by: nayib-jose-gloria <[email protected]>

* remove unused import

Signed-off-by: nayib-jose-gloria <[email protected]>

* correct id name

Signed-off-by: nayib-jose-gloria <[email protected]>

* lint fix

Signed-off-by: nayib-jose-gloria <[email protected]>

* id fix

Signed-off-by: nayib-jose-gloria <[email protected]>

* id fix

Signed-off-by: nayib-jose-gloria <[email protected]>

* more ids

Signed-off-by: nayib-jose-gloria <[email protected]>

* id fix

Signed-off-by: nayib-jose-gloria <[email protected]>

* version id fix

Signed-off-by: nayib-jose-gloria <[email protected]>

* lint fix

Signed-off-by: nayib-jose-gloria <[email protected]>

* autogenerate migration script for db field changes + update README links

Signed-off-by: nayib-jose-gloria <[email protected]>

* update makefile test command documentation

Signed-off-by: nayib-jose-gloria <[email protected]>

* pr feedback on docs

Signed-off-by: nayib-jose-gloria <[email protected]>

* update migration script + db/local/load-schema + migration docs

Signed-off-by: nayib-jose-gloria <[email protected]>

* lint fixes

Signed-off-by: nayib-jose-gloria <[email protected]>

* remove unused imports

Signed-off-by: nayib-jose-gloria <[email protected]>

* tabs

Signed-off-by: nayib-jose-gloria <[email protected]>

* tabs

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>

* feat: unpublished collections can be looked up by canonical id (#3907)

* feat: unpublished collections can be looked up by canonical collection_id

* complete docstring

* change

* valid -> invalid

* fix: Documentation Tooltip on Icon Hover Only (#3893)

* fix: icon doc hover only

* using question mark SVG for tooltip icon hover

* tooltip icon hover colors

* fixing linting error "Type 'Element' is not assignable to type"

* linting unused var

* linting

* importing svg directly and changing brightness

Co-authored-by: atarashansky <[email protected]>

* fix(coverage): adding code coverage (#3894)

- add code coverage for python unit tests
- run coverage for every PR and upload report to Codecov
- add make recipes to generate coverage report
- add allure test report for unit tests
- update coverage version

* fix: curation API differentiates unpublished collections for revisions (#3912)

* Remove feature branch from push_test (#3915)

* chore(redesign): Clean up processing container files (#3911)

* Clean up old processing container files
- adding json logs to processing container
- clean up old processing container files

* Update backend/layers/processing/process.py
* remove old upload_failure, upload_success, and submission lambda code

* feat(fmg): add analytics (#3896)

* branch init

* branch init

* feat(fmg): add analytics

Co-authored-by: Seve Badajoz <[email protected]>

* feat(fmg): default to ttest (#3919)

* branch init

* feat(fmg): default to ttest

Co-authored-by: Seve Badajoz <[email protected]>

* fix processing logs (#3922)

* feat(fmg): remove feature flag (#3943)

* branch init

* feat(fmg): remove feature flag

Co-authored-by: Seve Badajoz <[email protected]>

* feat: newly unpublished collections return canonical collection_id in API (#3944)

* feat: newly published collections return canonical collection_id in the API

* refactor to function

* lint

* add test for canonical id

Co-authored-by: Trent Smith <[email protected]>

* feat: newly unpublished collections return canonical collection_id in API (#3944)

* feat: newly published collections return canonical collection_id in the API

* refactor to function

* lint

* add test for canonical id

Co-authored-by: Trent Smith <[email protected]>
(cherry picked from commit ec2f61f)

* chore: release main->staging (#4019)

* fix: FMG icon not hiding for < 25 cells (#3946)

* feat(curation API): newly unpublished collections return canonical collection_id in API (#3948)

* feat: data submitters want collections to document their consortia (#3875)

feat: data submitters want collections to document their consortia

feat: add default empty consortia to create collection

fix: fix texts for consortia on portal-api

fix: update default for consortia to empty list

feat: Add optional Consortia dropdown in New and Edit Collection with the required enumeration of approved consortia (#3642)

feat: update edit collection form consortia dropdown with latest SDS dropdown component and style adjustments.

fix: Linting and minor style updates.

feat: added tests for updating consortia

feat: validate consortia

feat: add consortia to curation/discover API

refactor: mock valid consortia

fix: also override valid_consortia in business layer tests for #348

refactor: add explicit exception type for invalid collection metadata for #348

fix: fix format errors for #348

fix: fix lint errors for #348

fix: make consortia required on portal API collection response for #348

* fix: disable portal on consortia dropdown to maintain focus within create collection dialog.

* refactor: break apart large tests

* feat: added new discovery api test

Co-authored-by: Fran McDade <[email protected]>

* chore: cxg links are permalinks for new unpublished collections (#3957)

* chore: cxg links are permalinks for new unpublished collections

* curation API

* rename

* lint

* chore: use canonical collection_id in curation API responses (#3964)

* Fix curation post dataset for unpublished collections (#3966)

* fix: support null metadata and double lookup for write datasets (#3968)

* fix: support null metadata and double lookup for write datasets

* PR changes

* split test into 2

* fix: improve dataset identifiers (#3967)

* fix: improve dataset identifiers

* Linter

* Test declutter

* new test+overhaul

* refactor: Refactor consortia type (#3973) (#3974)

Co-authored-by: Fran McDade <[email protected]>

* fix: local dev database config / secret name (#3985)

Recent commit 4e6c4e2 simplified the '../database_local' secret name
to '../database'. However, local dev case exceptions still caused the
database config to look for a '../database_local' secret.

* fix: handle null metadata in curation API GET collection (take 2) (#3986)

* fix: handle null metadata case (take 2)

* Linter

* Update scripts/setup_dev_data.sh

* fix: Portal Get /collections (#3980)

* Fix Portal Get /collections

- We need to return the canonical collection_id for a new unpublished collections when list all the collections. This technically isn't user facing problem but it would mean we no longer have working functional test if we don't fix this way. Since we do not openly disclose the version ID we wont be able to determine if a collection was successfully listed by get collection since creating a collection only return the canonical collection ID.

* test: update the revision functional test to use the right explorer_url (#3993)

* test: update the revision functional test to use the correct explorer_url

* lint

Co-authored-by: Trent Smith <[email protected]>

* fix: add double lookup in dataset submissions lambda (#3994)

* feat: adding search and details to dataset filter (#3905)

* chore: POST /datasets returns the canonical dataset_id (#3999)

* fix(curation api): GET /collections/{collection_id}/datasets/{dataset_id} (#3995)

* fix(curation api): GET /collections/{collection_id}/datasets/{dataset_id}
- added tests for revision_of in GET /collections/{collection_id}/datasets/{dataset_id}
- revision_of only refers to the published dataset when revising a dataset. Otherwise it is NULL.
- fix typing in test code

* return the canonical dataset id when POSTing a dataset.

* fix: check whether revision's consortia list has changed when determining whether collection has updates (#4003)

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>

* chore: GET collections/:collection_id should return the correct dataset_id (#4006)

* fix: add retries to dataset pipeline for failure and success handlers. (#4004)

* fix: add retries to dataset pipeline for failure and success handlers.

* fix typo

* feat: cronjob to remove old rdev stacks (#4008)

* fix: verion of happy-cleanup

* fix: Adapt RevisionStatusTag to use redesigned data model (#4009)

* fix: Adapt RevisionStatusTag to use redesigned data model

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix 'updated' logic

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>

* fix: stack name parsing bug

This version update fixes a JSON parsing bug in the Github Action

* docs: add valid consortia to swagger apis (#3997) (#4010)

* fix: sort consortia on collection create or edit (#3996) (#4015)

* fix: sort consortia on collection create or edit (#3996)

* fix: update collection handles DOI correctly (#4017)

* fix: check for Revision Changes accounts for deleted datasets in new data model + refactor Collection checks (#4016)

* fix: check for Revision Changes accounts for deleted datasets in new data model + refactor Collection checks

Signed-off-by: nayib-jose-gloria <[email protected]>

* pr feedback

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>
Co-authored-by: ashin-czi <[email protected]>
Co-authored-by: David Rogers <[email protected]>
Co-authored-by: Fran McDade <[email protected]>
Co-authored-by: Emanuele Bezzi <[email protected]>
Co-authored-by: Fran McDade <[email protected]>
Co-authored-by: Daniel Hegeman <[email protected]>
Co-authored-by: Nayib Gloria <[email protected]>
Co-authored-by: Jake Heath <[email protected]>

Signed-off-by: nayib-jose-gloria <[email protected]>
Signed-off-by: Trent Smith <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Timmy Huang <[email protected]>
Co-authored-by: Alex Lokshin <[email protected]>
Co-authored-by: Emanuele Bezzi <[email protected]>
Co-authored-by: Andrew Tolopko <[email protected]>
Co-authored-by: Andrew Tolopko <[email protected]>
Co-authored-by: Daniel Hegeman <[email protected]>
Co-authored-by: Trent Smith <[email protected]>
Co-authored-by: Trent Smith <[email protected]>
Co-authored-by: SethFeingold <[email protected]>
Co-authored-by: Timmy Huang <[email protected]>
Co-authored-by: atarashansky <[email protected]>
Co-authored-by: atarashansky <[email protected]>
Co-authored-by: ashin-czi <[email protected]>
Co-authored-by: atarashansky <[email protected]>
Co-authored-by: Severiano Badajoz <[email protected]>
Co-authored-by: Seve Badajoz <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Rogers <[email protected]>
Co-authored-by: Fran McDade <[email protected]>
Co-authored-by: Fran McDade <[email protected]>
Co-authored-by: Jake Heath <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants