This Changelog tracks changes to this project. The notes below include a summary for each release, followed by details which contain one or more of the following tags:
addedfor new features.changedfor functionality and API changes.deprecatedfor soon-to-be removed features.removedfor now removed features.fixedfor any bug fixes.securityin case of vulnerabilities.
changedCIDC_MAILING_LIST to Essex-managed group
changedadmin email
removedunused cloud storage permissioning code (see related CIMAC-CIDC/cidc-ui#489)
addedflag for alembic to compare types for database migrationsaddedmigration to add N/A value for user organization
changedschemas bump for adding serum and allowing dna qc text
addednotes for WES analysis ExACdb assembly compatibility issues
changedschemas bump to remove microbiome reverse index requirement
fixedbug, prevent mutating an UploadJob that has already been merged
fixedbug, moved from join-on to where
fixedbug where WES T/O counting was affect by other trials
fixedpermission error where cross-assay single-trial permissions weren't revoked- null upload_type was being skipped as it received [None] instead of bare None
addedall ctdna files to source faceting in trial view
fixedtrial metadata summaries return correct trial id even when no samples
fixedLimit permission granting based on upload types for manifest granting
removedstale GitHub action for automatic prod releases
fixedfixed permission granting for manifest uploads to include participant and sample files
changedschemas bump to fix samples/participants prefix for file permissioning
changeddon't issue new permissions if user is disabled or not approved
changedpermissions handling to handle each trial instead of each trial / upload_type in revokefixedblob prefix generation error on uploads without files
changedschemas bump to add urine to manifests' type of sample
changedschemas bump to remove ATACseq analysis batch reportremovedfacets for the samefixedmultifile handling, where it was set backwards
changedschemas bump for bug fix, new quality of sample option
addedremoval of assay_uploads and manifest_uploads along with rest of relational db in migration
changedschemas bump for updated ASSAY_TO_FILEPATH constantchangedget GCS blob names to deduplicate across upload_typeschangedpermissions handling to handle each trial instead of each trial / upload_type in grant
changedschemas bump for dateparser version bump
addedspace handling to downloadable filelist.tsv url
removedreferences to permission system in cloud functions for biofxremovedunused is_group option in granting permissions- was only used by the above now-removed permissioning system
changedall analysis files added to analysis ready facet and download
changedschemas bump for WES analysis template folder update
fixedplotly dash error on profile shipments dropdown
changedschemas bump wes new bait set swap
fixedwes_tumor_only_analysis dashboard counting
removedfacets not used in the database or derived from current templatesaddedfacets missing from definitions that ARE used in the database- exclude metadata templates from facets/details
changedschemas bump for derive files returning None instead of error when no derivation is defined
changedschemas bumps for- adding batch to TCR meta csv
- handling of clinical CSV files to strip any initial BOM
- updated tumor-normal attempted pairing CSV
fixedWES TO counts also need to require a report- since tumor-normal pairing manifest schemas change
changedhow trial and network participants and samples are counted for- only count samples and participants that have associated assay data
- does NOT affect assay-level counts for the Data Overview dashboard
- DOES affect counts for Browse Data > Trial Table and Home Page
changedschemas bump for adding TCR meta csv
changedupdated readme
changedonly grant permissions on user reenable if they're approved
changedschemas bump for clinical data bug fix
changedschemas bump for clinical data bug fix
changedschemas bump for MIBI updates
addedN/A organization for users
changedschemas bump for biofx pipeline integration updates
changedfaceting for the previous microbiome and ctDNA analysis files
fixedreference to environment project name reference vs hard coded
fixedbug in building user permissions for files on the portal- allowed users with cross-assay permissions to see clinical data files
addedcredentialing to dataset updatefixedproject id bug
addedprotobuf package version for old generated code
addedbiquery permission delivery/revoking on role assignment
changedstaging uploader role to temp replacement
changedschemas bump for MIBI supportaddedfacets for MIBI files
fixedbug in permissions application, where cross-trial permissions were used instead of cross-assay permissions
addedPACT User role
addedencryption for participant IDs via CSMS API
removedrelational db codechangedCSMS API to rely solely on JSON and schemas
changedschemas bump and migration for docs update / DM clean-up
fixedbug in credentials handling
changedcredentials handling for generating signed urlschangeddefault app run to debug Falseaddeduser input sanitization when getting a specific template
changedbump schemas for microbiome metadata template changes and new shipping lab
changededit H&E faceting to correctly label all new image uploads re allowing jpg files
changedbump schemas for ctdna analysis nulls and hande jpeg fileschangedissuing permissions for upload job actually gets them all now
changedcross-assay permissions to not include clinical data
changedschemas bump for hande manifest req relaxation
changedschemas bump for hande req relaxation
changedschemas bump for clinical data participant count fix
changedschemas bump for new participant alert on manifest upload
changedall WES analysis file purpose to "analysis"
changedschemas bump to fix requirements
changedschemas bump for WES template autogeneration tweakschangedschemas bump for WES analysis cnvkit to copynumber
addedschemas bump, attach bytes to email for WES template autogeneration
addedschemas bump, facets, counting for WES v3addedmigration to move old WES analysisaddedexplicit migration for adding required for _etag, _created, _updated- missing from
0.26.15below
- missing from
addedhandling missing _etag in bulk inserts for testing
addedflask-cachecontrol to prevent caching of /users/data_access_report
addednew wes bait set to relational dbaddedcatch for underlying psycopg2 error being thrown
changedschema version for addition of new wes bait set
changedreverted max download size from 1GiB to 100MB- with instance_class in app.yaml as F2, 512MB memory limit
- testing on staging shows 100MB is likely functional limit for GAE-processed
addedrequirements for _etag, _created, _updated- to prevent future 412 error on account reactivation
changedschemas bump for shipping manifest requirement relaxing
changedcompression for downloaded batches from gztar to zipchangedmax download size from 100MB to 1GiB
changedschemas bump for Microbiome supportaddedfacets, details, analysis counting for Microbiome
changedschemas bump for ctDNA supportaddedfacets, details, analysis counting for ctDNA
changedschemas bump to add ctDNA for assay_type in blood manifestschangedWES counting againaddedclinical facets to the new downloadable_files endpointchangedcopy to deepcopy for facet dict and add test to check all facets directly
changedorder of user and url in single file downloadchangedWES counting to reflect new systemaddednew downloadable_files endpoint to return all facets grouped by assayaddedREADME.md as pypi long_description
removedsecurityHSTS that was brokenaddedmissing facets for RNA MSI and Microbiomechangedadd more try/catch as result of tests; will be integrated on next bump
addedsecurityHSTS (headers) to prod and staging
changedblack, flask, werkzeug, jinja2, schemas version bumpsremovedtyping_extensions dependency
changedschema version for wes-matching update
changedschema version to wes-matching to automate matching
changedschema version to peg regex to prevent errors
changedupdate schema dependency for miF schema preamble changes
changedalert email to new group [email protected]
changednever return empty user email list, ie trials / uploads with no active userschangedfunctions to reflect that iam=False option was never used
fixeddon't return user emails for a trial / upload if that user is disabled
addedfunction to return user emails for given trial_id / upload_typefixedpass session for when trial_id is None when building prefixesaddedgroup handling flag to granting/revoking permissions for BioFX group
changedschemas dependency for backwards compatible WES analysis
removednon-ACL based download permissions systems for productionremovedadmin end point to trigger download permissions cloud function; trigger manually from GCPchangeddisabling inactive users to only return emails for newly disabled
addedcalls to revoke permissions when user is disabled, both manually and for inactivity
fixedpoint to new ACL-controlled bucket in ACL control functions
addedname-based revoking functions for ACL equivalent to the granting oneschangedother existing download permissions functions to use the ACL name-based equivalentsfixednaming conventions that caused issues with cross-repo integrationaddedmore tests around ACL stuff
addedstring-based wrappers for ACL control instead of purely Blobsaddedfunction to return users allowed for a given trial id / upload type
addedlogging in ACL for non-specific KeyError
addedstorage client batching for ACL-based download permission granting/revoking
changed*_all_download_permissions to *_download_permissions, including endpoint addressaddedupload_type kwarg to grant_download_permissionsaddedtrial_id and upload_type kwargs to revoke_download_permissions
fixedpassed session to solve ACL-blocking KeyError
fixedattemptmade grant_all_download_permissions mimic grant_iam_permissionsaddedlogging for with_default_session failures
addedtrial_id kwarg to grant_all_download_permissions
changedschema version bump to add comments to biofx analysis templates
fixedcan't apply expiry condition to upload buckets as they are ACL-controlled
fixedtypo by adding missingand
addedback IAM download functionality for production environment only, partially reverting commit 7504926685dcd00b0c20b41911ec8aba7f8b98b0changeversion definition location fromsetup.pyto__init__.pyto match schemas/cli
changedadmin grant all download permissions to run through cloud function
removedall IAM conditions on data bucket
removedall conditional IAM expressions on data bucket
addedcalls to ACL save, and smoketestsaddedback calls for adding/removing lister permissions, and smoketests
fixedACL syntax again; see https://googleapis.dev/python/storage/latest/acl.html#google.cloud.storage.acl.ACL
fixedACL syntaxaddedfunction to call to add permissions for particular upload jobremovedGOOGLE_DATA_BUCKET entirely from API
adderror logging in Permission.insert
removeall gcloud client logic associated with download logic ie conditional IAM permissionsaddACL gcloud client logic for downloads insteadremoveall lister permission as no longer needed with ACL instead of IAMaddadmin endpoint to call already existing function to grant all download permissions
changedschemas dependency (bump) for WES pipeline updates
changedschemas dependency for WES paired analysis comments field
addeddry_run option for both CSMS insert functions
addedconversion for CSMS value 'pbmc' for processed sample typeaddedhandling in shipments dashboard for no shipment assay_type
fixedcorrectly pass session in more places
addedlogging to see ifinsert_manifest_into_blobis called as expected
fixedbug in iterating offset incsms.auth.get_with_paging
fixedCSMS bug from chainingdetect_manifest_changesandinsert_manifest_...
addedexcluded property to CSMS test data and testsfixedtrying to add CSMS properties to CIDC entries
addedde-identified whole manifest from CSMS directly to test datafixedreference to CIMAC ID in sample creation within models.templates.csms_api.insert_manifest_from_json()fixeddict.items() is unhashable, so use dict.keys() to generate a set to check for _calc_difference()
changedbump schemas dependencies for mIF DM bug fix
addedlogging around error in CSMS testing (deprecated)
changedversion for schemas dependency, for tweak to mIF template
addunstructured JSONB json_data column for shipments, participants, samplesaddcopy of original JSON or CSMS data into json_data columndeprecatednon-critical columns in relational manifests, adding to json_dataaddcorrect exclusion of legacy CSMS manifests
fixedfix mIF excluded samples tabfixedfix typo 'errrors'
addedsubquery for counting ATACseq analysis to get_summaries for Data Overview dashboard
changedbump schemas version for ATACseq analysis updates
fixedset os environ TZ = UTC before datetime is imported every time
fixedcorrectly pass session throughout models/templates/csms_api
removeincorrect accessing of CSMS manifest protocol_identifier which is only stored on the samples
addedfacets and file details for mIF report fileremoveTemplates facet entirely
fixedsecond call to get_with_authorization again
fixedsecond call to get_with_authorization
changedschemas bump for mIF QC report
changedmoved validation of trial's existing in the JSON blobs to better reflect name and usage
fixedpass limit and offset as params instead of kwargs to requests.get
addedhandling to remove old-style permissions
addedlogging to set_iam_policy errors
changedCSMS_BASE_URL and CSMS_TOKEN_URL to be pulled from secrets
fixedchanged prefix generator to correctly handle prefixes without regex support
changedGCP permissions from single conditions to multi-conditions using || and && operatorschangedexpiring permission to be on the general CIDC Lister role instead of every startsWith condition separately
addedfunction for finding CSMS changes and getting updates for relational dbaddedfunction to execute corresponding updates to JSON blob from CSMS changes
addedgrant_lister_access and revoke_lister_access for custom role CIDC Lister that is required for all downloads
addedAPI endpoint to add a new manifest given JSON from CSMS
addedadded TWIST enum values to WES in relational tables
addedschemas bump to add TWIST enum values to WES in JSON
addedmodule export for models.templates
changedschemas bump to add TCRseq controls
changedschemas bump for new TCRseq Adaptive template
Initial set up of tables and definition of needed classes for base metadata and assay uploads. Generated new-style templates and added full testing data for pbmc, tissueslide, h_and_e, wes<fastq/bam>; demo for clinical_data. Implemented JSON -> Relational sync function and wired for testing. Added relational hooks into existing manifest and assay/analysis uploads. Added way to trigger initial synchronization. Allows relational ClinicalTrials to be edited along with TrialMetadatas from the admin panel.
addedJIRA integration (#564)addedchangedStep 1 of Relational DB towards CSMS Integration (#549)addedAdd logging to syncall_from_blobs (#565)addedAdd admin controls for relational Clinical Trials (#567)fixedSome perfecting tweaks (#568)fixedMake sure that new templates are identical to old ones (#569)addedAdd some safety and flexibility to reading (#570)fixedFix header check; add better error handling and tests (#571)addedAdd 5 new optional columns to PBMC manifest for TCRseq (#572)