for release v3.5.0 Dataset Collections #465 #478
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
filter-groups.js : addFilterOrGroup() : filterGroup createRecord() : regularise case of model name filter-Group -> filter-group
manage-explorer.js :
addParentAndScopeLevels() : return [] if value is {}.
datasetFilter() : map2 : change from d3.group() to d3.nest() (as part of d3 upgrade in #370, related d3.nest() changes : b7e2ae2, deca791, f75ba24; earlier : 9e4ced3 changed from d3-array.group() to d3.nest() ). drop comment about using parentAndScope() as a key function for d3 group() - probably metaFilterFG fills that role now. map2 keys are now native values (undefined, true) instead of string representations. n[0] is [key, values] so address them as [0] [1] instead of .key .values
filterGroupsChanged() : add params fgComp, value (not used at this time).
filter-group.hbs : change filterOrGroup {{radio-button }} to <RadioButton >; prefix args with @; use (queue to add (action (mut this.data.filterOrGroup)) (these changes are applicable to the other {{radio-button}}s when they are updated).
part of #465 : #466, #467. manage-explorer.js : dataPre1() : insert filter by .cropFilterSelected. dataPre() : rename to dataPostNameFilter, and use alias to define it, so that filters can be inserted easily (dataPre is used in several places); e.g. dataPostCrop() was drafted to implement the crop filter, which is now implemented in dataPre1(), upstream of the name filter, whereas dataPre is downstream of the name filter, as indicated by the name dataPostNameFilter. add noCrop, cropFilterSelected, selectedCropChanged, cropField, withCrop, child1Crop, cropNames (these are based on respectively noGroup, groupFilterSelected, selectedGroupChanged, withParent, child1, parents, parentNames, substituting parent → crop). add cropsForFilter. dataset.js : add cropName : alias(_meta.Crop). filter-group.js : toString() : reduce trace. manage-explorer.hbs : Use form/select-group, with .cropFilterSelected, .cropsForFilter, .selectedCropChanged
install_and_configure.md : add docker ps --filter ...
(part of #465) select-group.hbs : define attribute multiple when @isMultiple. added preventDefault=false is possibly not required. select-group.js : selectedGroupChangedId() : handle gsP being a value instead of a promise, as is the case for categoriesForFilter(). manage-explorer.js : add noCategory, categoryFilterSelected, selectedCategoryChanged(). dataPre1() : filter by categoryFilterSelected if defined. cropsForFilter(): unshift(noCrop) instead of noGroup, which was copied from groupsInOwnNone(). add categoriesForFilter(). dataset.js : add categories(); this handles both ._meta.Categories and ._meta.Category*, which can be reduced after trial and discussion. manage-explorer.hbs : add select-multiple with params categoryFilterSelected categoriesForFilter selectedCategoryChanged add form/select-multiple.{hbs,js}, and select-multiple-test.js from ember generate; will probably change to use power-select-multiple from ember-power-select.
manage-explorer.js : categoryFilterSelected is now a string instead of an array of strings. selectedCategoryChanged() : add handling of categoryFilterSelected as a string and disable the array handling. dataPre1() : change filtering by categoryFilterSelected, which is now a string instead of an array of strings. categoriesForFilter() : enable unshift(noCategory). manage-explorer.hbs : use select-group in place of select-multiple
closes #463 The motivating example was this Error: VCF dataset dir is not configured This will also apply to other errors. auth.js : _ajax() : include in headsUp.tipText .message and .statusCode from .responseJSON.error from config.error() : XMLHttpRequest and promise.catch() error.
app.scss : #headsUp : wrap text (width : 55%), indent multiple rows from button (display: flex; #headsUp > div : margin-left: 1em) draw-map.hbs : add an X button to clear .headsUp.tipText, and wrap the text in <div> so that multiple rows of text can be indented from the button
pullAndBuild.bash : define backend=lb4app, so it can be switched back easily. define NVM_DIR; nvm deactivate for frontend and nvm use 16 for backend. add getInstanceTagName() (refn : https://stackoverflow.com/a/24549602 itaifrenkel) comment indicates getInstanceTagName() is not tested in cron, perhaps the reason for awsTagName being commented-out. recognise dev server by awsTagName = dev instead of HOSTNAME matching ip. use an inline comment so that ] is part of the if [ expression. add beServerLog() pkill node : match also on the param -r source-map-support, because there are other node processes. don't backupPretzelNohup if ~/pretzel/nohup.out is not present. (file from dev ~/scripts/cron/ cb4fc8e4 3300 Mar 9 2022)
pullAndBuild.bash: use beServerLoopLog (instead of beServer) if no process matches beServerLoop (file from dev ~/scripts/cron/ 940fab52 3410 May 29 2023 )
image_build.bash : add pb_set(), pb_show(). pb_build_feature() : git pull doesn't return a different status to indicate if any new commits were pulled, so instead of || return, use git rev-parse HEAD and define HEAD_unchanged. add pb_tag(), Dc, stage, L, pb_compose_down_up() add pullAndBuild.compose.bash, using image_build pb_ functions and some configuration from pullAndBuild.bash
closes #461. Use bcftools view instead of zcat | cut, -G to leave out genotype information, and use multi-threaded, suggested by DC. vcfGenotypeLookup.Makefile : change : zcat $< | cut -f1-10 | bgzip -c > $@ to : bcftools view --drop-genotypes --threads $(nproc) --output-type z $< --output $@ vcfGenotypeLookup.bash : same change as in .Makefile, with input $vcfGzSamples and output $vcfGzSNPList
…so don't also show that error in the error display above the graph auth.js : add get apiErrorShownInDialog(), and if true, don't put error text in headsUp.tipText.
spreadsheet-read.js : add requiredFieldsMeta, used in sheetToDataset() : push to dataset.warnings any of requiredFieldsMeta which dataset.meta does not define. vcf-genotype.js : checkVCFsAreInstalled() : handle undefined dataset.tags (this is incidental to the commit).
Before this fix, after an error reply to spreadsheet upload, the upload dialog continued to display : Uploading 1 files. (0%) / Processing... / Please wait. Updating database. data-base.js : uploadData() : set isProcessing: false when catch from auth.uploadData() auth.js : get apiErrorShownInDialog() : incorporate the value of userSettings.apiErrorShownInDialog. uploadData() : set userSettings.apiErrorShownInDialog while request is in progress and clear it afterwards via added onFinally().
api-server.js : add datasetsReport(). api-server.hbs : add button .datasetsReport. add datasets-csv.js, with datasetsReport(), clientIdToEmail(). The dataset.clientId.email will be defined if the Groups page is visited before returning to mapview to export the datasets. add file-download.js, with fileDownloadAsCSV(), based on a post on discuss.emberjs.com by skaterdav85, May 2018 : https://discuss.emberjs.com/t/whats-the-best-strategy-for-letting-users-download-ember-data-returns-as-csvs/14767/2, with added URL.revokeObjectURL(url)
spreadsheet-read.js : sheetToObj() : in rowObjects[] row values, recognise array values and parse them, using stringToArray(). add arrayRegexp, stringToArray(). upload.js : datasetSetMeta() : check for id === undefined, and in that case return 404.
manage-explorer.js : showDataSources() : enable dataSources if rootURL is not /
... after seeing displayNames containing commas on the test server datasets-csv.js : datasetsReport() : add needsQuoting() and quoteIfNeeded(), used in .map()
... these are changes that got the cron build and run working on dev. image_build.bash : pb_build() : drop & so that cron build can wait for pb_build before tag and down+up. pullAndBuild.compose.bash : chmod +x because this is executed from cron. define LOG_CRON, L. direct stdouts to log file L, from pb_build_feature, pb_tag, pb_compose_down_up
image_build.bash
enable pretzel_build and Dc to be pre-defined.
pb_build() : docker build ${build_arg_ROOT_URL[@]}
pullAndBuild.compose.bash : source ~/scripts/image_build.config.bash if it exists. This allows for local environment configuration
Main purpose of this commit is to trigger a build. matrix-view.js : updateTable() : no need to trace .table, it can be accessed as window.PretzelFrontend.matrixView.table in devel build
Main purpose of this commit is to trigger a build. paths-table.js : didInsertElement() : record this in .PretzelFrontend.pathsTable for devel. manageHoTable() : to reduce trace, trace .selectedBlock?.id instead of .table, this, as the latter can be acessed from .PretzelFrontend.pathsTable in devel.
image_build.bash : pb_build() : drop sudo from docker build; works OK without it. This enables this to be run from cron, which cannot support sudo. pullAndBuild.compose.bash : add an alternative to pb_build_feature to build when there is no new commit, using pb_set, pb_build
…nable dependency checking between datasets in a single spreadsheet closes #404. stacksAxes.js : featureNameClass() : handle typeof name === number, which is now prevented by the change to sheetToObj() in this commit, which ensures .name is a string. dataset.js : spreadsheetUploadInternalDatabase() : datasetRemovedPs : separate multiple calls to loadAfterDelete() by a progressive timeout, to enable dependency checking between dataset worksheets in a spreadsheet, e.g. QTL worksheet depends on GM via .parentName and flankingMarkers. spreadsheet-read.js : sheetToObj() : rowObjects : for .name field values with typeof number, convert the number to a string. This handles GMs with marker numbers (old); further notes in added comment. upload.js : uploadDatasetContent() and insert_features_recursive() : add trace, for checking timing of datasets insert
upload.js : datasetSetMeta() : move calculation of clientIdString, clientId out of rows[] loop, which reduces trace.
sequence-search.js : add develRefnSetup() : note this in .PretzelFrontend.sequenceSearch. drop inputIsActive(), actions.dnaSequenceInput(). factor formatText() out of paste() and drop actions.paste(). add setText(), and in clear() and fromSelectedFeatures() : use setText() in place of this.set(text, ), and setting text$ .val() (e.g. via text2Area()) is not required now that @value={{this.text}}. drop text2Area(). searchButtonDisabled() : use .text instead of .inputsOK. sequence-search.hbs : textarea : use @value={{this.text}} to set the value, and drop all actions which updated the text value : oninput -> inputIsActive, enter / newline / escape -> dnaSequenceInput
mapview loadBlock() was changed to a toggle in 137c366, but there are 4 calls which aim to view block, not toggle it, so pass param view=true to loadBlock() in : natural-search.js : nameChr2Dataset() manage-explorer.js : actions.loadBlock() manage-genotype.js : blockViewAndBrush() select-parent.js : actions.loadBlock() manage-genotype.js : blockBrushDomain() : in later() check for ! block.axis1d || block.axis1d.isDestroying entry-block-add-button.js : actions.loadBlock() : if block is toggling to un-viewed, skip setting visible / colour for Trait / Ontology. mapview.js : loadBlock() : add optional param viewOpt, if it is provided, block.isViewed is set to that.
There is a display update issue in this commit; details follow. goto-feature-list.js : add clearResults() : set selected.features and featureSearchResult() to [], same for .blocksOfFeatures and aliases, using .set() to achieve display update. added and used .showResult instead to hide the Received text when cleared. add emptyArray(), emptyObject(); not effective for display update. goto-feature-list.hbs : add button .clearResults. Wrap Received <ul> block with #if this.showResult.
table-brushed.js : add downloadCSVFile(). table-brushed.hbs : add button .downloadCSVFile file-download.js : add exportAsCSVFile(); this is factored out of utils/data/datasets-csv.js : datasetsReport(); it is probably not worth using this function there because that does some additional value-mapping which is not supported here : .join(_) for Categories and tags, clientIdToEmail, which could be incorporated into quoteIfNeeded.
upload.js : datasetSetMeta() : factor to form removeId(), accept either 'Current dataset name' or 'id' for row datasetId.
{,frontend/}package.json : version : 3.4.0 -> 3.5.0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Work Completed
Principal addition
development branch feature/datasetCollections
Sub-issues of #465, and other issues completed in this branch :
group datasets by meta.Crop #466
Display crop names as tabs in the Dataset Explorer #467
Add Category filter for Dataset Collections #471
Improve performance of VCF Genotype Request processing #461
Warn if dataset upload does not define meta.Crop #472
Generate a CSV report of datasets and export as a file download #473
Provide GUI error message when Genotype directory is not set up #463
Streamline the upload of Crop and Category metadata #474
Feature Search : add a clear button to enable remove the triangle after search #417
Feature / marker names which are numbers are being converted to numeric type instead of string #404
Change white-space filtering in text input fields for DNA Sequence search and Feature Search #397
Genotype request : filter out duplicate sample names entered by the user #409 - re-tested duplicate samples and SNPs - OK, closed with a comment with commit and test description. Found an issue in 460 :
Add a minus sign to remove the dataset in Dataset Explorer #460 - beacd7c changes required by loadBlock changing to a toggle
Prepare metadata for software release #477