Upgrade frontend frameworks and libraries

[Don-Isdale]

Task description

The application is built on top of multiple frameworks, which make it possible to provide complex GUI features over the web; these are open-source and continually upgrading and it is advisable to keep up with them, to remain current with changes in web browsers and operating systems.

branch : feature/upgradeFrontend

---

Sub-tasks

• start (investigating) frontend upgrade (Ember4 / Bootstrap4 / ...)

• establish a compatible set of package versions; successful build

• swap-in replacement for (sunsetted) ember-simple-auth mixins. configure router locationType

• prefix attributes with this. in .hbs
  add options async, inverse in belongsTo,hasMany().
  beforeModel() : session.setup()
  d3.schemeCategory20 -> 10
  Change {{#link-to }} to angle brackets , add @ and names to params
  auth-input.hbs : use @autocomplete

• Logged in. The datasets and blocks are requested, received, pushed to store.

• select a fork of ember-parachute which has replaced tryInvoke(),

• • use @voll fork of ember-parachute (replaces use of tryInvoke() ).

• details in commit message : start of frontend upgrade to Ember4 and Bootstrap4. aebbd64

• upgrade 2 forked dependencies :
  ember-cli-multi-store-service : fbccbca replace use of global Ember. update dependency ember-cli-babel version
  ember-split-view-modifier : b5a3da7 update Modifier Manager Capabilities

• displayed : left-panel nav tabs

• use this. in templates

• • fixes to auto-modified this. in templates
• • Apply ember-no-implicit-this-codemod. 8aab55d

• pass param options {async, inverse} to ember data belongsTo() and hasMany().

• change d3.{keys,values()} to Object.{keys,values()}

• install handsontable via npm (was bower)

• found an updated fork of d3-tip (cgav) and installed via npm

• import handsontable; route transition options;

• • relationship options, d3 keys,values,nest, capitalize, etc. b7e2ae2, cec261b

• switch off fade on BSTab-s

• support URL sub-directory in Reset Password

• • trim // in endpoint URL. f7a50c5

• GM histogram and brushed features displayed

• bs tab fade=false. link-to route. etc 8b4583c Thu Feb 29 16:21

• resolve popper.js, by including ember-bootstrap

• brushed features shown in features table, paths table, yellow circles when table hover.

• in progress : replacing sendAction() with closure actions and direct function calls. change uses of d3.event and d3.mouse. 3cac309

• explorer dataset search

• genotype table displayed, formatted and functioning OK.

• loading of received genotype features.

• Zoom button,

Continuing working through the application displays and actions :

• [4-8H] genotype table filters
• [4-8H] QTLs / Traits / Ontologies
• [4-8H] view adjustments

---

2024Mar21 status :

I've worked through the group pages in the last few days, and they are now functioning.
In these pages there are a number of updates (add group, change name, add/delete member, change visibility, dataset group, ...) and most of these required changes.
There are only a few db updates in the remainder of the app : dataset meta, feature values edit (Ontology)
and any changes required there will probably be similar to the solutions I have worked out in the groups.

The plan is to continuing working through the application displays and actions :

• genotype table filters
• QTLs / Traits / Ontologies
• view adjustments

The update process is a mix of :

• reading deprecation docs and searching for where those changes apply to the application source
• applying relevant code-mods
• running the application. This is the quickest way to discover most issues, as most produce an useful error or an exception which gives a fair indication of the area to look at and the nature of the issue.

In terms of estimation : we can model the application (~100k lines) of having a probability of a change being required of about .01% per line, and the time to determine and make the change has a broad distribution : most changes are simple, and there are automatic edits (code modifiers) which will make the change (mostly without error, but advisable to check the diff after the edit). Other changes will require investigation; some have the character of the challenges inherent in using libraries and frameworks : they amplify your efforts and provide a lot of functionality for small effort, but the effort of locating a functional mis-match is also amplified because of the large blocks of code/functionality. For example in the group updates, I took 1/2 day to find a workable solution to one issue, then found a better solution in a similar time, then from that I understood the problem more deeply and have implemented a more minimal solution. Most of the other changes have been easier than that. Importantly, I don't expect to hit a show-stopper which I can't solve in reasonable time - I've already worked through many or most of the main changes in Ember & D3, and I don't expect any really challenging problems to appear at this stage.
Very small changes in the change rate and intensity distribution have a large effect on the project time; after reading the change documentation we can make rough guesses at the change rate and the average change time, but the effect of this is compounded by changes presenting randomly and having a random impact, like a Markov chain.

The above indicates the challenges of estimation, and a more time-effective and workable estimate is simply a "gut feel" - the initial estimate I gave of 3 weeks was just based on experience of previous updates, combined with the types of changes I knew were in this update. That is without seeing the number or type of the actual changes, so it should be taken as a broad estimate.

---

(commits in time order)

b650e2d result of npx ember-cli-update --to 3.28
67f6969 npm install updated package-lock.json from 12 to 16. resolved merge conflicts in previous commit b650e2d - not tested
5e61df3 re-add env which was dropped in edit of merge conflict in 67f6969 environment.js
84273e4 result of ember-cli-update --to 4, with merge conflicts resolved
aebbd64 start of frontend upgrade to Ember4 and Bootstrap4
8aab55d Apply ember-no-implicit-this-codemod
b7e2ae2 upgrading frontend : relationship options, d3 keys,values,nest, capitalize, etc
cec261b update frontend/package-lock.json
8b4583c bs tab fade=false. link-to route. etc
3cac309 in progress : replacing sendAction() with closure actions and direct function calls. change uses of d3.event and d3.mouse
004fabf use native input in place of classic component
099b2a6 update frontend/package-lock.json
769d182 re-connect Zoom button to domain change
dd69ea1 reduce rendering from mousewheel zoom and pan

4edfb9b update remaining d3 .on functions
0643860 update remaining textareas
8c8d63c update axis drag, feature search results display

2024Mar15 -

0643860 update remaining textareas
4edfb9b update remaining d3 .on functions
d802eb1 updates for transitions, db edits and refreshes in groups pages
d2192e4 in Dockerfile update node version 12 to 18 and remove bower
deca791 feature search : change use of d3 nest to group for display of features blocks
aa8a3ce update use of popover
71f1041 update tab.pane and input parameters to restore layout in explore and servers list
74ef7ee pass closure action selectDataset down to entry-scope instead of using strings and actions defined in local js
f75ba24 update from d3.nest to d3.group requires implies in key values
4156b3b manual update of package-lock.json ember-ajax 5.0.0 to 5.1.2
52b413a regenerated package-lock.json
8dac400 changes to connection of adapter current request with server
857d5ca select correct server for groups requests during connection to secondary
a054db4 revert the change to glob in frontend/package-lock.json
272e831 minor updates
42582e6 Change remaining inputs with class form-control from class to angle bracket so that .form-control gives the element full-width
2df6c90 Change Textarea param input to oninput.
860b486 update params for FileDropzone
23422c3 log undefined levelMeta
2c76523 update file upload drag drop to work in Chrome also
45f95fa use closure action for onDelete; un-view before delete and refresh after

---

2024Apr11

• [2-3H] remainder of : replacing sendAction() with closure actions and direct function calls

[Don-Isdale]

Software maintenance and version upgrades of frameworks and dependencies

• add rollup to libraries for container build
• • 86f4601, 0a98037 add rollup to devDependencies
• built image for 2nd server on dirac; stashed feature/useBrAPI; started feature/upgradeFrontend2
• build of feature/upgradeFrontend2
• • 5c0ef18 Change plantinformatics/vcf-genotype-brapi dependency from local dev to version

[Don-Isdale]

2024Jul25-

b54f21b Set initial tab in left panel to Dataset Explorer

---

Testing

• VCF Genotype Table
• Blast / DNA Sequence Search

[Don-Isdale]

2024Aug07-

Characterising an issue : datasets / groups

In the Genotype Table on the internal test server an invalid value NaN was observed in the MAF columns, suggesting either a data error or an error in the feature/ongoingGenotype branch. The Network tab of the Web Inspector confirmed that the MAF values were valid (0); being a production build with minified code limited the ability to further drill into an issue, so the current development build (370 : feature/upgradeFrontend2) was used to connect to the test server as a secondary. This revealed that only public datasets were visible, not datasets in groups. Subsequent testing confirms that this is an issue for feature/upgradeFrontend2 also in the primary server datasets. Obviously the multiple server capability is not required for current user focus, although it does have value in development in accessing specific datasets on production servers via development-build applications, as in this case. The groups capability is required, so this issue is being investigated.

It has been characterised in local development, and seems to be related to changes in how relations are evaluated in the updated framework. The initial approach taken is to load the groups data earlier, so that it is available when dataset property .owner is evaluated.

After this is solved it should be possible to investigate the cause of the NaN in MAF in the Genotype Table, which is probably not specific to 370, but is required to work.

Changes made to re-enable display of datasets in groups

70e606d enable datasets in groups to display in Dataset Explorer

---

Correction to sample value calculations : exclude .INFO values

While investigating the datasets / groups issue it was observed that calculations on sample values were not excluding .INFO values of a feature SNP, so 'INFO' was added to the excluded values.

a1a4aab Exclude Feature .INFO from calculations on sample values

---

The MAF NaN issue is a known issue - not high priority at this point

It has been observed to appear when a SNP is mostly missing except for ALT calls (checking the DArT-seq AVR core dataset on the test server suggests this is the case).

[Don-Isdale]

Enable build on test server using docker build

• df1c3c8 change source of package ember-cli-build-date
• 57b0496 add glob and rollup. ensure ROOT_URL has a trailing slash.
• 738a6b3 Feb19 enable configuration of rootURL via environment variable ROOT_URL
• 4120726 Enable configuration of ROOT_URL in container image build

Fix reported issue related to absence of optional landingPageContent

• 09be10b catch exception from loading landingPageContent

[Don-Isdale]

Support use of external library

To enable the BrAPI and VCF Genotype functions of Pretzel to be used separately they were factored out to separate libraries during the BrAPI Hackathon. This requires changes to the module packaging in Pretzel. The following work was done to complete this process and enable a build which uses VCF Genotype and other functions from the external library.

8852b744 enable import of interval-bins binEvenLengthRound in package vcf-genotype-brapi
44de02a fix import from interval-bins

be1c684 adjust frontend dependency vcf-genotype-brapi

[Don-Isdale]

Complete known outstanding priority issues

Noted in meeting 2024Aug15

• initial panel position bottom/right change to right # this will be resolved in the merge with WG#3 branch

• in split axis features are initially small, update increases size
  643ae2e fix initial width of axis track rectangles

• increase font size of text throughout
  e219d60 increase base font size to match previous

• axis mousewheel-zoom : when zooming in near the top the zoom centre is offset
  c8d2729 adjust mousewheel zoom at top of axis

• changed to different fork of an external package ember-local-storage which is compatible with the updated frameworks and libraries.
  0c686c2 use github Facilecomm for ember-local-storage
  934ff56 import ember-local-storage from @Facilecomm

---

• left line of split axis runs through the tick labels, near their centre
  (screenshot in Teams)

• the right edge of split axis is not displayed - this only happens if you have QTLs loaded
  1bb4194 show right edge when axis with QTLs is split

• c70c3b1 change default extraOutsideMargin to 0

---

Lower Priority issues

• In Pretzel (current release) when the user clicks "reset zooms", it resets zooms and removes the zoom/reset buttons, regardless of where the zoom level is.
  In the update, it doesn't remove those buttons if you are fully zoomed out.

---

• error message received by front-end GUI should be displayed to user in GUI

    "error": {
        "statusCode": 400,
        "name": "Error",
        "message": "Error: VCF dataset dir is not configured, ...\n"
    }

---

2024Aug20

• right axis edge doesn't resize
  41fabf5 resize height of right axis edge when window resizes

[Don-Isdale]

Combine recent work done in parallel for the AGG Working Group, Use Case 3

Introduction

The update of framework and library versions has been done in a separate branch to avoid disruptions from other work, and to avoid disrupting other work. Now that the updated version is stable, the recent development work can be merged in, providing a single development path going forward.

A preliminary merge has been started, and looking at the merge conflicts, where changes have been made in the same portion of source code in both branches in parallel, the effort of resolving the merge conflicts does not look too great. It is possible that the new source code added to the update branch will require updates, but it is likely that these will be similar in type to the updates already done, and will not be too difficult.

Tasks

• merge with WG#3 (feature/workingGroup3_383_GenotypeSearch)
  Trial merge is complete - now updating the source from this branch and testing

b3eefdd merge changes from pretzel feature/workingGroup3_383_GenotypeSearch
1d32e4f Merge remote-tracking branch 'origin/feature/workingGroup3_383_GenotypeSearch' into feature/upgradeFrontend2

• updates for source from feature/workingGroup3_383_GenotypeSearch
• test the merge

Improve visibility of errors related to VCF Genotype directory / file configuration

This is related to the following comment and #401 .

• 0791873 show error message from vcfGenotypeLookup

---

Update : Merge with Working Group 3 branch

Development of Working Group 3 VCF Genotype Search facility was done in a branch feature/workingGroup3_383_GenotypeSearch parallel to this branch feature/upgradeFrontend2, to allow those 2 tasks to proceed without disrupting each other.
As they are now both complete, they have been merged together, and testing of the result has started.
When complete, the merged result will be merged to develop & master / main.

After some software changes to make the source from WG3 branch compatible with the updated frameworks and libraries, the WG3 search feature was performed succesfully, shown in the attached screenshot.

---

[Don-Isdale]

Test handling of VCF Directory misconfiguration

It was observed in an earlier version that misconfiguration of a VCF directory, i.e. not matching the uploaded datasetId, was able to cause the Pretzel server to exit, which is disruptive. The error recorded in the server log in that case is appended.

---

Test set-up

Rename one of the dataset directories temporarily to trigger the error :

# cd pretzel/lb4app/tmp/vcf
mkdir 2
mv -i  201028_40K_DAS5_samples_XT_exomeIDs 2/.

Test Results - OK

The server reported the misconfiguration in the API response, with status code 400.

HTTP/1.1 400 Bad Request

{"error":{"statusCode":400,"name":"Error","message":"Error: VCF dataset dir is not configured, 201028_40K_DAS5_samples_XT_exomeIDs\n"}}

server log:

++ cd tmp/vcf
++ cd 201028_40K_DAS5_samples_XT_exomeIDs
lb3app/scripts/vcfGenotypeLookup.bash: line 295: cd: 201028_40K_DAS5_samples_XT_exomeIDs: No such file or directory
++ echo Error: 'VCF dataset dir is not configured,' 201028_40K_DAS5_samples_XT_exomeIDs
++ return 1
+ vcfGz=
+ isecDatasetIdsArray=($(echo "$isecDatasetIds" | tr '!' ' '))
childProcess vcfGenotypeLookup.bash  errors : Error: VCF dataset dir is not configured, 201028_40K_DAS5_samples_XT_exomeIDs

The server continued to function after sending the error response, and did not exit.

---

Message recorded in server log in the original error report

child process exited with code 0
cbWrap Error: Error: 0 VCF dataset dir is not configured, Wheat_CSv2.1_..., PWD=/
undefined undefined
/app/lb3app/common/utilities/errorStatus.js:10
  let e = Error(text);
          ^

Error: Error: 0 VCF dataset dir is not configured, Wheat_CSv2.1_..., PWD=/

    at new exports.ErrorStatus (/app/lb3app/common/utilities/errorStatus.js:10:11)
    at ChildProcess.<anonymous> (/app/lb3app/common/utilities/child-process.js:183:10)
    at ChildProcess.emit (node:events:527:28)
    at maybeClose (node:internal/child_process:1092:16)
    at Socket.<anonymous> (node:internal/child_process:451:11)
    at Socket.emit (node:events:527:28)
    at Pipe.<anonymous> (node:net:709:12) {
  statusCode: 400
}

[Don-Isdale]

Content moved into earlier comment on the same subject.

--

Fix to an issue found during testing of v3.0.0 release, 2024Oct22

c17ceaa disable CSV Download button so that paths are shown in table

Tested OK after the above commit :

[gabrielkg]

Merged into Working Group 3 branch. Any new bugs will be recorded as separate issues.