Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Standardisation of Pretzel Deployment #377

Closed
kennyAgricultureVic opened this issue May 24, 2024 · 5 comments
Closed

Standardisation of Pretzel Deployment #377

kennyAgricultureVic opened this issue May 24, 2024 · 5 comments
Assignees
@Don-Isdale

Comments

@kennyAgricultureVic
Copy link
Contributor

kennyAgricultureVic commented May 24, 2024 •
edited by Don-Isdale
Loading

Pretzel is designed to be deployed in many different contexts :

  • in a public cloud such as AWS
  • in an internal data centre, associated with a supercomputing cluster
  • on a personal desktop or laptop, supporting the major PC operating systems : MS Windows, Linux and iOS / MacOS

The configuration requirements for each of these contexts is progressively being integrated together into a common deployment method which provides configuration to handle the variations.

This process streamlines system administration and reduces the effort of supporting multiple platforms. The focus is on making the deployment of Pretzel quick and easy to manage for staff administering the system, and enabling non-specialists to install Pretzel on their PC with minimum effort. This includes access to the connected databases such as VCF Genotypes, Blast databases, and other external systems.

The focus of this github issue is this process of standardising the deployment of Pretzel.
@kennyAgricultureVic
Copy link
Contributor Author

  • run blast micro-service on test server to enable blast within container

@Don-Isdale
Copy link
Collaborator

Don-Isdale commented May 31, 2024 •
edited
Loading

Deployment

  • pulses VCF config / error
    • check pretzel-hosting version and deploy.sh link on other hosts
    • commit additions to deploy script repo pretzel-hosting
      da0fc6b
      add functions prepareLogDir and doCommandOnServers
  • added --volume blast to container run configuration on test server
  • documented process for building and running Pretzel server containers on test server 14cfef6

@Don-Isdale Don-Isdale mentioned this issue Jun 21, 2024
Open
2 tasks
@Don-Isdale
Copy link
Collaborator

Don-Isdale commented Jun 25, 2024 •
edited
Loading

Simplify the installation process for self-hosting on a PC by adding a docker-compose yaml based on an existing docker-compose.yaml, with added shared volumes for VCF and blast databases, and including the python flask blastServer.py,
Targetting MS Windows PC - define the data directories relative to standard directories e.g. My Documents\

  • create a Dockerfile for the blastServer.py

  • add that Dockerfile to a docker-compose.yaml based on existing docker-compose.yaml with mongoDb + Pretzel server

    • change file path to reflect blastServer running within a container
      1e0f721 include the Python Flask blastServer.py in a docker-compose yaml

  • add volume : resultCache

@Don-Isdale Don-Isdale self-assigned this Jul 1, 2024
Don-Isdale added a commit that referenced this issue Jul 4, 2024
@Don-Isdale
include the Python Flask blastServer.py in a docker-compose yaml
1e0f721 
part of #377

blastServer.py :
change scriptsDir to /usr/src/app for the container case; this can be passed in from the environment to support command-line use.

blastn_cont.bash : change file paths for running in a container which imports just volume /mnt/data_blast/blast/ :
 add B_suffix
 re-structure conditionals -d queries and blastnIsInstalled -eq 0; retain support for ./queries (may not be required), and add support for blastDir/queries which is the case in the container.
 pass blastDir instead of B to sub-container and give it B_suffix/ in path of dbName.

add blastServer.Dockerfile (used as starting point : bash commands from blast.md converted via ChatGPT 4o to Dockerfile format)

add lb4app/lb3app/scripts/docker-compose/docker-compose.combined.yaml, based on pretzel/docker-compose.yaml with these changes :
  database:
    specify mongo image 9b5c4a4fdcb5 i.e. 4.2.24 which matches current dev db
    enable volumes: DATA_DIR
  api:
    environment: add : API_PORT_PROXY, EMAIL_USER, EMAIL_PASS
    volumes: add blast and vcf

  add service : blastserver
This was referenced Jul 8, 2024
Closed
Open
Don-Isdale added a commit that referenced this issue Jul 18, 2024
@Don-Isdale
include the Python Flask blastServer.py in a docker-compose yaml
15fa1a5 
part of #377

blastServer.py :
change scriptsDir to /usr/src/app for the container case; this can be passed in from the environment to support command-line use.

blastn_cont.bash : change file paths for running in a container which imports just volume /mnt/data_blast/blast/ :
 add B_suffix
 re-structure conditionals -d queries and blastnIsInstalled -eq 0; retain support for ./queries (may not be required), and add support for blastDir/queries which is the case in the container.
 pass blastDir instead of B to sub-container and give it B_suffix/ in path of dbName.

add blastServer.Dockerfile (used as starting point : bash commands from blast.md converted via ChatGPT 4o to Dockerfile format)

add lb4app/lb3app/scripts/docker-compose/docker-compose.combined.yaml, based on pretzel/docker-compose.yaml with these changes :
  database:
    specify mongo image 9b5c4a4fdcb5 i.e. 4.2.24 which matches current dev db
    enable volumes: DATA_DIR
  api:
    environment: add : API_PORT_PROXY, EMAIL_USER, EMAIL_PASS
    volumes: add blast and vcf

  add service : blastserver

(cherry picked from commit 1e0f721)
@Don-Isdale
Copy link
Collaborator

Adjust configuration for internal test server

To generalise the docker compose yaml to additional contexts, it will be used to run the internal test server.
By ensuring that the docker compose yaml can be configured to support all the context which we expect users will want to install the Pretzel application server on, we streamline the usability and uptake of the application.

This context is different in several ways to the web-facing AWS servers, e.g. database configuration. Also the test server is the second server on the host it is running on, so the ports for mongodb and blastserver should not confict, and ROOT_URL is used to position the server URL in a separate URL sub-directory. The servers on AWS use sub-domains instead to separate URL spaces of multiple servers on a single host, which is easy to configure in nginx and requires no special configuration in the application server.

Changes Required

I had to copy some extra binaries into the blastserver container : /usr/bin/parallel and vmtouch
I got them from the ncbi/blast image.
I'll add those to the blastserver image via blastServer.Dockerfile
I got a successful test with IWGSC_RefSeq_v1.0, WAPO1

There were a bunch of other configurations to add / change :

  • db.createUser( ) in pretzel2 database
  • add DB_USER, DB_PASS to the .env file
  • add volumes :
    • /group directories for data
    • /local-ssd/data/resultsCache:/app/node_modules/flat-cache/.cache

Results

/pretzelUpdate is now running from docker-compose, and blast works.

@Don-Isdale Don-Isdale mentioned this issue Oct 22, 2024
Merged
@Don-Isdale
Copy link
Collaborator

Don-Isdale commented Nov 8, 2024 •
edited
Loading

Complete and used in v3.0.0 Release

This work is complete and is used in the v3.0.0 release #414 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Don-Isdale Don-Isdale

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Don-Isdale @kennyAgricultureVic