Remove url-based uploads

[seppinho]

No description provided.

[abought]

Using the documented example curl command for url-based uploads (adjusted for our server), I tried to replicate the upload behavior for files-source=sftp and files-source=http uploads.

The fix appears to work as intended- but I noticed some oddities, so can't be sure that my validation plan was comprehensive.

1. Unclear feedback: The job is actually accepted as 200 OK, and the error only appears later when the job is run. It's a minor nit, but removing a programmatic feature might be more intuitive if the request generates a 400 (BAD REQUEST) on job creation. If you're sending jobs programmatically, the difference lies in how many failure emails you get at once. ;)

2. The initial bug report identified a parameter called input. When I used that parameter (and omitted files altogether), the job was accepted, then failed during "input validation: analyze files" with the following wildly unhelpful log. Does the ability to submit a job with no files indicate some other mechanism of submission? Should I test this also?

curl https://test-server/api/v2/jobs/submit/[email protected] -H "X-Auth-Token: $TOKEN" -F "input=https://imputationserver.sph.umich.edu/static/downloads/hapmap300.chr1.recode.vcf.gz" -F "refpanel=apps@[email protected]" -F "population=all" -F "mode=qconly" --insecure

job.txt:

23/04/07 17:36:14 Executing Job installation....
23/04/07 17:36:14 Preparing Application 'Genotype Imputation (Minimac4)'...
23/04/07 17:36:14 Application 'Genotype Imputation (Minimac4)'is already installed.
23/04/07 17:36:14 Preparing Application 'TOPMed r2'...
23/04/07 17:36:14 Application 'TOPMed r2'is already installed.
23/04/07 17:36:14 Setup External Workspace on Amazon S3
23/04/07 17:36:14 Executing Job setups....
23/04/07 17:36:14 Planner: WDL evaluated.
23/04/07 17:36:14 Planner: DAG created.
23/04/07 17:36:14 Nodes: 3
23/04/07 17:36:14 Input Validation
23/04/07 17:36:14 Inputs:
23/04/07 17:36:14 Outputs:
23/04/07 17:36:14 Quality Control
23/04/07 17:36:14 Inputs:
23/04/07 17:36:14 Outputs: mafFile chunkFileDir statisticDir
23/04/07 17:36:14 Quality Control (Report)
23/04/07 17:36:14 Inputs: mafFile files myseparator
23/04/07 17:36:14 Outputs: qcreport
23/04/07 17:36:14 Dependencies: 2
23/04/07 17:36:14 Input Validation->Quality Control
23/04/07 17:36:14 Quality Control->Quality Control (Report)
23/04/07 17:36:14 Executor: execute DAG...
23/04/07 17:36:14 ------------------------------------------------------
23/04/07 17:36:14 Input Validation
23/04/07 17:36:14 ------------------------------------------------------
23/04/07 17:36:14 Versions:
23/04/07 17:36:14 Pipeline: michigan-imputationserver-1.7.2
23/04/07 17:36:14 Imputation-Engine: minimac4-1.0.2
23/04/07 17:36:14 Phasing-Engine: eagle-2.4
23/04/07 17:36:14 Executing onFailure...
23/04/07 17:36:14 ------------------------------------------------------
23/04/07 17:36:14 Send Notification on Failure
23/04/07 17:36:14 ------------------------------------------------------
23/04/07 17:36:15 Send Notification on Failure [0 sec]
23/04/07 17:36:15 onFailure execution failed.
23/04/07 17:36:15 23/04/07 17:36:15 Job execution failed: null
23/04/07 17:36:15 Cleaning up...
23/04/07 17:36:15 Cleaning up uploaded local files...
23/04/07 17:36:15 Cleaning up temporary local files...
23/04/07 17:36:15 Cleaning up temporary hdfs files...
23/04/07 17:36:15 Cleaning up hdfs files...
23/04/07 17:36:15 Cleanup successful.

std.out:

We have sent an email to [email protected] with the error message.

[seppinho]

Hi,
Regarding your point 2 The error occured since the "files" parameter has been ommited (which resulted in a null pointer exception). We added an additonal check to the MIS repo. See here: genepi/imputationserver#106 and genepi/imputationserver#107

[lukfor]

Thanks Andy, good points! We added a check before job creation to detect url-based uploads and to return 400 (should fix your point 1). See PR #96