Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update - Patches for third party links used by Ariba #24010

Merged
merged 6 commits into from
Aug 20, 2020

Conversation

rpetit3
Copy link
Member

@rpetit3 rpetit3 commented Aug 19, 2020

This PR includes two patches for Ariba related to changes in API endpoints.

pubmlst_getter.patch fixes broken PubMLST downloads due to a change in API endpoint.

Original Issue: sanger-pathogens/ariba#304

Submitted Ariba PR: sanger-pathogens/ariba#305

ref_genes_getter.patch fixes a change in CARD links that causes RGI to be downloading instead of CARD

Original Issue: sanger-pathogens/ariba#301

Submitted Ariba PR: sanger-pathogens/ariba#303

This my first time submitting a PR with patches so any advice would be appreaciated. I generated the patches through GitHub by appending .patch to the PR link.

https://github.com/sanger-pathogens/ariba/pull/303.patch

https://github.com/sanger-pathogens/ariba/pull/305.patch


Please read the guidelines for Bioconda recipes before opening a pull request (PR).

  • If this PR adds or updates a recipe, use "Add" or "Update" appropriately as the first word in its title.
  • New recipes not directly relevant to the biological sciences need to be submitted to the conda-forge channel instead of Bioconda.
  • PRs require reviews prior to being merged. Once your PR is passing tests and ready to be merged, please issue the @BiocondaBot please add label command.
  • Please post questions on Gitter or ping @bioconda/core in a comment.
Please use the following BiocondaBot commands:

Everyone has access to the following BiocondaBot commands, which can be given in a comment:

@BiocondaBot please update Merge the master branch into a PR.
@BiocondaBot please add label Add the please review & merge label.
@BiocondaBot please fetch artifacts Post links to CI-built packages/containers.
You can use this to test packages locally.

For members of the Bioconda project, the following command is also available:

@BiocondaBot please merge Upload built packages/containers and merge a PR.
Someone must approve a PR first!
This reduces CI build time by reusing built artifacts.

Also, the bot watches for comments from non-members that include @bioconda/<team> and will automatically re-post them to notify the addressed <team>.

@rpetit3
Copy link
Member Author

rpetit3 commented Aug 19, 2020

@bioconda-bot please fetch artifacts

@BiocondaBot
Copy link
Collaborator

Package(s) built on CircleCI are ready for inspection:

Arch Package Repodata
linux-64 ariba-2.14.5-py36hf0b53f7_2.tar.bz2 repodata.json
linux-64 ariba-2.14.5-py37hfa133b6_2.tar.bz2 repodata.json
osx-64 ariba-2.14.5-py36h043d725_2.tar.bz2 repodata.json
osx-64 ariba-2.14.5-py37hbeab489_2.tar.bz2 repodata.json

You may also use conda to install these:

  • For packages on linux-64:
conda install -c https://120647-42372094-gh.circle-artifacts.com/0/tmp/artifacts/packages <package name>
  • For packages on osx-64:
conda install -c https://120648-42372094-gh.circle-artifacts.com/0/tmp/artifacts/packages <package name>

Docker image(s) built:

Package Tag Install with docker
ariba 2.14.5--py36hf0b53f7_2
showcurl "https://120647-42372094-gh.circle-artifacts.com/0/tmp/artifacts/images/ariba%3A2.14.5--py36hf0b53f7_2.tar.gz" | gzip -dc | docker load
ariba 2.14.5--py37hfa133b6_2
showcurl "https://120647-42372094-gh.circle-artifacts.com/0/tmp/artifacts/images/ariba%3A2.14.5--py37hfa133b6_2.tar.gz" | gzip -dc | docker load

@rpetit3
Copy link
Member Author

rpetit3 commented Aug 19, 2020

I'm going to test locally, to verify patches worked

@rpetit3
Copy link
Member Author

rpetit3 commented Aug 19, 2020

conda create -n ariba2 -c https://120647-42372094-gh.circle-artifacts.com/0/tmp/artifacts/packages -c conda-forge -c bioconda 
conda activate

# Get ariba version
ariba version
Traceback (most recent call last):
  File "/local/home/rpetit/miniconda3/envs/ariba2/bin/ariba", line 312, in <module>
    args.func(args)
  File "/local/home/rpetit/miniconda3/envs/ariba2/lib/python3.6/site-packages/ariba/tasks/version.py", line 5, in run
    extern_progs, report_lines = versions.get_all_versions(raise_error=False)
  File "/local/home/rpetit/miniconda3/envs/ariba2/lib/python3.6/site-packages/ariba/versions.py", line 21, in get_all_versions
    extern_progs = external_progs.ExternalProgs(fail_on_error=False, using_spades=using_spades)
  File "/local/home/rpetit/miniconda3/envs/ariba2/lib/python3.6/site-packages/ariba/external_progs.py", line 89, in __init__
    if prog in min_versions and LooseVersion(version) < LooseVersion(min_versions[prog]):
  File "/local/home/rpetit/miniconda3/envs/ariba2/lib/python3.6/distutils/version.py", line 52, in __lt__
    c = self._cmp(other)
  File "/local/home/rpetit/miniconda3/envs/ariba2/lib/python3.6/distutils/version.py", line 335, in _cmp
    if self.version == other.version:
AttributeError: 'LooseVersion' object has no attribute 'version'

# Current Bioconda version of Bowtie2 doesn't output version correctly
conda install -y -c conda-forge -c bioconda bowtie2=2.3.5

# Test version again
ariba version
WARNING: I tried to get the version of spades with: "/local/home/rpetit/miniconda3/envs/ariba2/bin/spades.py --version" and the output didn't match this regular expression: "SPAdes\s+v([0-9\.]+)"
ARIBA version: 2.14.5

External dependencies:
bowtie2 2.3.5.1 /local/home/rpetit/miniconda3/envs/ariba2/bin/bowtie2
cdhit   4.8.1   /local/home/rpetit/miniconda3/envs/ariba2/bin/cd-hit-est
nucmer  3.1     /local/home/rpetit/miniconda3/envs/ariba2/bin/nucmer
spades  ERROR   /local/home/rpetit/miniconda3/envs/ariba2/bin/spades.py

External dependencies OK: True

Python version:
3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:09:42)
[GCC 7.5.0]

Python packages:
ariba   2.14.5  /local/home/rpetit/miniconda3/envs/ariba2/lib/python3.6/site-packages/ariba/__init__.py
bs4     4.9.1   /local/home/rpetit/miniconda3/envs/ariba2/lib/python3.6/site-packages/bs4/__init__.py
dendropy        4.4.0   /local/home/rpetit/miniconda3/envs/ariba2/lib/python3.6/site-packages/dendropy/__init__.py
pyfastaq        3.17.0  /local/home/rpetit/miniconda3/envs/ariba2/lib/python3.6/site-packages/pyfastaq/__init__.py
pymummer        0.10.3  /local/home/rpetit/miniconda3/envs/ariba2/lib/python3.6/site-packages/pymummer/__init__.py
pysam   0.16.0.1        /local/home/rpetit/miniconda3/envs/ariba2/lib/python3.6/site-packages/pysam/__init__.py

Python packages OK: True

Everything looks OK: True

# Test PubMLST
ariba pubmlstget "Staphylococcus aureus" test_pubmlst
WARNING: I tried to get the version of spades with: "/local/home/rpetit/miniconda3/envs/ariba2/bin/spades.py --version" and the output didn't match this regular expression: "SPAdes\s+v([0-9\.]+)"
WARNING: Median sequence length is 456 but arcC.548 has length 522 which is too long or short. Removing.
WARNING: Median sequence length is 456 but arcC.567 has length 516 which is too long or short. Removing.
ariba db directory prepared. You can use it like this:
ariba run test_pubmlst/ref_db reads_1.fq reads_2.fq output_directory

ls test_pubmlst/pubmlst_download/
arcC.tfa  aroE.tfa  glpF.tfa  gmk.tfa  profile.txt  pta.tfa  tpi.tfa  yqiL.tfa

# Test Card
ariba getref card card
Getting available CARD versions
Downloading "https://card.mcmaster.ca/download" and saving as "download.html" ... done
Found versions:
1.0.0   https://card.mcmaster.ca/download/0/broadstreet-v1.0.0.tar.bz2
1.0.1   https://card.mcmaster.ca/download/0/broadstreet-v1.0.1.tar.bz2
1.0.2   https://card.mcmaster.ca/download/0/broadstreet-v1.0.2.tar.bz2
1.0.3   https://card.mcmaster.ca/download/0/broadstreet-v1.0.3.tar.bz2
1.0.4   https://card.mcmaster.ca/download/0/broadstreet-v1.0.4.tar.bz2
1.0.5   https://card.mcmaster.ca/download/0/broadstreet-v1.0.5.tar.bz2
1.0.6   https://card.mcmaster.ca/download/0/broadstreet-v1.0.6.tar.bz2
1.0.7   https://card.mcmaster.ca/download/0/broadstreet-v1.0.7.tar.bz2
1.0.8   https://card.mcmaster.ca/download/0/broadstreet-v1.0.8.tar.bz2
1.0.9   https://card.mcmaster.ca/download/0/broadstreet-v1.0.9.tar.bz2
1.1.0   https://card.mcmaster.ca/download/0/broadstreet-v1.1.0.tar.bz2
1.1.1   https://card.mcmaster.ca/download/0/broadstreet-v1.1.1.tar.bz2
1.1.2   https://card.mcmaster.ca/download/0/broadstreet-v1.1.2.tar.bz2
1.1.3   https://card.mcmaster.ca/download/0/broadstreet-v1.1.3.tar.bz2
1.1.4   https://card.mcmaster.ca/download/0/broadstreet-v1.1.4.tar.bz2
1.1.5   https://card.mcmaster.ca/download/0/broadstreet-v1.1.5.tar.bz2
1.1.6   https://card.mcmaster.ca/download/0/broadstreet-v1.1.6.tar.bz2
1.1.7   https://card.mcmaster.ca/download/0/broadstreet-v1.1.7.tar.bz2
1.1.8   https://card.mcmaster.ca/download/0/broadstreet-v1.1.8.tar.bz2
1.1.9   https://card.mcmaster.ca/download/0/broadstreet-v1.1.9.tar.bz2
1.2.0   https://card.mcmaster.ca/download/0/broadstreet-v1.2.0.tar.bz2
1.2.1   https://card.mcmaster.ca/download/0/broadstreet-v1.2.1.tar.bz2
2.0.0   https://card.mcmaster.ca/download/0/broadstreet-v2.0.0.tar.gz
2.0.1   https://card.mcmaster.ca/download/0/broadstreet-v2.0.1.tar.gz
2.0.2   https://card.mcmaster.ca/download/0/broadstreet-v2.0.2.tar.gz
2.0.3   https://card.mcmaster.ca/download/0/broadstreet-v2.0.3.tar.gz
3.0.0   https://card.mcmaster.ca/download/0/broadstreet-v3.0.0.tar.gz
3.0.1   https://card.mcmaster.ca/download/0/broadstreet-v3.0.1.tar.gz
3.0.2   https://card.mcmaster.ca/download/0/broadstreet-v3.0.2.tar.gz
3.0.3   https://card.mcmaster.ca/download/0/broadstreet-v3.0.3.tar.gz
3.0.4   https://card.mcmaster.ca/download/0/broadstreet-v3.0.4.tar.gz
3.0.5   https://card.mcmaster.ca/download/0/broadstreet-v3.0.5.tar.gz
3.0.6   https://card.mcmaster.ca/download/0/broadstreet-v3.0.6.tar.gz
3.0.7   https://card.mcmaster.ca/download/0/broadstreet-v3.0.7.tar.gz
3.0.8   https://card.mcmaster.ca/download/0/broadstreet-v3.0.8.tar.bz2
3.0.9   https://card.mcmaster.ca/download/0/broadstreet-v3.0.9.tar.bz2
3.1.0   https://card.mcmaster.ca/download/0/broadstreet-v3.1.0.tar.bz2
Getting version 3.1.0
Working in temporary directory /local/home/rpetit/temp_blast/card.download
Downloading data from card: https://card.mcmaster.ca/download/0/broadstreet-v3.1.0.tar.bz2
syscall: wget -O card.tar.bz2 https://card.mcmaster.ca/download/0/broadstreet-v3.1.0.tar.bz2
...finished downloading
Extracted json data file ./card.json. Reading its contents...
Found 3022 records in the json file. Analysing...
Extracted data and written ARIBA input files

Finished. Final files are:
        /local/home/rpetit/temp_blast/card.fa
        /local/home/rpetit/temp_blast/card.tsv

You can use them with ARIBA like this:
ariba prepareref -f /local/home/rpetit/temp_blast/card.fa -m /local/home/rpetit/temp_blast/card.tsv output_directory

If you use this downloaded data, please cite:
"The Comprehensive Antibiotic Resistance Database", McArthur et al 2013, PMID: 23650175
and in your methods say that version 3.1.0 of the database was used

Looks like everything is working correctly. I'm going to pin the version of Bowtie2 to be <2.4 (See for more info: #22836)

Test version, since there are open Issues related to bowtie2 version info and it affects ariba
@rpetit3
Copy link
Member Author

rpetit3 commented Aug 20, 2020

@bioconda-bot please fetch artifacts

@rpetit3
Copy link
Member Author

rpetit3 commented Aug 20, 2020

@bioconda-bot please label

@BiocondaBot
Copy link
Collaborator

Package(s) built on CircleCI are ready for inspection:

Arch Package Repodata
linux-64 ariba-2.14.5-py36hf0b53f7_2.tar.bz2 repodata.json
linux-64 ariba-2.14.5-py37hfa133b6_2.tar.bz2 repodata.json
osx-64 ariba-2.14.5-py36h043d725_2.tar.bz2 repodata.json
osx-64 ariba-2.14.5-py37hbeab489_2.tar.bz2 repodata.json

You may also use conda to install these:

  • For packages on linux-64:
conda install -c https://120660-42372094-gh.circle-artifacts.com/0/tmp/artifacts/packages <package name>
  • For packages on osx-64:
conda install -c https://120661-42372094-gh.circle-artifacts.com/0/tmp/artifacts/packages <package name>

Docker image(s) built:

Package Tag Install with docker
ariba 2.14.5--py36hf0b53f7_2
showcurl "https://120660-42372094-gh.circle-artifacts.com/0/tmp/artifacts/images/ariba%3A2.14.5--py36hf0b53f7_2.tar.gz" | gzip -dc | docker load
ariba 2.14.5--py37hfa133b6_2
showcurl "https://120660-42372094-gh.circle-artifacts.com/0/tmp/artifacts/images/ariba%3A2.14.5--py37hfa133b6_2.tar.gz" | gzip -dc | docker load

@rpetit3
Copy link
Member Author

rpetit3 commented Aug 20, 2020

@bioconda-bot please add label

@BiocondaBot BiocondaBot added the please review & merge set to ask for merge label Aug 20, 2020
@rpetit3
Copy link
Member Author

rpetit3 commented Aug 20, 2020

Retested, good to go

(base) rpetit@staphopia:~$ conda create -y -n ariba-patched -c https://120660-42372094-gh.circle-artifacts.com/0/tmp/artifacts/packages -c conda-forge -c bioconda ariba
(base) rpetit@staphopia:~$ conda activate ariba-patched
(ariba-patched) rpetit@staphopia:~$ ariba version
WARNING: I tried to get the version of spades with: "/local/home/rpetit/miniconda3/envs/ariba-patched/bin/spades.py --version" and the output didn't match this regular expression: "SPAdes\s+v([0-9\.]+)"
ARIBA version: 2.14.5

External dependencies:
bowtie2 2.3.5.1 /local/home/rpetit/miniconda3/envs/ariba-patched/bin/bowtie2
cdhit   4.8.1   /local/home/rpetit/miniconda3/envs/ariba-patched/bin/cd-hit-est
nucmer  3.1     /local/home/rpetit/miniconda3/envs/ariba-patched/bin/nucmer
spades  ERROR   /local/home/rpetit/miniconda3/envs/ariba-patched/bin/spades.py

External dependencies OK: True

Python version:
3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:09:42)
[GCC 7.5.0]

Python packages:
ariba   2.14.5  /local/home/rpetit/miniconda3/envs/ariba-patched/lib/python3.6/site-packages/ariba/__init__.py
bs4     4.9.1   /local/home/rpetit/miniconda3/envs/ariba-patched/lib/python3.6/site-packages/bs4/__init__.py
dendropy        4.4.0   /local/home/rpetit/miniconda3/envs/ariba-patched/lib/python3.6/site-packages/dendropy/__init__.py
pyfastaq        3.17.0  /local/home/rpetit/miniconda3/envs/ariba-patched/lib/python3.6/site-packages/pyfastaq/__init__.py
pymummer        0.10.3  /local/home/rpetit/miniconda3/envs/ariba-patched/lib/python3.6/site-packages/pymummer/__init__.py
pysam   0.16.0.1        /local/home/rpetit/miniconda3/envs/ariba-patched/lib/python3.6/site-packages/pysam/__init__.py

Python packages OK: True

Everything looks OK: True
(ariba-patched) rpetit@staphopia:~$ cd temp_blast/
(ariba-patched) rpetit@staphopia:~/temp_blast$ ariba pubmlstget "Staphylococcus aureus" test_pubmlst2
WARNING: I tried to get the version of spades with: "/local/home/rpetit/miniconda3/envs/ariba-patched/bin/spades.py --version" and the output didn't match this regular expression: "SPAdes\s+v([0-9\.]+)"
WARNING: Median sequence length is 456 but arcC.548 has length 522 which is too long or short. Removing.
WARNING: Median sequence length is 456 but arcC.567 has length 516 which is too long or short. Removing.
ariba db directory prepared. You can use it like this:
ariba run test_pubmlst2/ref_db reads_1.fq reads_2.fq output_directory
(ariba-patched) rpetit@staphopia:~/temp_blast$ ariba getref card card
Getting available CARD versions
Downloading "https://card.mcmaster.ca/download" and saving as "download.html" ... done
Found versions:
1.0.0   https://card.mcmaster.ca/download/0/broadstreet-v1.0.0.tar.bz2
1.0.1   https://card.mcmaster.ca/download/0/broadstreet-v1.0.1.tar.bz2
1.0.2   https://card.mcmaster.ca/download/0/broadstreet-v1.0.2.tar.bz2
1.0.3   https://card.mcmaster.ca/download/0/broadstreet-v1.0.3.tar.bz2
1.0.4   https://card.mcmaster.ca/download/0/broadstreet-v1.0.4.tar.bz2
1.0.5   https://card.mcmaster.ca/download/0/broadstreet-v1.0.5.tar.bz2
1.0.6   https://card.mcmaster.ca/download/0/broadstreet-v1.0.6.tar.bz2
1.0.7   https://card.mcmaster.ca/download/0/broadstreet-v1.0.7.tar.bz2
1.0.8   https://card.mcmaster.ca/download/0/broadstreet-v1.0.8.tar.bz2
1.0.9   https://card.mcmaster.ca/download/0/broadstreet-v1.0.9.tar.bz2
1.1.0   https://card.mcmaster.ca/download/0/broadstreet-v1.1.0.tar.bz2
1.1.1   https://card.mcmaster.ca/download/0/broadstreet-v1.1.1.tar.bz2
1.1.2   https://card.mcmaster.ca/download/0/broadstreet-v1.1.2.tar.bz2
1.1.3   https://card.mcmaster.ca/download/0/broadstreet-v1.1.3.tar.bz2
1.1.4   https://card.mcmaster.ca/download/0/broadstreet-v1.1.4.tar.bz2
1.1.5   https://card.mcmaster.ca/download/0/broadstreet-v1.1.5.tar.bz2
1.1.6   https://card.mcmaster.ca/download/0/broadstreet-v1.1.6.tar.bz2
1.1.7   https://card.mcmaster.ca/download/0/broadstreet-v1.1.7.tar.bz2
1.1.8   https://card.mcmaster.ca/download/0/broadstreet-v1.1.8.tar.bz2
1.1.9   https://card.mcmaster.ca/download/0/broadstreet-v1.1.9.tar.bz2
1.2.0   https://card.mcmaster.ca/download/0/broadstreet-v1.2.0.tar.bz2
1.2.1   https://card.mcmaster.ca/download/0/broadstreet-v1.2.1.tar.bz2
2.0.0   https://card.mcmaster.ca/download/0/broadstreet-v2.0.0.tar.gz
2.0.1   https://card.mcmaster.ca/download/0/broadstreet-v2.0.1.tar.gz
2.0.2   https://card.mcmaster.ca/download/0/broadstreet-v2.0.2.tar.gz
2.0.3   https://card.mcmaster.ca/download/0/broadstreet-v2.0.3.tar.gz
3.0.0   https://card.mcmaster.ca/download/0/broadstreet-v3.0.0.tar.gz
3.0.1   https://card.mcmaster.ca/download/0/broadstreet-v3.0.1.tar.gz
3.0.2   https://card.mcmaster.ca/download/0/broadstreet-v3.0.2.tar.gz
3.0.3   https://card.mcmaster.ca/download/0/broadstreet-v3.0.3.tar.gz
3.0.4   https://card.mcmaster.ca/download/0/broadstreet-v3.0.4.tar.gz
3.0.5   https://card.mcmaster.ca/download/0/broadstreet-v3.0.5.tar.gz
3.0.6   https://card.mcmaster.ca/download/0/broadstreet-v3.0.6.tar.gz
3.0.7   https://card.mcmaster.ca/download/0/broadstreet-v3.0.7.tar.gz
3.0.8   https://card.mcmaster.ca/download/0/broadstreet-v3.0.8.tar.bz2
3.0.9   https://card.mcmaster.ca/download/0/broadstreet-v3.0.9.tar.bz2
3.1.0   https://card.mcmaster.ca/download/0/broadstreet-v3.1.0.tar.bz2
Getting version 3.1.0
Working in temporary directory /local/home/rpetit/temp_blast/card.download
Downloading data from card: https://card.mcmaster.ca/download/0/broadstreet-v3.1.0.tar.bz2
syscall: wget -O card.tar.bz2 https://card.mcmaster.ca/download/0/broadstreet-v3.1.0.tar.bz2
...finished downloading
Extracted json data file ./card.json. Reading its contents...
Found 3022 records in the json file. Analysing...
Extracted data and written ARIBA input files

Finished. Final files are:
        /local/home/rpetit/temp_blast/card.fa
        /local/home/rpetit/temp_blast/card.tsv

You can use them with ARIBA like this:
ariba prepareref -f /local/home/rpetit/temp_blast/card.fa -m /local/home/rpetit/temp_blast/card.tsv output_directory

If you use this downloaded data, please cite:
"The Comprehensive Antibiotic Resistance Database", McArthur et al 2013, PMID: 23650175
and in your methods say that version 3.1.0 of the database was used

@simonbray
Copy link
Member

@bioconda-bot please merge

Looks good @rpetit3

@BiocondaBot
Copy link
Collaborator

I will attempt to upload artifacts and merge this PR. This may take some time, please have patience.

@BiocondaBot BiocondaBot merged commit b65890c into master Aug 20, 2020
@BiocondaBot BiocondaBot deleted the rpetit3-ariba-patch-1 branch August 20, 2020 08:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
please review & merge set to ask for merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants