Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ariba fails without --noclean depending on database #205

Closed
kimleeng opened this issue Nov 29, 2017 · 10 comments
Closed

Ariba fails without --noclean depending on database #205

kimleeng opened this issue Nov 29, 2017 · 10 comments
Labels
bug

Comments

@kimleeng
Copy link

kimleeng commented Nov 29, 2017 •
edited
Loading

Recently updated ariba from 2.10.1 and noticed this issue emerge. Running on our installed resfinder DB has no issues however running others such as a mlst db or virulence finder leads to issues, see below). All mlst DBs were updated on Nov 23 (and custom ones added the clonal_complex) Adding the option for --noclean causes no errors to be generated.

Traceback (most recent call last):
  File "/tools/linuxbrew/opt/python3/lib/python3.6/site-packages/ariba/clusters.py", line 546, in run
    self._run()
  File "/tools/linuxbrew/opt/python3/lib/python3.6/site-packages/ariba/clusters.py", line 579, in _run
    raise Error('At least one cluster failed! Stopping...')
ariba.clusters.Error: At least one cluster failed! Stopping...
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/tools/linuxbrew/bin/ariba", line 292, in <module>
    args.func(args)
  File "/tools/linuxbrew/opt/python3/lib/python3.6/site-packages/ariba/tasks/run.py", line 63, in run
    c.run()
  File "/tools/linuxbrew/opt/python3/lib/python3.6/site-packages/ariba/clusters.py", line 549, in run
    raise Error('Something went wrong during ariba run. Cannot continue. Error was:\n' + str(err))
ariba.clusters.Error: Something went wrong during ariba run. Cannot continue. Error was:
At least one cluster failed! Stopping...
@andrewjpage
Copy link
Member

Could you send us the output of running

ariba test output

@andrewjpage andrewjpage added the bug label Feb 1, 2018
@kimleeng
Copy link
Author

kimleeng commented Feb 5, 2018 •
edited
Loading

Yep no problem

Running ARIBA on test data...

|=============================================================================|
| Preparing input data |
|=============================================================================|

Made output directory output. Copying test data files into it:
copied ref_seqs.fa
copied metadata.tsv
copied reads_1.fq
copied reads_2.fq

|=============================================================================|
| Try running ariba prepareref |
|=============================================================================|

Running ariba prepareref with:
/tools/linuxbrew/bin/ariba prepareref --verbose -f ref_seqs.fa -m metadata.tsv --threads 1 PREPAREREF

ARIBA version: 2.10.3

External dependencies:
bowtie2 2.3.4 /tools/linuxbrew/bin/bowtie2
cdhit 4.6 /tools/linuxbrew/bin/cd-hit-est
nucmer 3.1 /tools/linuxbrew/bin/nucmer

External dependencies OK: True

Python version:
3.6.3 (default, Oct 16 2017, 11:11:16)
[GCC 5.4.0]

Python packages:
ariba 2.10.3 /tools/linuxbrew/opt/python3/lib/python3.6/site-packages/ariba/init.py
bs4 4.6.0 /tools/linuxbrew/opt/python3/lib/python3.6/site-packages/bs4/init.py
dendropy 4.3.0 /tools/linuxbrew/opt/python3/lib/python3.6/site-packages/dendropy/init.py
pyfastaq 3.15.0 /tools/linuxbrew/opt/python3/lib/python3.6/site-packages/pyfastaq/init.py
pymummer 0.10.3 /tools/linuxbrew/opt/python3/lib/python3.6/site-packages/pymummer/init.py
pysam 0.13 /tools/linuxbrew/opt/python3/lib/python3.6/site-packages/pysam/init.py

Python packages OK: True

Everything looks OK: True

Loading and checking input data

Running cdhit
syscall: /tools/linuxbrew/bin/cd-hit-est -i /home/ssi.ad/kimn/test/output/PREPAREREF/02.cdhit.noncoding.fa -o /home/ssi.ad/kimn/test/output/tmp.run_cd-hit.6wq64xsk/cdhit -c 0.9 -T 1 -s 0.0 -d 0 -bak 1
syscall: /tools/linuxbrew/bin/cd-hit-est -i /home/ssi.ad/kimn/test/output/PREPAREREF/02.cdhit.noncoding.varonly.fa -o /home/ssi.ad/kimn/test/output/tmp.run_cd-hit.kjp2v3mv/cdhit -c 0.9 -T 1 -s 0.0 -d 0 -bak 1
syscall: /tools/linuxbrew/bin/cd-hit-est -i /home/ssi.ad/kimn/test/output/PREPAREREF/02.cdhit.gene.fa -o /home/ssi.ad/kimn/test/output/tmp.run_cd-hit.2c6e4rl_/cdhit -c 0.9 -T 1 -s 0.0 -d 0 -bak 1
syscall: /tools/linuxbrew/bin/cd-hit-est -i /home/ssi.ad/kimn/test/output/PREPAREREF/02.cdhit.gene.varonly.fa -o /home/ssi.ad/kimn/test/output/tmp.run_cd-hit.lt7zykj8/cdhit -c 0.9 -T 1 -s 0.0 -d 0 -bak 1

Writing clusters to file. 10 in total

ariba prepareref finished OK

|=============================================================================|
| Try running ariba run |
|=============================================================================|

Running ARIBA with:
/tools/linuxbrew/bin/ariba run --verbose --threads 1 PREPAREREF reads_1.fq reads_2.fq OUT

Found reads file: reads_1.fq
Found reads file: reads_2.fq
ARIBA version: 2.10.3

External dependencies:
bowtie2 2.3.4 /tools/linuxbrew/bin/bowtie2
cdhit 4.6 /tools/linuxbrew/bin/cd-hit-est
nucmer 3.1 /tools/linuxbrew/bin/nucmer

External dependencies OK: True

Python version:
3.6.3 (default, Oct 16 2017, 11:11:16)
[GCC 5.4.0]

Python packages:
ariba 2.10.3 /tools/linuxbrew/opt/python3/lib/python3.6/site-packages/ariba/init.py
bs4 4.6.0 /tools/linuxbrew/opt/python3/lib/python3.6/site-packages/bs4/init.py
dendropy 4.3.0 /tools/linuxbrew/opt/python3/lib/python3.6/site-packages/dendropy/init.py
pyfastaq 3.15.0 /tools/linuxbrew/opt/python3/lib/python3.6/site-packages/pyfastaq/init.py
pymummer 0.10.3 /tools/linuxbrew/opt/python3/lib/python3.6/site-packages/pymummer/init.py
pysam 0.13 /tools/linuxbrew/opt/python3/lib/python3.6/site-packages/pysam/init.py

Python packages OK: True

Everything looks OK: True
Temporary directory: /home/ssi.ad/kimn/test/output/OUT/ariba.tmp.90om8xqp
______________________ Mapping reads to clustered genes _______________________
Finished mapping

_____________________________ Generating clusters _____________________________
syscall: sort -k1,1 -k 2,2n /home/ssi.ad/kimn/test/output/OUT/minimap.reads > /home/ssi.ad/kimn/test/output/OUT/read_store
Compressing file /home/ssi.ad/kimn/test/output/OUT/read_store
Found 51 proper read pairs from minimap
Total clusters to perform local assemblies: 9

Insert size information from reads mapped to reference genes:
Insert size: 185.0
Insert sspace sd: 0.27
Max insert: 220.55

___________________________ Assembling each cluster ___________________________
Will run 1 cluster(s) in parallel
Not constructing cluster cluster because it only has 2 reads (1 of 9)
Constructing cluster cluster_1 (2 of 9)
Constructing cluster cluster_2 (3 of 9)
Constructing cluster cluster_3 (4 of 9)
Constructing cluster cluster_4 (5 of 9)
Constructing cluster gene (6 of 9)
Constructing cluster gene_1 (7 of 9)
Constructing cluster noncoding (8 of 9)
Constructing cluster noncoding_1 (9 of 9)
Start running cluster cluster_1 in directory /home/ssi.ad/kimn/test/output/OUT/ariba.tmp.90om8xqp/cluster_1
Finished running cluster cluster_1 in directory /home/ssi.ad/kimn/test/output/OUT/ariba.tmp.90om8xqp/cluster_1
Deleting cluster dir /home/ssi.ad/kimn/test/output/OUT/ariba.tmp.90om8xqp/cluster_1
Finished assembling clusters

Traceback (most recent call last):
File "/tools/linuxbrew/opt/python3/lib/python3.6/site-packages/ariba/clusters.py", line 546, in run
self._run()
File "/tools/linuxbrew/opt/python3/lib/python3.6/site-packages/ariba/clusters.py", line 579, in _run
raise Error('At least one cluster failed! Stopping...')
ariba.clusters.Error: At least one cluster failed! Stopping...

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/tools/linuxbrew/bin/ariba", line 292, in
args.func(args)
File "/tools/linuxbrew/opt/python3/lib/python3.6/site-packages/ariba/tasks/run.py", line 63, in run
c.run()
File "/tools/linuxbrew/opt/python3/lib/python3.6/site-packages/ariba/clusters.py", line 549, in run
raise Error('Something went wrong during ariba run. Cannot continue. Error was:\n' + str(err))
ariba.clusters.Error: Something went wrong during ariba run. Cannot continue. Error was:
At least one cluster failed! Stopping...

Something went wrong. See above for error message(s). Return code was 1

@kimleeng
Copy link
Author

kimleeng commented Feb 7, 2018

As a minor update I believe this could be the exact same issue as #203

@dswan
Copy link

dswan commented Aug 10, 2018

Hitting this issue too. Did wonder whether it was due to the precarious set of packages I had installed, but in a clean conda python3.6 environment. Also getting this when building from github, or installation through brew:

From the clean conda install:

Running ARIBA on test data...
-------------------------------------------------------------------------------
|=============================================================================|
|                            Preparing input data                             |
|=============================================================================|
-------------------------------------------------------------------------------
Made output directory output. Copying test data files into it:
    copied ref_seqs.fa
    copied metadata.tsv
    copied reads_1.fq
    copied reads_2.fq
-------------------------------------------------------------------------------
|=============================================================================|
|                        Try running ariba prepareref                         |
|=============================================================================|
-------------------------------------------------------------------------------

Running ariba prepareref with:
/home/ubuntu/miniconda3/envs/py37/bin/ariba prepareref --verbose -f ref_seqs.fa -m metadata.tsv --threads 1 PREPAREREF

ARIBA version: 2.12.1

External dependencies:
bowtie2 2.3.4.2 /home/ubuntu/miniconda3/envs/py37/bin/bowtie2
cdhit   4.7     /home/ubuntu/miniconda3/envs/py37/bin/cd-hit-est
nucmer  3.1     /home/ubuntu/miniconda3/envs/py37/bin/nucmer

External dependencies OK: True

Python version:
3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:53:17)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

Python packages:
ariba   2.12.1  /home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/ariba/__init__.py
bs4     4.6.1   /home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/bs4/__init__.py
dendropy        4.4.0   /home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/dendropy/__init__.py
pyfastaq        3.17.0  /home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/pyfastaq/__init__.py
pymummer        0.10.3  /home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/pymummer/__init__.py
pysam   0.14.1  /home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/pysam/__init__.py

Python packages OK: True

Everything looks OK: True

Loading and checking input data

Running cdhit
syscall: /home/ubuntu/miniconda3/envs/py37/bin/cd-hit-est -i /data/aribaMLST/output/PREPAREREF/02.cdhit.noncoding.fa -o /data/aribaMLST/output/tmp.run_cd-hit.z_q1f1sm/cdhit -c 0.9 -T 1 -s 0.0 -d 0 -bak 1
syscall: /home/ubuntu/miniconda3/envs/py37/bin/cd-hit-est -i /data/aribaMLST/output/PREPAREREF/02.cdhit.noncoding.varonly.fa -o /data/aribaMLST/output/tmp.run_cd-hit.s4zphih3/cdhit -c 0.9 -T 1 -s 0.0 -d 0 -bak 1
syscall: /home/ubuntu/miniconda3/envs/py37/bin/cd-hit-est -i /data/aribaMLST/output/PREPAREREF/02.cdhit.gene.fa -o /data/aribaMLST/output/tmp.run_cd-hit.0f_toxxx/cdhit -c 0.9 -T 1 -s 0.0 -d 0 -bak 1
syscall: /home/ubuntu/miniconda3/envs/py37/bin/cd-hit-est -i /data/aribaMLST/output/PREPAREREF/02.cdhit.gene.varonly.fa -o /data/aribaMLST/output/tmp.run_cd-hit.ppokl2o1/cdhit -c 0.9 -T 1 -s 0.0 -d 0 -bak 1

Writing clusters to file. 10 in total

ariba prepareref finished OK
-------------------------------------------------------------------------------
|=============================================================================|
|                            Try running ariba run                            |
|=============================================================================|
-------------------------------------------------------------------------------

Running ARIBA with:
/home/ubuntu/miniconda3/envs/py37/bin/ariba run --verbose --threads 1 PREPAREREF reads_1.fq reads_2.fq OUT

Found reads file: reads_1.fq
Found reads file: reads_2.fq
ARIBA version: 2.12.1

External dependencies:
bowtie2 2.3.4.2 /home/ubuntu/miniconda3/envs/py37/bin/bowtie2
cdhit   4.7     /home/ubuntu/miniconda3/envs/py37/bin/cd-hit-est
nucmer  3.1     /home/ubuntu/miniconda3/envs/py37/bin/nucmer

External dependencies OK: True

Python version:
3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:53:17)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

Python packages:
ariba   2.12.1  /home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/ariba/__init__.py
bs4     4.6.1   /home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/bs4/__init__.py
dendropy        4.4.0   /home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/dendropy/__init__.py
pyfastaq        3.17.0  /home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/pyfastaq/__init__.py
pymummer        0.10.3  /home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/pymummer/__init__.py
pysam   0.14.1  /home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/pysam/__init__.py

Python packages OK: True

Everything looks OK: True
Temporary directory: /data/tmp/ariba.tmp.cd_6654c
______________________ Mapping reads to clustered genes _______________________
Finished mapping

_____________________________ Generating clusters _____________________________
syscall: sort -k1,1 -k 2,2n /data/aribaMLST/output/OUT/minimap.reads > /data/aribaMLST/output/OUT/read_store
Compressing file /data/aribaMLST/output/OUT/read_store
Found 51 proper read pairs from minimap
Total clusters to perform local assemblies: 9

Insert size information from reads mapped to reference genes:
Insert size:    185.0
Insert sspace sd:       0.27
Max insert:     220.55

___________________________ Assembling each cluster ___________________________
Will run 1 cluster(s) in parallel
Not constructing cluster cluster because it only has 2 reads (1 of 9)
Constructing cluster cluster_1 (2 of 9)
Constructing cluster cluster_2 (3 of 9)
Constructing cluster cluster_3 (4 of 9)
Constructing cluster cluster_4 (5 of 9)
Constructing cluster gene (6 of 9)
Constructing cluster gene_1 (7 of 9)
Constructing cluster noncoding (8 of 9)
Constructing cluster noncoding_1 (9 of 9)
Start running cluster cluster_1 in directory /data/tmp/ariba.tmp.cd_6654c/cluster_1
cluster_1 detected 1 threads available to it
Failed cluster: cluster_1
Finished running cluster cluster_1 in directory /data/tmp/ariba.tmp.cd_6654c/cluster_1
Deleting cluster dir /data/tmp/ariba.tmp.cd_6654c/cluster_1
Other clusters failed. Will not start cluster cluster_2
Other clusters failed. Will not start cluster cluster_3
Other clusters failed. Will not start cluster cluster_4
Other clusters failed. Will not start cluster gene
Other clusters failed. Will not start cluster gene_1
Other clusters failed. Will not start cluster noncoding
Other clusters failed. Will not start cluster noncoding_1
Final value of remaining_clusters counter: Value('l', 7)
Finished assembling clusters

Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/ariba/clusters.py", line 602, in run
    self._run()
  File "/home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/ariba/clusters.py", line 636, in _run
    raise Error('At least one cluster failed! Stopping...')
ariba.clusters.Error: At least one cluster failed! Stopping...

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/py37/bin/ariba", line 298, in <module>
    args.func(args)
  File "/home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/ariba/tasks/run.py", line 65, in run
    c.run()
  File "/home/ubuntu/miniconda3/envs/py37/lib/python3.6/site-packages/ariba/clusters.py", line 605, in run
    raise Error('Something went wrong during ariba run. Cannot continue. Error was:\n' + str(err))
ariba.clusters.Error: Something went wrong during ariba run. Cannot continue. Error was:
At least one cluster failed! Stopping...

Something went wrong. See above for error message(s). Return code was 1

@martinghunt
Copy link
Collaborator

I can't reproduce this bug, but could you confirm if you're using NFS? If so, maybe try on a different filesystem to confirm if it's only happening on NFS.

I don't really have time to fix this in a nice way, so may just go for the ugly solution of replacing every shutil.rmtree with a call to rm -r.

@dswan
Copy link

dswan commented Aug 10, 2018 •
edited
Loading

@martinghunt no, I saw the NFS ticket, but this is running on an EBS volume on an EC2 instance. On the root fs which is ext4. Running with --noclean does not help. I had a tmpdir specified to another ext4 partition above, but the behaviour was reproducible with tmpdir on the same partition

@martinghunt martinghunt mentioned this issue Aug 10, 2018
Merged
@martinghunt
Copy link
Collaborator

I just put rm -fr everywhere instead of shutil.rmtree, on tip of master branch. @dswan would you mind giving it a go please and see if it works for you?

@dswan
Copy link

dswan commented Aug 10, 2018

@martinghunt that does not fix my issue, however I've just run the same install procedure on an Ubuntu VM - effectively:

conda create -n py36 python=3.6 anaconda
source activate py36
git clone https://github.com/sanger-pathogens/ariba.git
cd ariba/
python setup.py test
python setup.py install
ariba test output

So the virtual env's should be similar, there were no 'historical' ariba install components kicking around. It fails on EC2, and works fine locally. So there's some environmental issue here I can't pin down. I will see if I can dig up any more reasoning behind this over the next few days if it is helpful.

@dswan
Copy link

dswan commented Aug 13, 2018 •
edited
Loading

@martinghunt I'm sorry this was a false alarm. The only difference between the environments was really available memory. I was on a t2.micro (1GB) image for installation and testing but bumping to a t2.small (2GB) passes the tests fine. If I'd seen the responses to #186 I might have twigged this earlier.

@kpepper
Copy link
Member

kpepper commented Mar 21, 2019

Closing this now as the issue is resolved.

@kpepper kpepper closed this as completed Mar 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@andrewjpage @dswan @martinghunt @kimleeng @kpepper