Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed Cluster #186

Closed
GretchenWilson opened this issue Jul 20, 2017 · 7 comments
Closed

Failed Cluster #186

GretchenWilson opened this issue Jul 20, 2017 · 7 comments

Comments

@GretchenWilson
Copy link

Hello!

I'm hoping you can help me with some confusing error behavior. I am consistently receiving the following error on runs of ARIBA (version info listed below) and I'm out of ideas in looking for the source. My pipeline calls ARIBA on a dataset of 65 isolates, I receive this error on approx. 30% of ARIBA runs. This error does not persist with the same isolates at the same cluster if I rerun the isolate. Also, not the second error thrown during the handling of the original error.

Have you seen this issue before? Is this potentially an issue with my dependencies?

Thank you for your time!

Gretchen

Error Output

Stopping! Signal received: 28
Failed cluster: CRP
Finished running cluster CRP in directory /home/wilsonge/MyPrograms/vdh_amd_project/card.ariba.90%ID.2017-07-20/SRR1648160/RUN/ariba.tmp.a31cj4io/CRP
Deleting cluster dir /home/wilsonge/MyPrograms/vdh_amd_project/card.ariba.90%ID.2017-07-20/SRR1648160/RUN/ariba.tmp.a31cj4io/CRP
Other clusters failed. Will not start cluster Chlamydophila_psittaci_16S
Other clusters failed. Will not start cluster Enterobacter_aerogenes_omp36
Other clusters failed. Will not start cluster Escherichia_coli_EF_Tu
Other clusters failed. Will not start cluster Escherichia_coli_UhpT
Other clusters failed. Will not start cluster H_NS
Other clusters failed. Will not start cluster Helicobacter_pylori_16S
Other clusters failed. Will not start cluster Klebsiella_OmpK36_conferring
Other clusters failed. Will not start cluster MdtK
Other clusters failed. Will not start cluster Mycobacterium_abscessus_16S+
Other clusters failed. Will not start cluster Neisseria_-
Other clusters failed. Will not start cluster Pasteurella_multocida_16S
Other clusters failed. Will not start cluster Propionibacterium_acnes_16S
Other clusters failed. Will not start cluster TolC
Other clusters failed. Will not start cluster acrA
Other clusters failed. Will not start cluster acrA_1
Other clusters failed. Will not start cluster acrA_2
Other clusters failed. Will not start cluster acrB
Other clusters failed. Will not start cluster bacA
Other clusters failed. Will not start cluster cpxA
Other clusters failed. Will not start cluster emrB
Other clusters failed. Will not start cluster emrR
Other clusters failed. Will not start cluster fabI
Other clusters failed. Will not start cluster golS
Other clusters failed. Will not start cluster gyrA_6
Other clusters failed. Will not start cluster gyrA_7
Other clusters failed. Will not start cluster gyrB_1
Other clusters failed. Will not start cluster gyrB_2
Other clusters failed. Will not start cluster kdpE
Other clusters failed. Will not start cluster mdsA
Other clusters failed. Will not start cluster mdsB
Other clusters failed. Will not start cluster mdsC
Other clusters failed. Will not start cluster murA_2
Other clusters failed. Will not start cluster ompF
Other clusters failed. Will not start cluster parC_2
Other clusters failed. Will not start cluster parC_3
Other clusters failed. Will not start cluster parE_1
Other clusters failed. Will not start cluster parE_2
Other clusters failed. Will not start cluster ramR
Other clusters failed. Will not start cluster rpoB
Other clusters failed. Will not start cluster rrsB+
Other clusters failed. Will not start cluster sdiA
Other clusters failed. Will not start cluster soxR_1
Other clusters failed. Will not start cluster soxS
Finished assembling clusters

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/ariba/clusters.py", line 546, in run
    self._run()
  File "/usr/local/lib/python3.5/dist-packages/ariba/clusters.py", line 579, in _run
    raise Error('At least one cluster failed! Stopping...')
ariba.clusters.Error: At least one cluster failed! Stopping...

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/wilsonge/ariba-2.10.0/scripts/ariba", line 292, in <module>
    args.func(args)
  File "/usr/local/lib/python3.5/dist-packages/ariba/tasks/run.py", line 63, in run
    c.run()
  File "/usr/local/lib/python3.5/dist-packages/ariba/clusters.py", line 549, in run
    raise Error('Something went wrong during ariba run. Cannot continue. Error was:\n' + str(err))
ariba.clusters.Error: Something went wrong during ariba run. Cannot continue. Error was:
At least one cluster failed! Stopping...
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 67, in apport_excepthook
    binary = os.path.realpath(os.path.join(os.getcwd(), sys.argv[0]))
FileNotFoundError: [Errno 2] No such file or directory

Original exception was:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/ariba/clusters.py", line 546, in run
    self._run()
  File "/usr/local/lib/python3.5/dist-packages/ariba/clusters.py", line 579, in _run
    raise Error('At least one cluster failed! Stopping...')
ariba.clusters.Error: At least one cluster failed! Stopping...

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/wilsonge/ariba-2.10.0/scripts/ariba", line 292, in <module>
    args.func(args)
  File "/usr/local/lib/python3.5/dist-packages/ariba/tasks/run.py", line 63, in run
    c.run()
  File "/usr/local/lib/python3.5/dist-packages/ariba/clusters.py", line 549, in run
    raise Error('Something went wrong during ariba run. Cannot continue. Error was:\n' + str(err))
ariba.clusters.Error: Something went wrong during ariba run. Cannot continue. Error was:
At least one cluster failed! Stopping...

Version Info

ARIBA version: 2.10.0

External dependencies:
bowtie2	2.2.6	/usr/bin/bowtie2
cdhit	4.6	/usr/bin/cdhit-est
nucmer	3.1	/usr/bin/nucmer

External dependencies OK: True

Python version:
3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609]

Python packages:
ariba	2.10.0	/usr/local/lib/python3.5/dist-packages/ariba/__init__.py
bs4	4.4.1	/usr/lib/python3/dist-packages/bs4/__init__.py
dendropy	4.2.0	/usr/local/lib/python3.5/dist-packages/dendropy/__init__.py
pyfastaq	3.15.0	/usr/local/lib/python3.5/dist-packages/pyfastaq/__init__.py
pymummer	0.10.2	/usr/local/lib/python3.5/dist-packages/pymummer/__init__.py
pysam	0.11.2.2	/usr/local/lib/python3.5/dist-packages/pysam/__init__.py

Python packages OK: True

Everything looks OK: True
@martinghunt
Copy link
Collaborator

Are you using the --threads option? If so, try it with 1 thread, as using >1 can cause this.

@GretchenWilson
Copy link
Author

I'm using the default of 1 thread and cluster assembly starts with the following output.

___________________________ Assembling each cluster ___________________________
Will run 1 cluster(s) in parallel
Constructing cluster AAC_3__IV (1 of 66)
Constructing cluster AAC_6___I-_3 (2 of 66)
Constructing cluster APH_3___Ia (3 of 66)

My initial command is..

ariba run --assembled_threshold 0.90 --verbose card.ariba SRR1694090_2.fastq.gz SRR1694090_1.fastq.gz card.ariba.90%ID.2017-07-20/SRR1694090/RUN

@martinghunt
Copy link
Collaborator

How much ram do you have? I've only seen this happen with the threads option or on machines with small ram, like 1gb.

@GretchenWilson
Copy link
Author

I am running a virtual environment with 8.5G of ram. I often run multiple scripts simultaneously, but currently the processes combined are only using 4G of memory.

@martinghunt
Copy link
Collaborator

Can you try running one at a time to see if you still get the errors? As you're getting different clusters failing from the same sample, I think memory might be an issue, even though it doesn't look like it. I've had this happen under similar circumstances.

@GretchenWilson
Copy link
Author

It is a different cluster failing within the same sample from run to run, but eventually a run will complete for that sample. Currently, I am running RAxML bootstrapping in another terminal. So, I think you're right. I'll be sure to take a look at what inference round RAxML is in next time the errors occur in ARIBA.

Thank you for helping me narrow this down. Its good to know that this is potentially a problem with my own environment and that I can still rely on the ARIBA output from a completed run.

We've really enjoyed learning and using your tool!

Gretchen

@martinghunt
Copy link
Collaborator

Great, I'll close this issue for now.

@dswan dswan mentioned this issue Aug 13, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@martinghunt @GretchenWilson