ValueError in test_fusion

[hackdna]

Version info

• bcbio version (bcbio_nextgen.py --version): 1.2.3
• OS name and version: Ubuntu 18.04

To Reproduce

• Exact bcbio command you have used:
  pytest tests/integration/test_S3_pipelines.py::test_fusion
• Your sample configuration file: run_info-fusion_S3.yaml

Observed behavior

FAILED tests/integration/test_S3_pipelines.py::test_fusion - subprocess.CalledProcessError: Command '['bcbio_nextgen.py', '/tmp/bcbio/test_automated_output/bcbio_system.yaml', '/vagrant/tests/data/auto...

Expected behavior
No failure

Log files

_______________________________________________________________________________________________ test_fusion ________________________________________________________________________________________________

install_test_files = None, data_dir = '/vagrant/tests/data/automated'

    @pytest.mark.S3
    @pytest.mark.install_required
    def test_fusion(install_test_files, data_dir):
        """Run an RNA-seq analysis and test fusion genes, with human-mouse
        disambiguation.
        Requires minikraken database.
        """
        with make_workdir() as workdir:
            cl = ["bcbio_nextgen.py",
                  get_post_process_yaml(data_dir, workdir),
                  os.path.join(data_dir, os.pardir, "test_fusion"),
                  os.path.join(data_dir, "run_info-fusion_S3.yaml")]
>           subprocess.check_call(cl)

tests/integration/test_S3_pipelines.py:21:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['bcbio_nextgen.py', '/tmp/bcbio/test_automated_output/bcbio_system.yaml', '/vagrant/tests/data/automated/../test_fusion', '/vagrant/tests/data/automated/run_info-fusion_S3.yaml'],)
kwargs = {}, retcode = 1
cmd = ['bcbio_nextgen.py', '/tmp/bcbio/test_automated_output/bcbio_system.yaml', '/vagrant/tests/data/automated/../test_fusion', '/vagrant/tests/data/automated/run_info-fusion_S3.yaml']

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.

        The arguments are the same as for the call function.  Example:

        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['bcbio_nextgen.py', '/tmp/bcbio/test_automated_output/bcbio_system.yaml', '/vagrant/tests/data/automated/../test_fusion', '/vagrant/tests/data/automated/run_info-fusion_S3.yaml']' returned non-zero exit status 1.

/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/subprocess.py:311: CalledProcessError
------------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------------
Running bcbio version: 1.2.3
global config: /tmp/bcbio/test_automated_output/bcbio_system.yaml
flowcell directory: /vagrant/tests/data/test_fusion
run info config: /vagrant/tests/data/automated/run_info-fusion_S3.yaml
------------------------------------------------------------------------------------------- Captured stderr call -------------------------------------------------------------------------------------------
[2020-05-13T18:47Z] System YAML configuration: /tmp/bcbio/test_automated_output/bcbio_system.yaml.
[2020-05-13T18:47Z] Locale set to C.UTF-8.
[2020-05-13T18:47Z] Resource requests: picard; memory: 3.00; cores: 16
[2020-05-13T18:47Z] Configuring 1 jobs to run, using 1 cores each with 3.00g of memory reserved for each job
[2020-05-13T18:47Z] Timing: organize samples
[2020-05-13T18:47Z] multiprocessing: organize_samples
[2020-05-13T18:47Z] Using input YAML configuration: /vagrant/tests/data/automated/run_info-fusion_S3.yaml
[2020-05-13T18:47Z] Checking sample YAML configuration: /vagrant/tests/data/automated/run_info-fusion_S3.yaml
/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/gffutils/interface.py:214: UserWarning: It appears that this database has not had the ANALYZE sqlite3 command run on it. Doing so can dramatically speed up queries, and is done by default for databases created with gffutils >0.8.7.1 (this database was created with version 0.8.2) Consider calling the analyze() method of this object.
  "method of this object." % self.version)
[2020-05-13T18:47Z] Retreiving program versions from /home/vagrant/local/share/bcbio/manifest/python-packages.yaml.
[2020-05-13T18:47Z] Retreiving program versions from /home/vagrant/local/share/bcbio/manifest/r-packages.yaml.
[2020-05-13T18:47Z] Testing minimum versions of installed programs
[2020-05-13T18:47Z] multiprocessing: prepare_sample
[2020-05-13T18:47Z] Preparing Test1
[2020-05-13T18:47Z] Converting /vagrant/tests/data/test_fusion/1_1.fq.gz to Sanger format.
[2020-05-13T18:47Z] Converting /vagrant/tests/data/test_fusion/1_2.fq.gz to Sanger format.
[2020-05-13T18:47Z] Resource requests: picard, samtools, star; memory: 3.00, 3.00, 3.00; cores: 16, 16, 16
[2020-05-13T18:47Z] Configuring 1 jobs to run, using 1 cores each with 3.00g of memory reserved for each job
[2020-05-13T18:47Z] Timing: alignment
[2020-05-13T18:47Z] multiprocessing: disambiguate_split
[2020-05-13T18:47Z] multiprocessing: process_alignment
[2020-05-13T18:47Z] Aligning lane Test1 with star aligner
[2020-05-13T18:47Z] Running 1st pass of STAR aligner on /tmp/bcbio/test_automated_output/fastq_convert/1_1.groom.fq.gz and /vagrant/tests/data/genomes/hg19/star/
[2020-05-13T18:48Z] Running 2nd pass of STAR aligner on /tmp/bcbio/test_automated_output/fastq_convert/1_1.groom.fq.gz and /vagrant/tests/data/genomes/hg19/star/
[2020-05-13T18:48Z] Index BAM file: Test1.bam
[2020-05-13T18:48Z] Aligning lane Test1 with star aligner
Traceback (most recent call last):
  File "/home/vagrant/local/share/bcbio/anaconda/bin/bcbio_nextgen.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/vagrant/scripts/bcbio_nextgen.py", line 245, in <module>
    main(**kwargs)
  File "/vagrant/scripts/bcbio_nextgen.py", line 46, in main
    run_main(**kwargs)
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 50, in run_main
    fc_dir, run_info_yaml)
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 91, in _run_toplevel
    for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 246, in rnaseqpipeline
    samples = run_parallel("process_alignment", samples)
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
    return run_multicore(fn, items, config, parallel=parallel)
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
    for data in joblib.Parallel(parallel["num_jobs"], batch_size=1, backend="multiprocessing")(joblib.delayed(fn)(*x) for x in items):
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 1007, in __call__
    while self.dispatch_one_batch(iterator):
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 835, in dispatch_one_batch
    self._dispatch(tasks)
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 754, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 209, in apply_async
    result = ImmediateResult(func)
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 590, in __init__
    self.results = batch()
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 256, in __call__
    for func, args, kwargs in self.items]
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 256, in <listcomp>
    for func, args, kwargs in self.items]
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/bcbio/utils.py", line 55, in wrapper
    return f(*args, **kwargs)
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/multitasks.py", line 124, in process_alignment
    return sample.process_alignment(*args)
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/sample.py", line 128, in process_alignment
    data = align_to_sort_bam(fastq1, fastq2, aligner, data)
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/alignment.py", line 80, in align_to_sort_bam
    aligner_index = get_aligner_index(aligner, data)
  File "/home/vagrant/local/share/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/alignment.py", line 141, in get_aligner_index
    (aligner, data["reference"]))
ValueError: Did not find reference indices for aligner star in genome: {'fasta': {'base': '/vagrant/tests/data/genomes/mm9/seq/mm9.fa'}, 'star': {}, 'rtg': {'base': '/vagrant/tests/data/genomes/mm9/rtg/mm9.sdf/mainIndex', 'indexes': []}, 'twobit': '/vagrant/tests/data/genomes/mm9/ucsc/mm9.2bit', 'genome_context': [], 'viral': None, 'versions': None}

Additional context
Discovered in the context of working on #3176