Parameter Sets

Read mappers usually have a large number of command-line parameters, that can be used to fine-tune algorithms and influence results. However, the effect of changes in the values of these parameters are often hard to understand.

Parameter sets in Teaser allow simultaneously benchmarking the performance of a mapper for multiple parameter combinations. This can help to get a better understanding of the effect of changing certain parameter values, or to optimize mappers for accuracy and speed.

A parameter set essentially consists of:

• A list of parameters that should be varied
• For each parameter, a list of values that should be tested

Adding existing Parameter Sets to Teaser

Individual or collections of parameter sets should be stored in YAML files with names matching the pattern parameter_<name>.yaml, <name> being a short descriptive name. In order to add a parameter set file to Teaser, simply copy the file into the setups directory.

Using Parameter Sets: Web Interface

See Web Browser Interface for general information.

To test individual parameter sets using the web interface, simply choose them from the dropdown list in the "Evaluation" step. The web interface must be restarted in order for newly added parameter sets to appear.

Using Parameter Sets: Command-Line

See Command Line for general information.

./teaser.py my_benchmark.yaml -lp will list the available parameter sets.

./teaser.py my_benchmark.yaml -p bowtie2_presets,bowtie2_exhaustive will test Bowtie 2 using two parameter sets for the data sets defined in my_benchmark.yaml.

Using Parameter Sets: In Benchmark Configuration

The parameter sets that should be tested can alternatively be chosen in the benchmark configuration, using the test_parameters field. For example, in case of the Bowtie 2 example parameter sets, add the following lines to your benchmark configuration file:

test_parameters:
  - bowtie2_presets
  - bowtie2_exhaustive

Creating new Parameter Sets

Parameter sets are defined as subsections of the parameters section in configuration files. Two ways of defining a parameter set exist.

Pre-defined combinations

Here, you supply the different combinations as a simple flat list of strings which are directly added to the command line arguments of the mapper. We encourage users to share their personal parameter sets, which we will then make publicly available for others to download. In order for your parameter set to be automatically loaded, it should be named according to the parameter_x.yaml pattern and be placed in the Teaser/setups directory.

# In this section, additional parameter sets are defined.
parameters:
   bowtie2_presets: #Internal ID of the parameter set
      title: Presets #Displayed name of the parameter set
      mapper: bowtie2 #Internal ID of the mapper this parameter set is for.

      #In the 'define' subsection of a parameter set, Teaser expects a flat list of parameters which are then tested individually. (in this case 8 combinations)
      define: 
        - "--very-fast"
        - "--fast"
        - "--sensitive"
        - "--very-sensitive"
        - "--very-fast-local"
        - "--fast-local"
        - "--sensitive-local"
        - "--very-sensitive-local"

Enumerating Combinations

Here supply a set of values each, for a number of parameters. Teaser will then test the mappers on all possible combinations of these parameters.

parameters:
   bowtie2_exhaustive:
      title: Exhaustive
      mapper: bowtie2

      #In the 'generate' subsection of a parameter set, Teaser expects a list of parameters (in this case -D, -R and -L) and
      #for each parameter again a list of values that should be tested.
      #In the end, Teaser will enumerate and evaluate the mapper for all possible combinations of parameter values. (in this case 3*3*3=27 combinations)
      #e.g.: -D 5 -R 1 -L 15, -D 5 -R 1 -L 30, etc.
      generate: 
        -
         - "-D 5"
         - "-D 15"
         - "-D 30"

        -
         - "-R 1"
         - "-R 5"
         - "-R 20"

        -
         - "-L 15"
         - "-L 20"
         - "-L 30"

Value Ranges

Instead of supplying single numbers, a value range for each parameter can be defined in terms of min, max and step.

parameters:
    ngm_sensitivity:
       title: NGM Sensitivity Range
       mapper: ngm
       define:
         #Range definition: ["Parameter", min, max, step]
         - ["-s", 0.1, 0.9, 0.1]