Parameter Sets

Read mappers usually have a large number of command-line parameters, that can be used to fine-tune algorithms and influence results. However, the effect of changes in the values of these parameters are often hard to understand.

Parameter sets in Teaser allow simultaneously benchmarking the performance of a mapper for multiple parameter combinations. This can help to get a better understanding of the effect of changing certain parameter values, or to optimize mappers for accuracy and speed.

A parameter set essentially consists of:

• A list of parameters that should be varied
• For each parameter, a list of values that should be tested

##Adding existing Parameter Sets to Teaser Individual or collections of parameter sets should be stored in YAML files with names matching the pattern parameter_<name>.yaml, <name> being a short descriptive name. In order to add a parameter set file to Teaser, simply copy the file into the setups directory.

##Using Parameter Sets: Web Interface See Web Browser Interface for general information.

To test individual parameter sets using the Teaser web interface, simply choose them from the dropdown list in the "Evaluation" step. The web interface must be restarted in order for newly added parameter sets to appear.

##Using Parameter Sets: Command-Line See Command Line for general information.

./teaser.py my_benchmark.yaml -lp will list the available parameter sets.

./teaser.py my_benchmark.yaml -p bowtie2_presets,bowtie2_exhaustive will test Bowtie 2 using two parameter sets for the data sets defined in my_benchmark.yaml.

##Using Parameter Sets: In Benchmark Configuration The parameter sets that should be tested can alternatively be chosen in the benchmark configuration, using the test_parameters field. For example, in case of the Bowtie 2 example parameter sets, add the following lines to your benchmark configuration file:

test_parameters:
  - bowtie2_presets
  - bowtie2_exhaustive

##Creating new Parameter Sets Parameter sets are defined as subsections of the parameters section in configuration files. Two ways of defining a parameter set exist.

###Pre-defined combinations

Here, you supply a simple flat list of strings which are separately added to the command line arguments of the mapper.

#In this section, additional parameter sets are defined.
parameters:
   bowtie2_presets: #Internal ID of the parameter set
      title: Presets #Displayed name of the parameter set
      mapper: bowtie2 #Internal ID of the mapper this parameter set is for.

      #In the 'define' subsection of a parameter set, Teaser expects a flat list of parameters which are then tested individually. (in this case 8 combinations)
      define: 
        - "--very-fast"
        - "--fast"
        - "--sensitive"
        - "--very-sensitive"
        - "--very-fast-local"
        - "--fast-local"
        - "--sensitive-local"
        - "--very-sensitive-local"

###Enumerating Combinations

Here, you supply a list of values each for a number of parameters. Teaser then enumerates all possible combinations of these parameters. 

####Value Ranges

Instead of supplying single numbers, a value range for a parameter can be defined in terms of `min`, `max` and `step`.

parameters: bowtie2_exhaustive: title: Exhaustive mapper: bowtie2

  #In the 'generate' subsection of a parameter set, Teaser expects a list of parameters (in this case -D, -R and -L) and
  #for each parameter again a list of values that should be tested.
  #In the end, Teaser will enumerate and evaluate the mapper for all possible combinations of parameter values. (in this case 3*3*3=27 combinations)
  #ie.: -D 5 -R 1 -L 15, -D 5 -R 1 -L 30, etc.
  generate: 
    -
     - "-D 5"
     - "-D 15"
     - "-D 30"

    -
     - "-R 1"
     - "-R 5"
     - "-R 20"

    -
     - "-L 15"
     - "-L 20"
     - "-L 30"

parameters:
    ngm_sensitivity:
       title: NGM Sensitivity Range
       mapper: ngm
       define:
         #Range definition: ["Parameter", min, max, step]
         - ["-s", 0.1, 0.9, 0.1]

We encourage users to share their personal parameter sets, which we will then make publicly available for others to download. In order for your parameter set to be automatically loaded, it should be named according to the parameter_x.yaml pattern and be placed in the Teaser/setups directory.