Parameter Sets

Read mappers usually have a large number of command-line parameters, that can be used to fine-tune algorithms and influence results. However, the effect of changes in the values of these parameters are hard to understand.

Mapper parameter allow simultaneously benchmarking the performance of a mapper for multiple parameter combinations. This can help to get a better understanding of the effect of changing certain parameter values, or to optimize mappers for accuracy and speed.

A parameter set essentially consists of:

• A list of parameters that should be varied
• For each parameter, a list of values that should be tested

##Adding existing Parameter Sets to Teaser Individual or collections of parameter sets should be stored in YAML files with names matching the pattern parameter_<name>.yaml, <name> being a short descriptive name. In order to add a parameter set file to Teaser, simply copy the file into the setups directory.

##Using Parameter Sets: Web Interface See Web Interface for general information.

To test individual parameter sets using the Teaser web interface, simply choose them from the dropdown list in the "Evaluation" step. The web interface must be restarted in order for newly added parameter sets to appear.

##Using Parameter Sets: Command-Line See Command Line for general information.

./teaser.py my_benchmark.yaml -lp will list the available parameter sets.

./teaser.py my_benchmark.yaml -p bowtie2_presets,bowtie2_exhaustive will test Bowtie 2 using two parameter sets for the data sets defined in my_benchmark.yaml.

##Using Parameter Sets: In Benchmark Configuration The parameter sets that should be tested can alternatively be chosen in the benchmark configuration, using the test_parameters field. For example, in case of the Bowtie 2 example parameter sets, add the following lines to your benchmark configuration file:

test_parameters:
  - bowtie2_presets
  - bowtie2_exhaustive

##Creating new Parameter Sets Parameter sets are defined as subsections of the parameters section in configuration files. Two ways of defining a parameter set exist: Either a flat list of values, or a list of values for single parameters, which is used to enumerate all possible combinations. The example below shows the definition of two parameter sets for the Bowtie 2 mapper.

The example ngm_sensitivity on the bottom shows how the range of values for a parameter can be defined in terms of min, max and step.

In order for your parameter set to be automatically loaded, it should be named according to the parameter_x.yaml pattern and be placed in the Teaser/setups directory. We encourage users to share their personal parameter sets, which we will then make publicly available for others to download.

#In this section, additional parameter sets are defined.
parameters:
   bowtie2_presets: #Internal ID of the parameter set
      title: Presets #Displayed name of the parameter set
      mapper: bowtie2 #Internal ID of the mapper this parameter set is for.

      #In the 'define' subsection of a parameter set, Teaser expects a flat list of parameters which are then tested individually. (in this case 8 combinations)
      define: 
        - "--very-fast"
        - "--fast"
        - "--sensitive"
        - "--very-sensitive"
        - "--very-fast-local"
        - "--fast-local"
        - "--sensitive-local"
        - "--very-sensitive-local"

   bowtie2_exhaustive:
      title: Exhaustive
      mapper: bowtie2

      #In the 'generate' subsection of a parameter set, Teaser expects a list of parameters (in this case -D, -R and -L) and
      #for each parameter again a list of values that should be tested.
      #In the end, Teaser will enumerate and evaluate the mapper for all possible combinations of parameter values. (in this case 3*3*3=27 combinations)
      #ie.: -D 5 -R 1 -L 15, -D 5 -R 1 -L 30, etc.
      generate: 
        -
         - "-D 5"
         - "-D 15"
         - "-D 30"

        -
         - "-R 1"
         - "-R 5"
         - "-R 20"

        -
         - "-L 15"
         - "-L 20"
         - "-L 30"

    ngm_sensitivity:
       title: NGM Sensitivity Range
       mapper: ngm
       define:
         #Range definition: ["Parameter", min, max, step]
         - ["-s", 0.1, 0.9, 0.1]