Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: remove distinction between poolers and samplers in experimentalist.md #611

Merged
merged 1 commit into from
Jun 5, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 7 additions & 63 deletions docs/contribute/modules/experimentalist.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,6 @@ by an experiment runner. Experimentalists are generally implemented as functions

![Experimentalist Module](../../img/experimentalist.png)

Experimentalists can be implemented as *poolers* or as *samplers*.
- **Poolers** return a pool of candidate experimental conditions, which can be passed to a sampler that selects
a subset of conditions from the pool to be used in the next experiment.
- **Samplers** directly return a subset of experimental conditions from a pool of candidate experimental conditions that already exist.

## Repository Setup

We recommend using the [cookiecutter template](https://github.com/AutoResearch/autora-template-cookiecutter) to set up
Expand All @@ -22,80 +17,30 @@ a repository for your experimentalist. Alternatively, you can use the
cookiecutter https://github.com/AutoResearch/autora-template-cookiecutter
```

Make sure to select the `experimentalist` option when prompted. You may also select whether you want to implement an experimentalist as a sampler, pooler, or custom function. You can skip all other prompts pertaining to other modules
Make sure to select the `experimentalist` option when prompted. You can skip all other prompts pertaining to other modules
(e.g., experiment runners) by pressing enter.

## Implementation

Irrespective of whether you are implementing a pooler or a sampler,
you should implement a function that returns a set of experimental conditions. This set may be
For an experimentalist, you should implement a function that returns a set of experimental conditions. This set may be
a numpy array, iterator variable or other data format.

!!! hint
We generally **recommend using 2-dimensional numpy arrays as outputs** in which
each row represents a set of experimental conditions. The columns of the array correspond to the independent variables.

### Implementing Poolers

Once you've created your repository, you can implement your experimentalist pooler by editing the `init.py` file in
``src/autora/experimentalist/pooler/name_of_your_experimentalist/``.
Once you've created your repository, you can implement your experimentalist by editing the `init.py` file in
``src/autora/experimentalist/name_of_your_experimentalist/``.
You may also add additional files to this directory if needed.
It is important that the `init.py` file contains a function called `name_of_your_experimentalist`
which returns a pool of experimental conditions (e.g., as an iterator object or numpy array).

The following example ``init.py`` illustrates the implementation of a simple experimentalist pooler
that generates a grid of samples within the specified bounds of each independent variable (IV):

```python

"""
Example Experimentalist Pooler
"""

from itertools import product
from typing import List
from autora.variable import IV


def grid_pool(ivs: List[IV]):
"""
Creates exhaustive pool from discrete values using a Cartesian product of sets

Arguments:
ivs {List[IV]}: List of independent variables
which returns a set of experimental conditions (e.g., as a numpy array).

Returns:
pool: An iterator over all possible combinations of IV values
"""

l_iv_values = []
for iv in ivs:
assert iv.allowed_values is not None, (
f"gridsearch_pool only supports independent variables with discrete allowed values, "
f"but allowed_values is None on {iv=} "
)
l_iv_values.append(iv.allowed_values)

# Return Cartesian product of all IV values
return product(*l_iv_values)


```

### Implementing Samplers

Once you've created your repository, you can implement your experimentalist sampler by editing the `init.py` file in
``src/autora/experimentalist/sampler/name_of_your_experimentalist/``.
You may also add additional files to this directory if needed.
It is important that the `init.py` file contains a function called `name_of_your_experimentalist`
which returns a set of experimental conditions (e.g., as a numpy array) given a pool of candidate experimental conditions.

The following example ``init.py`` illustrates the implementation of a simple experimentalist sampler
The following example ``init.py`` illustrates the implementation of a simple experimentalist
that uniformly samples without replacement from a pool of candidate conditions.

```python
"""
Example Experimentalist Sampler
Example Experimentalist
"""

import random
Expand All @@ -120,7 +65,6 @@ random_sample(conditions: Union[Iterable, Sequence], n: int = 1):
return samples
```


## Next Steps: Testing, Documentation, Publishing

For more information on how to test, document, and publish your experimentalist, please refer to the
Expand Down