Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default batch_run to 1 CPU #1300

Merged
merged 2 commits into from
May 1, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 3 additions & 30 deletions docs/tutorials/intro_tutorial.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -840,7 +840,7 @@
"\n",
"* `number_processes`\n",
"<br/><br/>\n",
"Number of processors used to run the sweep in parallel. Optional. If not specified, defaults to use all the available processors.\n",
"If not specified, defaults to 1. Set it to `None` to use all the available processors.\n",
"<br/><br/>\n",
"Note: Multiprocessing does make debugging challenging. If your parameter sweeps are resulting in unexpected errors set `number_processes = 1`.\n",
"<br/><br/>\n",
Expand Down Expand Up @@ -892,7 +892,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"**Note for Windows OS users:** If you are running this tutorial in Jupyter, we recommend changing the the parameter that controls the number of processors you are using from `number_processes = None` (this uses all available processors) to `number_processes = 1.` More information on leveraging `batch_run`'s multiprocessing capability is below. "
"**Note for Windows OS users:** If you are running this tutorial in Jupyter, make sure that you set `number_processes = 1` (single process). If `number_processes` is greater than 1, it is less straightforward to set up. You can read [Mesa's collection of useful snippets](https://github.com/projectmesa/mesa/blob/main/docs/useful-snippets/snippets.rst), in 'Using multi-process `batch_run` on Windows' section for how to do it."
]
},
{
Expand All @@ -908,39 +908,12 @@
" parameters=params,\n",
" iterations=5,\n",
" max_steps=100,\n",
" number_processes=None,\n",
" number_processes=1,\n",
" data_collection_period=1,\n",
" display_progress=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you would still like to run your code in Jupyter your will still need to adjust the cell as noted above. Then you can**Note for Windows OS users:** You will have an issue with `batch_run` and `number_processes = None`. Your cell will show no progress, and in your terminal you will receive *AttributeError: Can't get attribute 'MoneyModel' on <module '__main__' (built-in)>*. One way to overcome this is to take your code outside of Jupyter and adjust the above code as follows. \n",
"\n",
"```\n",
"from multiprocessing import freeze_support\n",
"\n",
"params = {\"width\": 10, \"height\": 10, \"N\": range(10, 500, 10)}\n",
"\n",
"if __name__ == '__main__':\n",
" freeze_support()\n",
" results = batch_run(\n",
" MoneyModel,\n",
" parameters=params,\n",
" iterations=5,\n",
" max_steps=100,\n",
" number_processes=None,\n",
" data_collection_period=1,\n",
" display_progress=True,\n",
" )\n",
"```\n",
"\n",
"If you would still like to run your code in Jupyter you will need to adjust the cell as noted above. Then you can you can add the [nbmultitask library](https://nbviewer.org/github/micahscopes/nbmultitask/blob/39b6f31b047e8a51a0fcb5c93ae4572684f877ce/examples.ipynb) or look at this [stackoverflow](https://stackoverflow.com/questions/50937362/multiprocessing-on-python-3-jupyter). "
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down
38 changes: 4 additions & 34 deletions docs/tutorials/intro_tutorial.rst
Original file line number Diff line number Diff line change
Expand Up @@ -936,7 +936,8 @@ We call ``batch_run`` with the following arguments:
* ``number_processes``

Number of processors used to run the sweep in parallel. Optional.
If not specified, defaults to use all the available processors.
If not specified, defaults to 1. Set it to `None` to use all the available
processors.

Note: Multiprocessing does make debugging challenging. If your
parameter sweeps are resulting in unexpected errors set ``number_processes = 1``.
Expand Down Expand Up @@ -984,9 +985,7 @@ iteration).
.. code:: python

from mesa.batchrunner import batch_run
**Note for Windows OS users:** If you are running this tutorial in Jupyter, we recommend changing the the parameter
that controls the number of processors you are using from `number_processes = None` (this uses all available processors)
to `number_processes = 1`. More information on leveraging batch_run's multiprocessing capability is below.
**Note for Windows OS users:** If you are running this tutorial in Jupyter, make sure that you set `number_processes = 1` (single process). If `number_processes` is greater than 1, it is less straightforward to set up. You can read `Mesa's collection of useful snippets <https://github.com/projectmesa/mesa/blob/main/docs/useful-snippets/snippets.rst>`__, in "Using multi-process ```batch_run``` on Windows" section for how to do it.

.. code:: python

Expand All @@ -997,7 +996,7 @@ to `number_processes = 1`. More information on leveraging batch_run's multiproce
parameters=params,
iterations=5,
max_steps=100,
number_processes=None,
number_processes=1,
data_collection_period=1,
display_progress=True,
)
Expand All @@ -1007,35 +1006,6 @@ to `number_processes = 1`. More information on leveraging batch_run's multiproce

245it [00:25, 9.75it/s]

**Note for Windows OS users:** You will have an issue with `batch_run` and `number_processes = None`. Your cell will
show no progress, and in your terminal you will receive *AttributeError: Can't get attribute 'MoneyModel' on
<module '__main__' (built-in)>*. One way to overcome this is to take your code outside of Jupyter and adjust the above
code as follows.


.. code:: python

from multiprocessing import freeze_support

params = {"width": 10, "height": 10, "N": range(10, 500, 10)}

if __name__ == '__main__':
freeze_support()
results = batch_run(
MoneyModel,
parameters=params,
iterations=5,
max_steps=100,
number_processes=None,
data_collection_period=1,
display_progress=True,
)


If you would still like to run your code in Jupyter you will need to adjust the cell as noted above. Then you can
you can add the `nbmultitask library <(https://nbviewer.org/github/micahscopes/nbmultitask/blob/39b6f31b047e8a51a0fcb5c93ae4572684f877ce/examples.ipynb)>`__
or look at this `stackoverflow <https://stackoverflow.com/questions/50937362/multiprocessing-on-python-3-jupyter>`__.

To further analyze the return of the ``batch_run`` function, we convert
the list of dictionaries to a Pandas DataFrame and print its keys.

Expand Down
32 changes: 32 additions & 0 deletions docs/useful-snippets/snippets.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,35 @@ Sometimes you need to use ``numpy``'s ``random`` library, for example to get a P
self.random = np.random.default_rng(seed)

And just use `numpy`'s random as usual, e.g. ``self.random.poisson()``.

Using multi-process ```batch_run``` on Windows
-------------

You will have an issue with `batch_run` and `number_processes = None`. Your cell will
show no progress, and in your terminal you will receive *AttributeError: Can't get attribute 'MoneyModel' on
<module '__main__' (built-in)>*. One way to overcome this is to take your code outside of Jupyter and adjust the above
code as follows.


.. code:: python

from multiprocessing import freeze_support

params = {"width": 10, "height": 10, "N": range(10, 500, 10)}

if __name__ == '__main__':
freeze_support()
results = batch_run(
MoneyModel,
parameters=params,
iterations=5,
max_steps=100,
number_processes=None,
data_collection_period=1,
display_progress=True,
)


If you would still like to run your code in Jupyter you will need to adjust the cell as noted above. Then you can
you can add the `nbmultitask library <(https://nbviewer.org/github/micahscopes/nbmultitask/blob/39b6f31b047e8a51a0fcb5c93ae4572684f877ce/examples.ipynb)>`__
or look at this `stackoverflow <https://stackoverflow.com/questions/50937362/multiprocessing-on-python-3-jupyter>`__.
5 changes: 3 additions & 2 deletions mesa/batchrunner.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@
def batch_run(
model_cls: Type[Model],
parameters: Mapping[str, Union[Any, Iterable[Any]]],
number_processes: Optional[int] = None,
# We still retain the Optional[int] because users may set it to None (i.e. use all CPUs)
number_processes: Optional[int] = 1,
iterations: int = 1,
data_collection_period: int = -1,
max_steps: int = 1000,
Expand All @@ -50,7 +51,7 @@ def batch_run(
parameters : Mapping[str, Union[Any, Iterable[Any]]],
Dictionary with model parameters over which to run the model. You can either pass single values or iterables.
number_processes : int, optional
Number of processes used. Set to None (default) to use all available processors
Number of processes used, by default 1. Set this to None if you want to use all CPUs.
iterations : int, optional
Number of iterations for each parameter combination, by default 1
data_collection_period : int, optional
Expand Down
5 changes: 3 additions & 2 deletions tests/test_batch_run.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ def step(self):


def test_batch_run():
result = batch_run(MockModel, {})
result = batch_run(MockModel, {}, number_processes=2)
assert result == [
{
"RunId": 0,
Expand Down Expand Up @@ -125,11 +125,12 @@ def test_batch_run_with_params():
"variable_model_params": range(5),
"variable_agent_params": ["H", "E", "L", "L", "O"],
},
number_processes=2,
)


def test_batch_run_no_agent_reporters():
result = batch_run(MockModel, {"enable_agent_reporters": False})
result = batch_run(MockModel, {"enable_agent_reporters": False}, number_processes=2)
print(result)
assert result == [
{
Expand Down