Dask on SLURM ============= Deploy a Dask.distributed_ cluster on top of a cluster running a SLURM_ workload manager. Written under influence and with code borrowing from Dask-DRMAA_ project. .. _Dask.distributed: http://distributed.readthedocs.io/en/latest/ .. _SLURM: https://slurm.schedmd.com/ .. _Dask-DRMAA: https://github.com/dask/dask-drmaa/ Example ------- Launch cluster from Python code and do some simple calculation: .. code-block:: python from slurmified import Cluster slurm_kwargs = { 'mem-per-cpu': '100', 'time': '1-00:00:00' } cluster = Cluster(slurm_kwargs) cluster.start_workers(10) from distributed import Client client = Client(cluster) future = client.submit(lambda x: x + 1, 10) future.result() # returns 11 If you want cluster to terminate automatically after calculation finished, you can use the following: .. code-block:: python from slurmified import Cluster from distributed import Client slurm_kwargs = { 'mem-per-cpu': '100', 'time': '1-00:00:00' } inputs = list(range(0, 100)) with Cluster(slurm_kwargs).start_workers(10) as cluster: with Client(cluster) as client: incremented = client.map(lambda x: x+1, inputs) inverted = client.map(lambda x: -x, incremented) outputs = client.gather(inverted) print(outputs) # prints [-1, .. , -100] Quickly map function over arguments list: .. code-block:: python from slurmified import map_interactive slurm_kwargs = { 'mem-per-cpu': '100', 'time': '1-00:00:00' } inputs = list(range(0, 100)) map_func = map_interactive(10, slurm_kwargs=slurm_kwargs) outputs = map_func(lambda x: x+1, inputs) print(outputs) # prints [1, .. , 100] Installation ------------ Via pip:: pip install git+https://gitlab.com/slavoutich/slurmified.git