max_parallel_tasks and Dask

[bouweandela]

Issue reported by @schlunma

Since #2522, recipe_schlund20esd.yml hangs indefinitely after a while when using the default threaded scheduler and --max-parallel-tasks=16. Another example is an ICON recipe that evaluates 3D data, which now hangs with the distributed scheduler and --max-parallel-tasks=2. Before #2522, both setups run fine. With --max-parallel-tasks=1, the recipes also run fine with current main branch. So my guess is the changes in #2522 do not play nicely with our own custom parallelization.

[bouweandela]

Indeed, max_parallel_tasks does not play nice with Dask.

• When running the default scheduler, you will get way too many dask workers, i.e. max_parallel_tasks * num_workers (the ~/.config/dask setting) in the worst case and run out of memory.
• When using the distributed scheduler, it will submit multiple tasks graphs to the same scheduler while it is already running things and it cannot handle this if these task graphs are large: it will send too much work to the workers and the workers will run out of memory and get killed. I suspect that the reason that the distributed scheduler starts all of the tasks that are submitted to it immediately is to avoid deadlocks. Unfortunately, it does not seem to be able to adapt its task scheduling based on what is already running.

Note that this was already the case before #2522 was merged, but because #2522 builds one task graph for an entire preprocessing task (potentially many output files) instead of one per output file, the task graphs that are submitted to the scheduler are larger and therefore the problem will appear a bit more often. I added this note to the tutorial with a small explanation about it.

A nice solution, that better utilizes the available Dask workers, would be to also build the task graphs on the Dask workers, as implemented in #2316. However, we cannot proceed with that at the moment because it is blocked because ESMF does not run on a Dask worker because it only runs in a single thread (or maybe it is possible, but I couldn't find a way to configure the workers such that it works).

Another potential solution, that is less optimal but also requires fewer changes, would be to only build the tasks graphs in parallel using max_parallel_tasks and then submit them to the scheduler all at once (or one after the other).