Apache Beam Executor

[alxmrs]

The pangeo-forge pipeline model is adapted to run on Apache Beam. Here, we use dummy stage numbers ("steps") as the input and output PCollections needed to work with Beam's programming model. Where substantive inputs need to be mapped to stages with arguments, "mappables" are combined with these steps. The result is that each stage executes serially, and mappable stages execute in parallel.

Fixes #169.

[alxmrs]

I was going to take some time to update the docs to reflect Apache Beam support in this CL. However, it seems like the docs need a larger overhaul anyway. So, I'll mark this ready for review.

Edit: When I wrote the above, I didn't notice the BaseRecipe class. The docs have been updated.

[rabernat]

This is fantastic Alex! I just activated the workflows, but I predict they will fail because beam is not in the dev environments. You'll need to add beam here and here.

[rabernat]

I think conda forge wants a dash in the name apache-beam: https://github.com/conda-forge/apache-beam-feedstock

[rabernat]

I merged my suggested change to the py3.8.yml environment. This turned up an error in the beam executor: https://github.com/pangeo-forge/pangeo-forge-recipes/runs/4138427592?check_suite_focus=true#step:8:2443

[alxmrs]

Thanks, I should have caught that locally. Apparently, a variable name I chose to meet the linter created a type check conflict with Beam. func may have been used as an internal argument, and had a conflict? Anyways, I changed the parameter name to fun, and it meets Beam's type check and the precommit checks, locally.

[rabernat]

I think this has now turned up an upstream issue with Xarray. (0.20.0 was just released) I'll make a PR for that fix.

[rabernat]

For some reason, the 3.9 conda environment was taking over an hour to build. I'm restarting the build.

[rabernat]

For some reason, the python 3.9 environment seems to be unsolvable. https://github.com/pangeo-forge/pangeo-forge-recipes/runs/4143328072?check_suite_focus=true#step:5:302

There must be a version conflict somewhere, but I don't know how to debug it. Any ideas?

[TomAugspurger]

mamba tends to give better error messages when there are conflicts.

That said, mamba solves fine for me locally (linux). Maybe try using mamba if you aren't already? https://github.com/conda-incubator/setup-miniconda#example-6-mamba

[rabernat]

Maybe try using mamba if you aren't already?

A PR to switch the CI to Mamba would be most welcome!

[alxmrs]

For some reason, the python 3.9 environment seems to be unsolvable.

It just occurred to me: Apache Beam only officially supports Python 3.8 -- it doesn't yet have support for 3.9, though it's on the roadmap.

• https://beam.apache.org/get-started/quickstart-py/
• https://issues.apache.org/jira/browse/BEAM-12000

[rabernat]

If you merge master (now that #226 is in), the tests should pass here.

Just remove beam from the 3.9 env if it is not supported.

[rabernat]

Looks great Alex! Just one place I'd like to tighten up.

[rabernat]

Ok, great! We are almost there. Just a few tiny comments then I promise we can stop iterating and merge this.