New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Running pipelines on AWS? #42

Open

rsignell-usgs opened this issue Jan 2, 2022 · 1 comment

rsignell-usgs commented Jan 2, 2022 •

edited

Loading

Hello! Newbie here. :)
Any tips for someone who would like to try an xarray-beam rechunking pipeline on AWS?
(USGS is wedded to AWS at the moment)

Contributor

alxmrs commented Jan 19, 2022

Sorry for the late reply. I'm not too familiar with how we can run this on AWS (I haven't done it myself), but here's how I would do it:

Set up a (py)Spark cluster in AWS EMR, see the AWS docs or this step-by-step Medium post.
Specify the SparkRunner in the --runners flag of your python Beam Pipeline.
- See also: https://beam.apache.org/documentation/runners/spark/#running-on-a-pre-deployed-spark-cluster
- Likely, you'll need to install apache_beam with AWS's extra requirements: pip install apache_beam[aws].

I hope that helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment