consolidated three EOBS recipe in meta recipe #169

norlandrhagen · 2022-08-12T04:07:50Z

pangeo-forge-bot · 2022-08-12T04:08:41Z

When I tried to load recipes/EOBS/meta.yaml, I got a <class 'yaml.scanner.ScannerError'>.

You should be able to replicate this error yourself.

First make sure you're in the root of your cloned staged-recipes repo. Then run this code in a Python interpreter:

import yaml  # note: you may need to `pip install yaml` first

with open("recipes/EOBS/meta.yaml", "r") as f:
    yaml.safe_load(f)

Please correct meta.yaml so that you're able to run this code without error, then commit the corrected meta.yaml.

norlandrhagen · 2022-09-08T20:21:47Z

Hey there @cisaacstern. Any idea how to fix this pre-commit issue?

Also, I think this is ready for a test run.

cisaacstern · 2022-09-08T20:27:42Z

Thanks for checking in here, @norlandrhagen. The pre-commit issue is being tracked in #88. It's on my radar to address soon, but we can ignore here for now.

Re: test run, yes, I'll do that now. 😄

cisaacstern · 2022-09-08T20:28:13Z

/run eobs-tg-tn-tx-rr-hu-pp

cisaacstern · 2022-09-08T20:28:29Z

/run eobs-wind-speed

cisaacstern · 2022-09-08T20:29:04Z

/run eobs-surface-downwelling

norlandrhagen · 2022-09-08T20:29:30Z

Awesome thanks!

cisaacstern · 2022-09-09T22:40:04Z

I'm confused about what happened to these test runs. If they failed to submit to Dataflow, the bot should have added a 😕 emoji reaction to the comments above. And if they failed on Dataflow, we should have gotten a new comment back about that.

🤔 This will require some digging in the logs to get to the bottom of. I'll try to do that this afternoon. Feel free to ping me here next week if I haven't gotten back with a response yet.

andersy005 · 2022-09-29T01:03:43Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

andersy005 · 2022-09-29T01:42:46Z

you bet, @norlandrhagen!

cisaacstern · 2022-09-29T02:02:32Z

@andersy005, similar to my confusion in #169 (comment), I do not see jobs for these recipe runs on the Dataflow dashboard. I am wondering if, in light of pangeo-forge/pangeo-forge-orchestrator#147, the issue is that starting these tests in such close succession to each other is spiking memory on the backend, and causing their submissions to fail. I'm not done with pangeo-forge/pangeo-forge-orchestrator#143 yet, so actually "proving" this in the logs is difficult, but I'll try an experiment now, of just deploying one of these, and watching the logs in my terminal in real time...

cisaacstern · 2022-09-29T02:02:54Z

/run eobs-wind-speed

cisaacstern · 2022-09-29T02:11:17Z

recipes/EOBS/meta.yaml

+    object: 'recipe:eobs-tg-tn-tx-rr-hu-pp'
+  - id: eobs-wind-speed
+    object: 'recipe:eobs_wind_speed'
+  - id: eobs-surface-downwelling
+    object: 'recipe:eobs_surface_downwelling'


@norlandrhagen & @andersy005 the problem is that these objects defined in the meta.yaml don't exist in the recipe.py.

Raphael, I tried to figure out which of the recipes

tg_tn_tx_rr_hu_pp_recipe = XarrayZarrRecipe( file_pattern=tg_tn_tx_rr_hu_pp_pattern, target_chunks=target_chunks, subset_inputs=subset_inputs ) qq_pattern_recipe = XarrayZarrRecipe( file_pattern=qq_pattern, target_chunks=target_chunks, subset_inputs=subset_inputs ) fg_pattern_recipe = XarrayZarrRecipe( file_pattern=fg_pattern, target_chunks=target_chunks, subset_inputs=subset_inputs )

correspond to each of these objects, but I couldn't figure it out based on the names alone.

the problem is that these objects don't exist in the recipe.py

i presume you got this information from dataflow logs, right?

@andersy005, actually no. This is an important debugging trick, so I'll describe it here, but then also an open an issue elsewhere to track document it.

So what I did was watch the server logs right after /runing the test. And I saw:

2022-09-29T02:02:56.875589+00:00 app[web.1]: 2022-09-29 02:02:56,875 DEBUG - orchestrator - Dumping bakery config to json: {'Bake': {'bakery_class': 'pangeo_forge_runner.bakery.dataflow.DataflowBakery', 'job_name': 'a6170692e70616e67656f2d666f7267652e6f7267251191'}, 'TargetStorage': {'fsspec_class': 's3fs.S3FileSystem', 'fsspec_args': {'client_kwargs': {'endpoint_url': 'https://ncsa.osn.xsede.org'}, 'default_cache_type': 'none', 'default_fill_cache': False, 'use_listings_cache': False, 'key': SecretStr('**********'), 'secret': SecretStr('**********')}, 'root_path': 'Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1191/eobs-wind-speed.zarr', 'public_url': 'https://ncsa.osn.xsede.org/{root_path}'}, 'InputCacheStorage': {'fsspec_class': 'gcsfs.GCSFileSystem', 'fsspec_args': {'bucket': 'pangeo-forge-prod-cache'}, 'root_path': 'pangeo-forge-prod-cache'}, 'MetadataCacheStorage': {'fsspec_class': 'gcsfs.GCSFileSystem', 'fsspec_args': {}, 'root_path': 'pangeo-forge-prod-cache/metadata/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1191/eobs-wind-speed.zarr'}, 'DataflowBakery': {'temp_gcs_location': 'gs://pangeo-forge-prod-dataflow/temp'}} 2022-09-29T02:02:56.877275+00:00 app[web.1]: 2022-09-29 02:02:56,877 DEBUG - orchestrator - Running command: ['pangeo-forge-runner', 'bake', '--repo=https://github.com/norlandrhagen/staged-recipes', '--ref=be1d60de2f3b9aed5fe480328b9d60e1ef0694ef', '--json', '--prune', '--Bake.recipe_id=eobs-wind-speed', '-f=/tmp/tmplol5klds.json', '--feedstock-subdir=recipes/EOBS'] 2022-09-29T02:02:58.116682+00:00 heroku[web.1]: Process running mem=647M(120.0%) 2022-09-29T02:02:58.143937+00:00 heroku[web.1]: Error R14 (Memory quota exceeded) 2022-09-29T02:03:00.449388+00:00 app[web.1]: [2022-09-29 02:03:00 +0000] [60] [ERROR] Exception in ASGI application 2022-09-29T02:03:00.449397+00:00 app[web.1]: Traceback (most recent call last): 2022-09-29T02:03:00.449398+00:00 app[web.1]: File "/opt/app/pangeo_forge_orchestrator/routers/github_app.py", line 621, in run 2022-09-29T02:03:00.449398+00:00 app[web.1]: out = subprocess.check_output(cmd) 2022-09-29T02:03:00.449399+00:00 app[web.1]: File "/usr/lib/python3.9/subprocess.py", line 424, in check_output 2022-09-29T02:03:00.449399+00:00 app[web.1]: return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, 2022-09-29T02:03:00.449400+00:00 app[web.1]: File "/usr/lib/python3.9/subprocess.py", line 528, in run 2022-09-29T02:03:00.449400+00:00 app[web.1]: raise CalledProcessError(retcode, process.args, 2022-09-29T02:03:00.449414+00:00 app[web.1]: subprocess.CalledProcessError: Command '['pangeo-forge-runner', 'bake', '--repo=https://github.com/norlandrhagen/staged-recipes', '--ref=be1d60de2f3b9aed5fe480328b9d60e1ef0694ef', '--json', '--prune', '--Bake.recipe_id=eobs-wind-speed', '-f=/tmp/tmplol5klds.json', '--feedstock-subdir=recipes/EOBS']' returned non-zero exit status 1.

But I didn't get much detail on what the called process error issue actually was. So I copy-and-pasted the config in the first log line

2022-09-29T02:02:56.875589+00:00 app[web.1]: 2022-09-29 02:02:56,875 DEBUG - orchestrator - Dumping bakery config to json: {'Bake': {'bakery_class': 'pangeo_forge_runner.bakery.dataflow.DataflowBakery', 'job_name': 'a6170692e70616e67656f2d666f7267652e6f7267251191'}, 'TargetStorage': {'fsspec_class': 's3fs.S3FileSystem', 'fsspec_args': {'client_kwargs': {'endpoint_url': 'https://ncsa.osn.xsede.org'}, 'default_cache_type': 'none', 'default_fill_cache': False, 'use_listings_cache': False, 'key': SecretStr('**********'), 'secret': SecretStr('**********')}, 'root_path': 'Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1191/eobs-wind-speed.zarr', 'public_url': 'https://ncsa.osn.xsede.org/{root_path}'}, 'InputCacheStorage': {'fsspec_class': 'gcsfs.GCSFileSystem', 'fsspec_args': {'bucket': 'pangeo-forge-prod-cache'}, 'root_path': 'pangeo-forge-prod-cache'}, 'MetadataCacheStorage': {'fsspec_class': 'gcsfs.GCSFileSystem', 'fsspec_args': {}, 'root_path': 'pangeo-forge-prod-cache/metadata/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1191/eobs-wind-speed.zarr'}, 'DataflowBakery': {'temp_gcs_location': 'gs://pangeo-forge-prod-dataflow/temp'}}

into a local JSON file on my laptop

{"Bake": {"bakery_class": "pangeo_forge_runner.bakery.dataflow.DataflowBakery", "job_name": "a6170692e70616e67656f2d666f7267652e6f7267251191"}, "TargetStorage": {"fsspec_class": "s3fs.S3FileSystem", "fsspec_args": {"client_kwargs": {"endpoint_url": "https://ncsa.osn.xsede.org"}, "default_cache_type": "none", "default_fill_cache": false, "use_listings_cache": false, "key": "**********", "secret": "**********"}, "root_path": "Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1191/eobs-wind-speed.zarr", "public_url": "https://ncsa.osn.xsede.org/{root_path}"}, "InputCacheStorage": {"fsspec_class": "gcsfs.GCSFileSystem", "fsspec_args": {"bucket": "pangeo-forge-prod-cache"}, "root_path": "pangeo-forge-prod-cache"}, "MetadataCacheStorage": {"fsspec_class": "gcsfs.GCSFileSystem", "fsspec_args": {}, "root_path": "pangeo-forge-prod-cache/metadata/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1191/eobs-wind-speed.zarr"}, "DataflowBakery": {"temp_gcs_location": "gs://pangeo-forge-prod-dataflow/temp"}}

and then I copied the pangeo-forge-runner command from the second log line

2022-09-29T02:02:56.877275+00:00 app[web.1]: 2022-09-29 02:02:56,877 DEBUG - orchestrator - Running command: ['pangeo-forge-runner', 'bake', '--repo=https://github.com/norlandrhagen/staged-recipes', '--ref=be1d60de2f3b9aed5fe480328b9d60e1ef0694ef', '--json', '--prune', '--Bake.recipe_id=eobs-wind-speed', '-f=/tmp/tmplol5klds.json', '--feedstock-subdir=recipes/EOBS']

and, replacing '-f=/tmp/tmplol5klds.json' with the path to my local JSON config (c.json), ran

$ pangeo-forge-runner bake --repo=https://github.com/norlandrhagen/staged-recipes --ref=be1d60de2f3b9aed5fe480328b9d60e1ef0694ef --json --prune --Bake.recipe_id=eobs-wind-speed -f=c.json --feedstock-subdir=recipes/EOBS

which gave me a descriptive error

{"message": "Error during running: 'eobs-tg-tn-tx-rr-hu-pp'", "exc_info": "Traceback (most recent call last):\n File \"/Users/charlesstern/miniconda3/envs/pfo-new/bin/pangeo-forge-runner\", line 8, in <module>\n sys.exit(main())\n File \"/Users/charlesstern/miniconda3/envs/pfo-new/lib/python3.9/site-packages/pangeo_forge_runner/cli.py\", line 28, in main\n app.start()\n File \"/Users/charlesstern/miniconda3/envs/pfo-new/lib/python3.9/site-packages/pangeo_forge_runner/cli.py\", line 23, in start\n super().start()\n File \"/Users/charlesstern/miniconda3/envs/pfo-new/lib/python3.9/site-packages/traitlets/config/application.py\", line 462, in start\n return self.subapp.start()\n File \"/Users/charlesstern/miniconda3/envs/pfo-new/lib/python3.9/site-packages/pangeo_forge_runner/commands/bake.py\", line 112, in start\n recipes = feedstock.parse_recipes()\n File \"/Users/charlesstern/miniconda3/envs/pfo-new/lib/python3.9/site-packages/pangeo_forge_runner/feedstock.py\", line 55, in parse_recipes\n recipes[r[\"id\"]] = self._import(r[\"object\"])\n File \"/Users/charlesstern/miniconda3/envs/pfo-new/lib/python3.9/site-packages/pangeo_forge_runner/feedstock.py\", line 43, in _import\n return self._import_cache[module][export]\nKeyError: 'eobs-tg-tn-tx-rr-hu-pp'", "status": "failed"}

cisaacstern · 2022-09-29T02:12:44Z

For the record, my theory in #169 (comment) about memory overruns was incorrect. I believe the problem is #169 (comment)

andersy005 · 2022-09-29T02:13:22Z

thank you for the heads up about potential memory spikes and tracking this issue, @cisaacstern!

norlandrhagen · 2022-09-29T16:22:48Z

@cisaacstern Thanks for catching that issue. I updated the recipe, so that each id in the meta.yaml should now correspond to the correct recipe.

andersy005 · 2022-09-29T16:23:10Z

/run eobs-tg-tn-tx-rr-hu-pp

pangeo-forge · 2022-09-29T16:28:28Z

The test failed, but I'm sure we can find out why!

Pangeo Forge maintainers are working diligently to provide public logs for contributors.
That feature is not quite ready yet, however, so please reach out on this thread to a
maintainer, and they'll help you diagnose the problem.

cisaacstern · 2022-09-29T18:16:16Z

@andersy005, rather than manually grab these error logs myself, I'm going to try to add a dataflow logs endpoint to the orchestrator right now, so that you can be more self-sufficient in diagnosing these issues going forward. I'll check back here once it's ready!

andersy005 · 2022-09-29T18:32:55Z

@andersy005, rather than manually grab these error logs myself, I'm going to try to add a dataflow logs endpoint to the orchestrator right now, so that you can be more self-sufficient in diagnosing these issues going forward. I'll check back here once it's ready!

that would be awesome, @cisaacstern. I haven't had a chance to do a deep dive into the dataflow functionality. Ping me if you need any feedback/review.

andersy005 · 2022-10-20T23:40:18Z

/run eobs-tg-tn-tx-rr-hu-pp

pangeo-forge · 2022-10-20T23:45:01Z

The test failed, but I'm sure we can find out why!

Pangeo Forge maintainers are working diligently to provide public logs for contributors.
That feature is not quite ready yet, however, so please reach out on this thread to a
maintainer, and they'll help you diagnose the problem.

andersy005 · 2022-10-20T23:47:47Z

recipes/EOBS/recipe.py

+target_chunks = {'time': 40}
+dataset_version = 'v23.1e'
+grid_res = '0.1'
+subset_inputs = {'time': 700}
+
+
+def make_filename(time: str, variable: str) -> str:
+    return f'https://knmi-ecad-assets-prd.s3.amazonaws.com/ensembles/data/Grid_{grid_res}deg_reg_ensemble/{variable}_ens_mean_{grid_res}deg_reg_{dataset_version}.nc'  # noqa: E501


@norlandrhagen, it appears apache-beam doesn't work well with globally defined variables such as grid_res. Could you move these into the function body of make_filename?

File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 624, in apache_beam.runners.common.SimpleInvoker.invoke_process File "/usr/local/lib/python3.9/dist-packages/apache_beam/transforms/core.py", line 1956, in <lambda> File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/executors/beam.py", line 40, in exec_stage File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 155, in cache_input fname = config.file_pattern[input_key] File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/patterns.py", line 219, in __getitem__ fname = self.format_function(**format_function_kwargs) File "/tmp/tmpis_skwvg/recipes/EOBS/recipe.py", line 16, in make_filename NameError: name 'grid_res' is not defined [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/cache_input/Execute-ptransform-56']

Yeah totally. Thanks for looking into this @andersy005

andersy005 · 2022-10-21T18:38:03Z

/run eobs-tg-tn-tx-rr-hu-pp

andersy005 · 2022-10-21T18:46:59Z

/run eobs-wind-speed

pangeo-forge · 2022-10-21T19:02:28Z

🎉 The test run of eobs-wind-speed at d93d94b succeeded!

import xarray as xr

store = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1284/eobs-wind-speed.zarr"
ds = xr.open_dataset(store, engine='zarr', chunks={})
ds

andersy005 · 2022-10-21T19:03:08Z

/run eobs-surface-downwelling

andersy005 · 2022-10-21T19:04:35Z

@norlandrhagen, https://pangeo-forge.org/dashboard/recipe-run/1284?feedstock_id=1

norlandrhagen · 2022-10-21T19:14:29Z

Woohoo! Thanks for picking this back up @andersy005

pangeo-forge · 2022-10-21T19:20:43Z

🎉 The test run of eobs-surface-downwelling at d93d94b succeeded!

import xarray as xr

store = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1285/eobs-surface-downwelling.zarr"
ds = xr.open_dataset(store, engine='zarr', chunks={})
ds

pangeo-forge · 2022-10-21T19:28:00Z

🎉 The test run of eobs-tg-tn-tx-rr-hu-pp at d93d94b succeeded!

import xarray as xr

store = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1283/eobs-tg-tn-tx-rr-hu-pp.zarr"
ds = xr.open_dataset(store, engine='zarr', chunks={})
ds

andersy005 · 2022-10-21T19:29:51Z

Woohoo! Thanks for picking this back up @andersy005

You bet! Do you plan on pushing more commits to this or should I go ahead and merge it?

norlandrhagen · 2022-10-21T19:32:13Z

Don't plan on pushing any more commits, so ready to merge!

consolidated three EOBS recipe in meta recipe

97afc29

typo fix

4e61e7e

This was referenced Aug 18, 2022

EOBS surface_downwelling_shortwave_flux_in_air #168

Closed

EOBS wind speed recipe #167

Closed

EOBS (TG, TN, TX, RR, HU, PP) #89

Closed

norlandrhagen added 2 commits September 8, 2022 12:04

Merge branch 'pangeo-forge:master' into EOBS_meta_recipe

abb4d4f

Merge branch 'pangeo-forge:master' into EOBS_meta_recipe

0401239

cisaacstern added the test-failed:needs-admin-debug A test failed, and this requires an admin to debug why. label Sep 9, 2022

cisaacstern mentioned this pull request Sep 10, 2022

In test run failed comments, give recipe_id and head_sha pangeo-forge/pangeo-forge-orchestrator#108

Closed

pre-commit-ci bot and others added 12 commits September 29, 2022 01:03

[pre-commit.ci] auto fixes from pre-commit.com hooks

62e1fb7

for more information, see https://pre-commit.ci

fix formatting

8ec752a

simplify pre-commit

5f70ee8

update pre-commit and add prettier

4e629f0

trigger pre-commit.ci deployment

a4096c5

[pre-commit.ci] auto fixes from pre-commit.com hooks

8b8fe61

for more information, see https://pre-commit.ci

fix formatting

c5f3e9f

remove deprecated pre-commit workflow

9f0e784

don't autofix PRs

cf9d183

remove old recipes

e240bf0

rename dataset -> feestock

79ff7ba

remove additional recipes subdirs

d593e73

cisaacstern reviewed Sep 29, 2022

View reviewed changes

cisaacstern mentioned this pull request Sep 29, 2022

Document how to re-run failed pangeo-forge-runner cmds from server logs locally pangeo-forge/pangeo-forge-orchestrator#148

Closed

norlandrhagen added 2 commits September 29, 2022 10:15

Merge branch 'pangeo-forge:master' into EOBS_meta_recipe

0a8151f

fixed recipe ID's

b0c5dcf

cisaacstern mentioned this pull request Sep 29, 2022

Adding EOOffshore CCMP v0.2.1.NRT recipe #145

Merged

andersy005 reviewed Oct 20, 2022

View reviewed changes

moved globally defined vars into functions to satisfy beam

d93d94b

norlandrhagen merged commit 6cb2e03 into pangeo-forge:master Oct 21, 2022

consolidated three EOBS recipe in meta recipe #169

consolidated three EOBS recipe in meta recipe #169

Conversation

norlandrhagen commented Aug 12, 2022

pangeo-forge-bot commented Aug 12, 2022

norlandrhagen commented Sep 8, 2022

cisaacstern commented Sep 8, 2022

cisaacstern commented Sep 8, 2022

cisaacstern commented Sep 8, 2022

cisaacstern commented Sep 8, 2022

norlandrhagen commented Sep 8, 2022

cisaacstern commented Sep 9, 2022

andersy005 commented Sep 29, 2022

andersy005 commented Sep 29, 2022

cisaacstern commented Sep 29, 2022

cisaacstern commented Sep 29, 2022

cisaacstern Sep 29, 2022 • edited Loading

Choose a reason for hiding this comment

andersy005 Sep 29, 2022

Choose a reason for hiding this comment

cisaacstern Sep 29, 2022

Choose a reason for hiding this comment

cisaacstern commented Sep 29, 2022

andersy005 commented Sep 29, 2022

norlandrhagen commented Sep 29, 2022

andersy005 commented Sep 29, 2022

pangeo-forge bot commented Sep 29, 2022

cisaacstern commented Sep 29, 2022

andersy005 commented Sep 29, 2022

andersy005 commented Oct 20, 2022

pangeo-forge bot commented Oct 20, 2022

andersy005 Oct 20, 2022

Choose a reason for hiding this comment

norlandrhagen Oct 21, 2022

Choose a reason for hiding this comment

andersy005 commented Oct 21, 2022

andersy005 commented Oct 21, 2022

pangeo-forge bot commented Oct 21, 2022

andersy005 commented Oct 21, 2022

andersy005 commented Oct 21, 2022

norlandrhagen commented Oct 21, 2022

pangeo-forge bot commented Oct 21, 2022

pangeo-forge bot commented Oct 21, 2022

andersy005 commented Oct 21, 2022

norlandrhagen commented Oct 21, 2022

cisaacstern Sep 29, 2022 •

edited

Loading