Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cesm cat builder #726

Merged
merged 9 commits into from
Dec 27, 2024
Merged

Cesm cat builder #726

merged 9 commits into from
Dec 27, 2024

Conversation

wrongkindofdoctor
Copy link
Collaborator

Description

  • Add cesm parser to catalog builder
  • refactor builder with parsers in a separate module
  • move xarray metadata search to a separate function
  • clean up unused imports
  • Add entries to cesm and gfdl field lists
    Associated issue # (replace this phrase and parentheses with the issue number)

How Has This Been Tested?
Please describe the tests that you ran to verify your changes in enough detail that
someone can reproduce them. Include any relevant details for your test configuration
such as the Python version, package versions, expected POD wallclock time, and the
operating system(s) you ran your tests on.

Checklist:

  • My branch is up-to-date with the NOAA-GFDL main branch, and all merge conflicts are resolved
  • The scripts are written in Python 3.12 or above (preferred; required if funded by a CPO grant), NCL, or R
  • All of my scripts are in the diagnostics/[POD short name] subdirectory, and include a main_driver script, template html, and settings.jsonc file
  • I have made corresponding changes to the documentation in the POD's doc/ subdirectory
  • I have requested that the framework developers add packages required by my POD to the python3, NCL, or R environment yaml file if necessary, and my environment builds with conda_env_setup.sh
  • I have added any necessary data to input_data/obs_data/[pod short name] and/or input_data/model/[pod short name]
  • My code is portable; it uses MDTF environment variables, and does not contain hard-coded file or directory paths
  • I have provided the code to generate digested data files from raw data files
  • Each digested data file generated by the script contains numerical data (no figures), and is 3 GB or less in size
  • I have included copies of the figures generated by the POD in the pull request
  • The repository contains no extra test scripts or data files

@wrongkindofdoctor
move call_build function to base class and define a class parser attr…
b57a35f 
…ibute that is modified for each child class at initialization
@wrongkindofdoctor
move parsers and json utilities to a separate module
eaa0d63
@wrongkindofdoctor
refine gfdl parsers
07cef4b
@wrongkindofdoctor
add function to get metadata from file attributes
cba1d82
@wrongkindofdoctor
add debugging statements to catalog_builder
471f2cc 
make reference dict definition a function
refine the catalog population procedure
need to define time_range
@wrongkindofdoctor
fix cesm parser frequency def
0b0bd1b 
add procedure to generate time_range to parsers
add procedure to fix units in catalog
@wrongkindofdoctor
add entries to CESM and GFDL fieldlists
257b2a5
@wrongkindofdoctor
clean up cesm parser
5e32919 
include file name parts in cesm parser frequency search
@wrongkindofdoctor wrongkindofdoctor added the data catalogs Issues related to intake esm data catalogs label Dec 27, 2024
@wrongkindofdoctor wrongkindofdoctor self-assigned this Dec 27, 2024
github-advanced-security[bot]
github-advanced-security bot found potential problems Dec 27, 2024
View reviewed changes
tools/catalog_builder/catalog_builder.py Dismissed Show dismissed Hide dismissed
tools/catalog_builder/catalog_builder.py Dismissed Show dismissed Hide dismissed
tools/catalog_builder/catalog_builder.py Dismissed Show dismissed Hide dismissed
tools/catalog_builder/parsers.py

# custom parser for GFDL am5 data that uses fieldlist metadata and the DRS to populate
# required catalog fields
def parse_gfdl_am5_data(file_name: str):

Check notice

Code scanning / CodeQL

Explicit returns mixed with implicit (fall through) returns Note

Mixing implicit and explicit returns may indicate an error as implicit returns always return None.
tools/catalog_builder/parsers.py Dismissed Show dismissed Hide dismissed
@wrongkindofdoctor wrongkindofdoctor merged commit d61a0d3 into NOAA-GFDL:main Dec 27, 2024
5 checks passed
@wrongkindofdoctor wrongkindofdoctor deleted the cesm_cat_builder branch December 27, 2024 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jkrasting jkrasting Awaiting requested review from jkrasting jkrasting is a code owner

@aradhakrishnanGFDL aradhakrishnanGFDL Awaiting requested review from aradhakrishnanGFDL aradhakrishnanGFDL is a code owner

@jtmims jtmims Awaiting requested review from jtmims jtmims is a code owner

Assignees

@wrongkindofdoctor wrongkindofdoctor

Labels
data catalogs Issues related to intake esm data catalogs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@wrongkindofdoctor