Sync container branch with main #724
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Create Dockerfile works with synthetic example_multicase POD * Update Dockerfile * Update Dockerfile * Create docker-build-and-push.yml * Update docker-build-and-push.yml * Update docker-build-and-push.yml * Update docker-build-and-push.yml * Update docker-build-and-push.yml
* Create container_config_demo.jsonc * Create container_cat.csv * Create container_cat.json * Update container_config_demo.jsonc * docs * Update ref_container.rst * Update ref_container.rst * Update ref_container.rst * Update ref_container.rst * Update ref_container.rst * Update dev_start.rst * Update ref_container.rst * Update dev_start.rst * Update ref_container.rst * Update doc/sphinx/dev_start.rst Co-authored-by: Jess <[email protected]> * Update doc/sphinx/ref_container.rst Co-authored-by: Jess <[email protected]> * Update doc/sphinx/ref_container.rst Co-authored-by: Jess <[email protected]> * Update doc/sphinx/ref_container.rst Co-authored-by: Jess <[email protected]> * Update doc/sphinx/dev_start.rst Co-authored-by: Jess <[email protected]> --------- Co-authored-by: Jess <[email protected]>
* fix unresolved conda_root ref in pod_setup comment out no_translation setting for matching POD and runtime conventions for testing * fix coord_name def in translate_coord * define var_id separately in pp query * change new_coord definition to obtain ordered dict instead of generator object in translation.create_scalar_name so that deepcopy can pickle it * change logic in pod_setup to set translation object to no_translation only if translate_data is false in runtime config file * uncomment more set1 pods that pass initial testing in house * add checks for no_translation data source and assign query atts using the var object instead of the var.translation object if True to preprocessor * remove old comment from preprocessor * change value for for hourly data search in datelabel get_timedelta_kwargs to return 1hr instead of hr so that the frequency for hourly data matchew required catalog specification * comment out some set1 tests, since they are timing out on CI * rename github actions test config files split group 1 CI tests into 2 runs to avoid timeout issues * update mdtf_tests.yml to reference new config file names and clean up deprecated calls * update mdtf_tests.yml * update matrix refs in mdtf_tests.yml * revert changes to datelabel and move hr --> 1hr freq conversion to preprocessor * delete old test files just run 1 POD in set1 tests try adding timeouts mdtf_tests.yml * fix typo in timeout call in mdtf_tests * fix GFDL entries in test catalogs * fix varid entries for wvp in test catalogs * change atmosphere_mass_content_of_water_vapor id from prw to wvp in gfdl field table * comment out long_name check in translation.py * define src_unit for coords if available in preprocessor.ConvertUnitsFunction redefine dest_unit using var.units.units so that parm is a string instead of a Units.units object in call to units.convert_dataarray * log warning instead of raising error if attr name doesn't match in xr_parser.compare_attr so that values can be converted later * fix variable refs in xarray datasets in units.convertdatarray add check to convert mb to hPa to convertdataarray * fix frequency entries for static vars in test catalogs * remove duplicate realm entries from stc_eddy_heat_fluxes settings file * remove non alphanumeric chars from atts in xr_parser check_metadata * comment out non-working PODs in set 3 tests
#691) * infer 'start_time' and 'end_time' from 'time_range' due to type issues * add warning * fix ci issue
* move line setting date_range in query_catalog() * cleanup print
* fix hr -> 1hr freq conversion in pp query try using regex string contains standard_name in query * add check for parameter type to xr_parser approximate_attribute_value * remove regex from pp query standard_name * add check that bounds is populated in cf.assessor, then check coord attrs and only run coord bounds check if bounda s are not None in xr_parser
* fix ctd file formatting and typos * more formatting and typo fixes in ctd POD * uncomment convective transistion diag POD in 1a CI test config files * try moving convective_transition_pod to ubuntu suite 2 tests * add wkdir cleanup between each test run step and separate obs data fetching for set 1 tests in ci config file * move convective_transition_diag POD to set 1b tests * just run 1 POD in set 1a and 2 PODs in set 1b to avoid runner timeouts * reorganize 1b tests
…ects (#699) * add alternate_stanadard_names entries to precipitation_flux vars in CMIP and GFDL fieldlists add list of applicable realms to preciptitation flux * add alternate_standard_names attributes and property setters to DMDependentvariable class that is VarlistEntry parent class define realm parm as string or list * extend realm search in fieldlist lookup tables to use a realm list in the translation add list to realm type hints in translation module * extend standard_name query to list that includes alternate_standard_names if present in the translation object * break up rainfall_flux and precipitation_flux entries in CMIP and GFDL field tables since translator can't parse realm list correctly * revert realm type hints defined as string or list and casting realm strings to listsin translation module * change assertion to log errof if translation is None in varlist_util * define new standard_name for pp xarray vars using the translation standard_name if the query standard name is a list with alternates instead of a string
* add function check_multichunk to fix issue with chunk_freqs * fix function comment grammar grammar grammar * move log warning
* add plots link to pod_error_snippet.html * remove empty line
* add variable table script to docs * move file * Delete tools/get_POD_varname/MDTF_Variable_Lists.html
…fig_template.jsonc
…filePP init method
* Update README.md * Update start_install.rst
* define missing entries in dummy translation object returned by NoTranslationFieldlist.translate add logic to determine alternate_standard_names attribute to NoTranslationFieldlist.translate * set translate_data to false for testing * edit logging message for no translation setting in pod_setup * add todo to translation translate_coord and cleanup comments * remove checks for no_translation from preprocessor * define TranslatedVarlistEntry name attribute using data convention field table variable id * revert debugging changes from test config file * update docs for translate_data flag in the runtime config file * fix variable_id and var_id refs in dummy translate method
* add placeholder functions for date range cropping * refine crop_date_range function. Need to figure out how to pass calendar from subset df * continue reworking crop_date_range * revert changes to check_group_daterange, and add check that input files overlap start and end times add option aggregate=false to to_dataset_dict call look into replaceing check_time_bounds with crop date range call before the xarray merge * reorder crop_date_range call add calls to parse xr time coord and define start and end times for dataset * finalize logic in crop_date_range * remove start_time and end_time from, and add time_range column to catalog generated by define_pp_catalog_assets * replace start_time and end_time entries with time_range entries populated from information in processed xarray dataset in write_pp_catalog * remove unused dask import from preprocessor * replace hard coded time dimension name with var.T.name in call to xarray concatenate * add check_time_bounds call back to query and fix definitions for modified start and end points so that they use the dataset information * fix hour, min, sec defs in crop_date_range for new start and end times * strip non-numeric chars from strings passed to _coerce_to_datetime * add logic to define start and end points for situation where desired date range is contained by xarray dataset to crop_date_range
* fix forcing_feedback settings formatting * add check for user_pp_scripts attribute before looping through list to multifilepreprocessor add_user_pp_scripts method * add snakeviz to env_dev.yml * move drop_atts loop to a separate function that is called by crop_date_range and before merging xradate_range and before merging datasets in query_catalog in the preprocessor
* add snakeviz, gprof2dot, and intake-esgf packages to env_dev file * add viztracer to dev environment file * add kerchunk package to dev environment
* fix pp issues for seaice_suite * fix arg issue * rename functions * add default return for conversion function
jtmims
requested review from
jkrasting,
wrongkindofdoctor and
aradhakrishnanGFDL
as code owners
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
jtmims
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
I noticed the container was a little behind the latest version of the MDTF, as it has not had a pull since it was first released. This pull request will bring the container up to date. Hopefully when we feel it is stable, we can have pushes on the main branch go into the container.
Associated issue # (replace this phrase and parentheses with the issue number)
How Has This Been Tested?
Please describe the tests that you ran to verify your changes in enough detail that
someone can reproduce them. Include any relevant details for your test configuration
such as the Python version, package versions, expected POD wallclock time, and the
operating system(s) you ran your tests on.
Checklist:
conda_env_setup.sh