-
Notifications
You must be signed in to change notification settings - Fork 10
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #147 from NREL/gb/sup3rcc_example
Gb/sup3rcc example
- Loading branch information
Showing
19 changed files
with
1,046 additions
and
33 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| ################ | ||
| Sup3rCC Examples | ||
| ################ | ||
|
|
||
| Super-Resolution for Renewable Energy Resource Data with Climate Change Impacts (Sup3rCC) is one application of the sup3r software. In this work, we train generative models to create high-resolution (4km hourly) wind, solar, and temperature data based on coarse (100km daily) global climate model data (GCM). The generative models and high-resolution output data are publicly available on AWS S3 at the URI ``s3://nrel-pds-sup3rcc/`` and via HSDS at the bucket ``nrel-pds-hsds`` and path ``/nrel/sup3rcc/``. This set of examples lays out basic ways to use the Sup3rCC models and data. | ||
|
|
||
| Sup3rCC Data Access | ||
| =================== | ||
|
|
||
| The Sup3rCC data and models are publicly available in a public AWS S3 bucket. The data files and models can be downloaded directly from there to your local machine or an EC2 instance using the `AWS CLI <https://aws.amazon.com/cli/>`_. A word of caution: there's a lot of data here. The smallest Sup3rCC file for just a single variable is 18 GB, and a full year of data is 216 GB. | ||
|
|
||
| The Sup3rCC data is also loaded into `HSDS <https://www.hdfgroup.org/solutions/highly-scalable-data-service-hsds/>`_ so that you may stream the data via the `NREL developer API <https://developer.nrel.gov/signup/>`_ or your own HSDS server. This is the best option if you're not going to want the full annual dataset over the whole United States. See these `rex instructions <https://nrel.github.io/rex/misc/examples.hsds.html>`_ for more details on how to access this data with HSDS and rex. | ||
|
|
||
| Example Sup3rCC Data Usage | ||
| ========================== | ||
|
|
||
| The jupyter notebook in this example shows some basic code to access and explore the data. You can walk through the example notebook `here <https://github.com/NREL/sup3r/tree/main/examples/sup3rcc/using_the_data.ipynb>`_. You can also clone this repo, setup a basic python environment with `rex <https://github.com/NREL/rex>`_, and run the notebook on your own. | ||
|
|
||
| Running Sup3rCC Models | ||
| ====================== | ||
|
|
||
| In a first-of-a-kind data product, we have released the pre-trained Sup3rCC generative machine learning models along with the sup3r software so that anyone working at the intersection of energy and climate may create their own high-resolution renewable energy resource data from GCM input. You might want to do this if you have your own GCMs or climate scenarios that you're interested in studying. | ||
|
|
||
| To run the Sup3rCC models, follow these instructions: | ||
|
|
||
| #. Decide what kind of hardware you're going to use. You could technically run Sup3rCC on a desktop computer, but you will need lots of RAM (we use compute nodes with 170 GB of RAM). We recommend a high-performance-computing cluster if you have access to one, or an `AWS Parallel Cluster <https://aws.amazon.com/hpc/parallelcluster/>`_ if you do not. | ||
| #. Download the Sup3rCC models to your hardware using the AWS CLI: ``$ aws s3 cp s3://nrel-pds-sup3rcc/models/`` | ||
| #. Download the GCM data that you want to downscale from `CMIP6 <https://esgf-node.llnl.gov/search/cmip6/>`_ | ||
| #. Setup the Sup3rCC software. We recommend using `miniconda <https://docs.conda.io/en/latest/miniconda.html>`_ to manage your python environments. You can create a sup3r environment with the conda file in this example directory: ``$ conda env create -n sup3rcc --file sup3rcc_env.yml`` | ||
| #. Copy this examples directory to your hardware. You're going to be using the folder structure in ``/sup3r/examples/sup3rcc/run_configs`` as your project directories (``/sup3r/`` is a git clone of the sup3r software repo). | ||
| #. Navigate to ``/sup3r/examples/sup3rcc/run_configs/trh/`` and update all of the filepaths in the config files for the source GCM data, Sup3rCC models, and exogenous data sources (e.g. the ``nsrdb_clearsky.h5`` file). | ||
| #. Update the execution control parameters in the ``config_fwp.json`` file based on the hardware you're running on. | ||
| #. You can either run ``sup3r-batch`` to setup multiple run years, or ``sup3r-pipeline`` to run just one job. We recommend starting with ``sup3r-pipeline``. | ||
| #. To run ``sup3r-pipeline``, make sure you are in the directory with the ``config_pipeline.json`` and ``config_fwp.json`` files, and then run this command: ``python -m sup3r.cli -c config_pipeline.json pipeline`` | ||
| #. If you're running on a slurm cluster, this will kick off a number of jobs that you can see with the ``squeue`` command. If you're running locally, your terminal should now be running the Sup3rCC models. The software will create a ``./logs/`` directory in which you can monitor the progress of your jobs. | ||
| #. The ``sup3r-pipeline`` is designed to run several modules in serial, with each module running multiple chunks in parallel. Once the first module (forward-pass) finishes, you'll want to run ``python -m sup3r.cli -c config_pipeline.json pipeline`` again. This will clean up status files and kick off the next step in the pipeline (if the current step was successful). | ||
|
|
||
| Recommended Citation | ||
| ==================== | ||
|
|
||
| Grant Buster, Brandon Benton, Andrew Glaws, and Ryan King. "Super-Resolution for Renewable Energy Resource Data with Climate Change Impacts using Generative Machine Learning". Under review (April 2023). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| { | ||
| "pipeline_config": "./config_pipeline.json", | ||
| "sets": [ | ||
| { | ||
| "args": { | ||
| "file_paths": [ | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20150101-20151231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20160101-20161231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20170101-20171231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20180101-20181231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20190101-20191231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20200101-20201231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20210101-20211231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20220101-20221231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20230101-20231231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20240101-20241231.nc"], | ||
|
|
||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20500101-20501231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20510101-20511231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20520101-20521231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20530101-20531231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20540101-20541231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20550101-20551231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20560101-20561231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20570101-20571231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20580101-20581231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20590101-20591231.nc"] | ||
| ] | ||
| }, | ||
| "files": ["./config_fwp.json"], | ||
| "set_tag": "prod_solar_ssp585" | ||
| } | ||
| ] | ||
| } | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| { | ||
| "file_paths": "./chunks/*irradiance.h5", | ||
| "out_file": "./output/sup3r_data.h5", | ||
| "features": [ | ||
| "ghi", | ||
| "dni", | ||
| "dhi" | ||
| ], | ||
| "target_final_meta_file": "/projects/ntps/sup3r/final_models/meta/sup3rcc_mriesm20_25x_4km_meta.csv", | ||
| "log_file": "./logs/sup3r_collect.log", | ||
| "execution_control": { | ||
| "option": "eagle", | ||
| "feature": "--qos=high", | ||
| "walltime": 4, | ||
| "alloc": "ntps" | ||
| }, | ||
| "log_level": "DEBUG" | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| { | ||
| "file_paths": "PLACEHOLDER", | ||
| "model_kwargs": { | ||
| "spatial_solar_model_dirs": [ | ||
| "./sup3rcc_models_202303/sup3rcc_solar_step1_5x_1x_1f/", | ||
| "./sup3rcc_models_202303/sup3rcc_solar_step2_5x_1x_1f/" | ||
| ], | ||
| "spatial_wind_model_dirs": [ | ||
| "./sup3rcc_models_202303/sup3rcc_wind_step1_5x_1x_6f/", | ||
| "./sup3rcc_models_202303/sup3rcc_wind_step2_5x_1x_6f/" | ||
| ], | ||
| "temporal_solar_model_dirs": "./sup3rcc_models_202303/sup3rcc_solar_step3_1x_8x_1f/", | ||
| "t_enhance": 24, | ||
| "temporal_pad": 24 | ||
| }, | ||
| "model_class": "SolarMultiStepGan", | ||
| "out_pattern": "./chunks/sup3r_chunk_{file_id}.h5", | ||
| "log_pattern": "./logs/sup3r_fwp_log_{node_index}.log", | ||
| "bias_correct_method": "monthly_local_linear_bc", | ||
| "bias_correct_kwargs": { | ||
| "clearsky_ratio": {"feature_name": "clearsky_ratio", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_clearsky_ratio.h5", "out_range": [0, 1]}, | ||
| "U_10m": {"feature_name": "U_10m", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_u_10m.h5", "out_range": [-100, 100]}, | ||
| "V_10m": {"feature_name": "V_10m", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_v_10m.h5", "out_range": [-100, 100]}, | ||
| "U_100m": {"feature_name": "U_100m", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_u_100m.h5", "out_range": [-100, 100]}, | ||
| "V_100m": {"feature_name": "V_100m", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_v_100m.h5", "out_range": [-100, 100]}, | ||
| "U_200m": {"feature_name": "U_200m", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_u_200m.h5", "out_range": [-100, 100]}, | ||
| "V_200m": {"feature_name": "V_200m", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_v_200m.h5", "out_range": [-100, 100]} | ||
| }, | ||
| "spatial_pad": 10, | ||
| "temporal_pad": 1, | ||
| "fwp_chunk_shape": [26, 13, 1], | ||
| "input_handler": "DataHandlerNCforCC", | ||
| "input_handler_kwargs": { | ||
| "target": [23.2, -129], | ||
| "shape": [26, 59], | ||
| "temporal_slice": [null, null, null], | ||
| "nsrdb_source_fp": "/scratch/gbuster/sup3r/source_gcm_data/nsrdb_clearsky.h5", | ||
| "nsrdb_agg": 625, | ||
| "nsrdb_smoothing": 0, | ||
| "worker_kwargs": { | ||
| "max_workers": 1 | ||
| } | ||
| }, | ||
| "worker_kwargs": { | ||
| "max_workers": null, | ||
| "output_workers": 1, | ||
| "pass_workers": 4, | ||
| "ti_workers": 1 | ||
| }, | ||
| "max_nodes": 10, | ||
| "exo_kwargs": { | ||
| "file_paths": ["/scratch/gbuster/sup3r/source_gcm_data/wind_solar_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20150101-20151231.nc"], | ||
| "features": ["topography"], | ||
| "source_file": "/scratch/gbuster/sup3r/source_gcm_data/nsrdb_clearsky.h5", | ||
| "target": [23.2, -129], | ||
| "shape": [26, 59], | ||
| "s_enhancements": [1, 5, 5], | ||
| "agg_factors": [625, 25, 1], | ||
| "input_handler": "DataHandlerNCforCC", | ||
| "exo_steps": [0, 1, 2] | ||
| }, | ||
| "execution_control": { | ||
| "option": "eagle", | ||
| "feature": "--qos=normal", | ||
| "walltime": 4, | ||
| "memory": 173, | ||
| "alloc": "setosa" | ||
| }, | ||
| "log_level": "INFO" | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| { | ||
| "logging": {"log_level": "DEBUG"}, | ||
| "pipeline": [ | ||
| {"forward-pass": "./config_fwp.json"}, | ||
| {"solar": "./config_solar.json"}, | ||
| {"data-collect": "./config_collect.json"} | ||
| ] | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "fp_pattern": "./chunks/sup3r*.h5", | ||
| "nsrdb_fp": "/scratch/gbuster/sup3r/source_gcm_data/nsrdb_clearsky.h5", | ||
| "log_level": "INFO", | ||
| "log_pattern": "./logs/sup3r_solar_log_{node_index}.log", | ||
| "execution_control": { | ||
| "option": "eagle", | ||
| "feature": "--qos=normal", | ||
| "walltime": 0.2, | ||
| "alloc": "setosa" | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| { | ||
| "pipeline_config": "./config_pipeline.json", | ||
| "sets": [ | ||
| { | ||
| "args": { | ||
| "file_paths": [ | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20150101-20151231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20160101-20161231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20170101-20171231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20180101-20181231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20190101-20191231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20200101-20201231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20210101-20211231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20220101-20221231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20230101-20231231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20240101-20241231.nc"], | ||
|
|
||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20500101-20501231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20510101-20511231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20520101-20521231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20530101-20531231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20540101-20541231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20550101-20551231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20560101-20561231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20570101-20571231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20580101-20581231.nc"], | ||
| ["/scratch/gbuster/sup3r/source_gcm_data/temp_humi_day_MRI-ESM2-0_ssp585_r1i1p1f1_gn_20590101-20591231.nc"] | ||
| ] | ||
| }, | ||
| "files": ["./config_fwp.json"], | ||
| "set_tag": "prod_trh_ssp585" | ||
| } | ||
| ] | ||
| } | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| { | ||
| "file_paths": "./chunks/sup3r_chunk_*.h5", | ||
| "out_file": "./output/sup3r_data.h5", | ||
| "features": [ | ||
| "temperature_2m", | ||
| "relativehumidity_2m" | ||
| ], | ||
| "target_final_meta_file": "/projects/ntps/sup3r/final_models/meta/sup3rcc_mriesm20_25x_4km_meta.csv", | ||
| "log_file": "./logs/collect.log", | ||
| "execution_control": { | ||
| "option": "eagle", | ||
| "feature": "--qos=high", | ||
| "walltime": 4, | ||
| "alloc": "ntps" | ||
| }, | ||
| "log_level": "DEBUG" | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| { | ||
| "file_paths": "PLACEHOLDER", | ||
| "model_kwargs": { | ||
| "surface_model_kwargs": { | ||
| "model_dir": "./sup3rcc_models_202303/sup3rcc_trh_step1_25x_1x_2f/" | ||
| }, | ||
| "temporal_model_kwargs": { | ||
| "model_dirs": [ | ||
| "./sup3rcc_models_202303/sup3rcc_trh_step2_1x_24x_2f/" | ||
| ] | ||
| } | ||
| }, | ||
| "model_class": "MultiStepSurfaceMetGan", | ||
| "out_pattern": "./chunks/sup3r_chunk_{file_id}.h5", | ||
| "log_pattern": "./logs/sup3r_fwp_log_{node_index}.log", | ||
| "bias_correct_method": "monthly_local_linear_bc", | ||
| "bias_correct_kwargs": { | ||
| "temperature_2m": {"feature_name": "temperature_2m", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_temperature_2m.h5", "out_range": [-100, 100]}, | ||
| "temperature_max_2m": {"feature_name": "temperature_max_2m", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_temperature_max_2m.h5", "out_range": [-100, 100]}, | ||
| "temperature_min_2m": {"feature_name": "temperature_min_2m", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_temperature_min_2m.h5", "out_range": [-100, 100]}, | ||
| "relativehumidity_2m": {"feature_name": "relativehumidity_2m", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_relativehumidity_2m.h5", "out_range": [0, 100]}, | ||
| "relativehumidity_max_2m": {"feature_name": "relativehumidity_max_2m", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_relativehumidity_max_2m.h5", "out_range": [0, 100]}, | ||
| "relativehumidity_min_2m": {"feature_name": "relativehumidity_min_2m", "bias_fp": "/scratch/gbuster/sup3r/source_gcm_data/bc_conus_mriesm20_ssp585_r1i1p1f1_relativehumidity_min_2m.h5", "out_range": [0, 100]} | ||
| }, | ||
| "spatial_pad": 5, | ||
| "temporal_pad": 2, | ||
| "fwp_chunk_shape": [13, 13, 3], | ||
| "input_handler": "DataHandlerNCforCC", | ||
| "input_handler_kwargs": { | ||
| "target": [23.2, -129], | ||
| "shape": [26, 59], | ||
| "temporal_slice": [null, null, null], | ||
| "worker_kwargs": { | ||
| "max_workers": 1 | ||
| } | ||
| }, | ||
| "worker_kwargs": { | ||
| "max_workers": null, | ||
| "output_workers": 1, | ||
| "pass_workers": 3, | ||
| "ti_workers": 1 | ||
| }, | ||
| "max_nodes": 10, | ||
| "exo_kwargs": { | ||
| "file_paths": ["/datasets/sup3rcc/source/temp_humi_day_MRI-ESM2-0_ssp245_r1i1p1f1_gn_20500101-20501231.nc"], | ||
| "features": ["topography"], | ||
| "source_file": "/datasets/sup3rcc/source/nsrdb_clearsky.h5", | ||
| "target": [23.2, -129], | ||
| "shape": [26, 59], | ||
| "s_enhancements": [1, 25], | ||
| "agg_factors": [625, 1], | ||
| "input_handler": "DataHandlerNCforCC", | ||
| "exo_steps": [0, 1] | ||
| }, | ||
| "execution_control": { | ||
| "option": "eagle", | ||
| "feature": "--qos=normal", | ||
| "walltime": 4, | ||
| "memory": 173, | ||
| "alloc": "ntps" | ||
| }, | ||
| "log_level": "INFO" | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| { | ||
| "logging": {"log_level": "DEBUG"}, | ||
| "pipeline": [ | ||
| {"forward-pass": "./config_fwp.json"}, | ||
| {"data-collect": "./config_collect.json"} | ||
| ] | ||
| } |
Oops, something went wrong.