Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WW3 Integration #7

Closed
uturuncoglu opened this issue Jun 20, 2023 · 96 comments
Closed

WW3 Integration #7

uturuncoglu opened this issue Jun 20, 2023 · 96 comments
Assignees
@uturuncoglu @yunfangsun
Labels
old_integration

Comments

@uturuncoglu
Copy link
Collaborator

@pvelissariou1 I am opening this issue for WW3 integration and testing. Since WW3 is integral part of the ufs-weather-model, I'll try to test existing configurations under CoastalApp-testsuite in here to see the issues that we might face.
@uturuncoglu uturuncoglu added the enhancement New feature or request label Jun 20, 2023
@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 I tried to run ike_shinnecock.atm2ww3 with the WW3 that is found in ufs-coastal and I am getting following error from WW3 cap,

11:fv3_coastal.exe    0000000001E11790  wav_shel_inp_mp_r         774  wav_shel_inp.F90
11:fv3_coastal.exe    0000000001B29726  wav_comp_nuopc_mp        1598  wav_comp_nuopc.F90

The error is coming from following line in WW3,

      call print_logmsg(740+IAPROC, ' fnmpre'//trim(fnmpre), w3_debuginit_flag)
      open (newunit=ndsi,file=trim(fnmpre)//'ww3_shel.inp',status='old',iostat=ierr)
      rewind (ndsi)

So, it seems that ww3_shel.inp has some issue. Maybe it is because we are using newer version of the model. Do you have any idea?

@uturuncoglu
Copy link
Collaborator Author

I think this configuration does not have ww3_shel.inp. Do we need to create one?

@pvelissariou1
Copy link
Collaborator

pvelissariou1 commented Jun 20, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 Thanks. Let me try with other cap. Maybe that will work.

@uturuncoglu
Copy link
Collaborator Author

uturuncoglu commented Jun 20, 2023 •
edited
Loading

@pvelissariou1 Okay. It am able to build with old cap which seems CoastallApp was using. Anyway, I need to create input files with ww3_grid and ww3_bounc. I think those are not part of the WW3. Could you give me more information about it? Thanks.

@uturuncoglu uturuncoglu self-assigned this Jun 20, 2023
@pvelissariou1
Copy link
Collaborator

pvelissariou1 commented Jun 20, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 The existing WW3 is not building those tools like ww3_grid. So, let me check the CostallApp to see how those tools are building and maybe I could make some change in the WW3 CMake build to make it work.

@uturuncoglu
Copy link
Collaborator Author

It seems that WW3 is not building tools when it is build for coupling. I am not sure why but I could fix it.

@pvelissariou1
Copy link
Collaborator

pvelissariou1 commented Jun 21, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 I opened an issue in WW3 repo. NOAA-EMC/WW3#1025 Let's wait for their response and I could make a very simple PR to WW3 to fix it.

@pvelissariou1
Copy link
Collaborator

pvelissariou1 commented Jun 21, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

uturuncoglu commented Jun 21, 2023 •
edited
Loading

@pvelissariou1 I am very close to make it work. The only issue again ATM forcing for CDEPS. Which forcing do we need to use for this case? I could again force with ERA5 but I think it is low resolution for this case.

@pvelissariou1
Copy link
Collaborator

pvelissariou1 commented Jun 21, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 Thanks. Yes. I saw that one but it is in vector format and designed to work with ATMMESH. Since we are using CDEPS at this point, I need to have same dataset but in its original form (on lat-on grid). I could not use this 1d data under CDEPS.

@pvelissariou1
Copy link
Collaborator

pvelissariou1 commented Jun 21, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 I am using Wind_Pressure_HWRF_Florence_ExtendedSmooth.nc for it but this case cover different dates (I am not sure why and I think all the tests need to use same dataset - by this way you could compare their results). The data used by ADCRIC covers only 2018-09-09 06:00:00 to 2018-09-18 17:00:00 but the WW3 configuration starts from 2018-09-04 12:00. I think that we need to make consistent all the configurations in terms of used data and time of simulation. So, we could use the same ATM input for all the similar configurations.

@uturuncoglu
Copy link
Collaborator Author

uturuncoglu commented Jun 22, 2023 •
edited
Loading

@pvelissariou1 I could run ike_shinnecock.atm2ww3 test case with ERA5 input. We could change the input if you want or we could keep like it is. Let me know what do you want. My run directory is in /glade/scratch/turuncu/COASTAL_APP/ike_shinnecock.atm2ww3 on Cheyenne. It might be nice to check the output but I am not sure which file do I need to look with which tool? Anyway, if you give me more information about it, I could check the output. If everything is fine and then we could add this as a RT.

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 Maybe I have no output but model seems working. If you don't mind could you check my run directory. If this run is okay, maybe we could try to run fvcom with ww3 case but I need to first solve the build issue.

@pvelissariou1
Copy link
Collaborator

pvelissariou1 commented Jun 22, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 @saeed-moghimi-noaa I could able to run WW3 ike_shinnecock.atm2ww3 case with CMEPS mediator after I got some key information from Jessica from NOAA and fixing CMEPS. I did not check the results but I'll finalize the fork update the configuration files and define as a RT under ufs-coastal. so, you could run and test. This will also give us a capability to extend exiting RTs with three way coupling by including WW3.

@saeed-moghimi-noaa
Copy link

This is a great news! Thanks @uturuncoglu

FYI @yunfangsun

@pvelissariou1
Copy link
Collaborator

pvelissariou1 commented Aug 1, 2023 via email

@yunfangsun
Copy link
Collaborator

@uturuncoglu Hi Ufuk,

I can't access to Cheyenne, could I know if the codes have been updated on Github?

Thank you!

Yunfang

@uturuncoglu
Copy link
Collaborator Author

@yunfangsun I did not push the modifications to repository yet but I'll do it soon. Cheyenne is down this week. So, at this point I am continue to development on Orion.

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 Did you modify the configuration of WW3 related tests? Still I could not get output from ike_shinnecock.atm2ww3. If you could double check in your side, that would be great. Thanks.

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 @yunfangsun I added coastal_ike_shinnecock_atm2ww3 test case as RT and push all the changes to the ufs-coastal. The ATMMESH+WW3 (the version used by bfs-weather model) coupling working with CMEPS. You just need to create ESMF mesh file as extra input (the ww3_grid with this version of WW3 creates SCRIP grid definition file and this can be used to create ESMF mesh file). We still need to modify WW3 configuration since it is not write any output to check. Anyway, please try and let me know if you have any issue. I also updated my input directory (/work/noaa/nems/tufuk/RT - just reminder, you need to update DISKNM under rt.sh with this for Orion) on Orion to reflect changes related to WW3. Anyway, let me know if you need any other information. I am also planning to bring other WW3 related tests one-by-one starting from the standalone one.

@yunfangsun
Copy link
Collaborator

@uturuncoglu Thank you for the good news, I will try.

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 I think following lines are missing in ww3 standalone test ike_shinnecock.ww3 setup scripts,

cp -fp mod_def.ww3 mod_def.inlet
cp -fp mod_def.ww3 mod_def.points

without those I am getting error from run. Any idea?

@yunfangsun
Copy link
Collaborator

@uturuncoglu Thank you! I have just submitted it

@yunfangsun
Copy link
Collaborator

@uturuncoglu

Now it could run 40 hours, it stopped at 09-16-16:00, and the mediator.log shows:


  Add wevap to budgets with index           20
  Add wrunoff to budgets with index           21
  Add wfrzrof to budgets with index           22
  Add saltf to budgets with index           23
  Add     inst to budgets with index            1
  Add all_time to budgets with index            2

(med.F90:DataInitialize) read_restart =  F

(med_time_alarmInit): creating alarm med_profile_alarm

(med_time_alarmInit): creating alarm alarm_stop

and log.ww3 shows:

     1332|    37|            13:00:00 |   X                   | X                |
  --------+------+---------------------+-----------------------+------------------+
      1368|    38|            14:00:00 |   X                   | X                |
  --------+------+---------------------+-----------------------+------------------+
      1404|    39|            15:00:00 |   X                   | X                |
  --------+------+---------------------+-----------------------+------------------+
      1440|    40|            16:00:00 |   X                   | X                |
  --------+------+---------------------+-----------------------+------------------+
ymd2date currTime wav_comp_nuopc hh,mm,ss,ymd  16   0   0  20220916

do you have any suggestions?

@uturuncoglu
Copy link
Collaborator Author

@yunfangsun is there anything in other log files such as err, out and datm.log? If you don't mind could you submit job again and see if is failing in the same place or not. If it dies not help, then please send me all the information to me to reproduce the run in my end.

@yunfangsun
Copy link
Collaborator

@uturuncoglu the datm.log seems there is no problem:

(shr_strdata_readstrm) reading file ub: era5/download_inv_fix.nc     370
 atm : model date     20220916       28800
(shr_strdata_readstrm) reading file ub: era5/download_inv_fix.nc     371
 atm : model date     20220916       32400
(shr_strdata_readstrm) reading file ub: era5/download_inv_fix.nc     372
 atm : model date     20220916       36000
(shr_strdata_readstrm) reading file ub: era5/download_inv_fix.nc     373
 atm : model date     20220916       39600
(shr_strdata_readstrm) reading file ub: era5/download_inv_fix.nc     374
(dshr_restart_write)  writing ufs.cpld.datm.r.2022-09-16-43200.nc20220916  43200
 atm : model date     20220916       43200
(shr_strdata_readstrm) reading file ub: era5/download_inv_fix.nc     375
 atm : model date     20220916       46800
(shr_strdata_readstrm) reading file ub: era5/download_inv_fix.nc     376
 atm : model date     20220916       50400
(shr_strdata_readstrm) reading file ub: era5/download_inv_fix.nc     377
 atm : model date     20220916       54000
(shr_strdata_readstrm) reading file ub: era5/download_inv_fix.nc     378
 atm : model date     20220916       57600
(shr_strdata_readstrm) reading file ub: era5/download_inv_fix.nc     379
 atm : model date     20220916       61200

The out file is also normal


101:  No. of solver iterations          20     2379724   3.65856522059484
 101:    3.00000000000000
 101:  No. of solver iterations          30     2435127   1.41561414261967
 101:    3.00000000000000
 101:  No. of solver iterations           0     1720715   30.3380762027680
 101:    3.00000000000000
 101:  No. of solver iterations          10     2163503   12.4121187290848
 101:    3.00000000000000
 101:  No. of solver iterations          20     2379713   3.65901054777672
 101:    3.00000000000000
 101:  No. of solver iterations          30     2435206   1.41241588376799
 101:    3.00000000000000
 101:  No. of solver iterations           0     1720715   30.3380762027680
 101:    3.00000000000000
 101:  No. of solver iterations          10     2164108   12.3876257340814
 101:    3.00000000000000
 101:  No. of solver iterations          20     2379921   3.65058981561026
 101:    3.00000000000000
 101:  No. of solver iterations          30     2435584   1.39711282242700
 101:    3.00000000000000
 101:  No. of solver iterations           0     1720715   30.3380762027680
 101:    3.00000000000000
 101:  No. of solver iterations          10     2164606   12.3674645580290
 101:    3.00000000000000
 101:  No. of solver iterations          20     2380067   3.64467910937802
 101:    3.00000000000000
 101:  No. of solver iterations          30     2435578   1.39735572816257
 101:    3.00000000000000
 101:  No. of solver iterations           0     1720715   30.3380762027680
 101:    3.00000000000000
 101:  No. of solver iterations          10     2164854   12.3574244542920
 101:    3.00000000000000
 101:  No. of solver iterations          20     2379836   3.65403098019751
 101:    3.00000000000000

The error file is


+ ESMF_RUNTIME_PROFILE=ON
+ export ESMF_RUNTIME_PROFILE_OUTPUT=SUMMARY
+ ESMF_RUNTIME_PROFILE_OUTPUT=SUMMARY
+ [[ intel == gnu ]]
+ sync
+ sleep 1
+ srun --label -n 5000 ./fv3.exe
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
4196: forrtl: error (78): process killed (SIGTERM)
4196: Image              PC                Routine            Line        Source
4196: libc.so.6          000014E95E53CD90  Unknown               Unknown  Unknown
4196: fv3.exe            0000000001D474CA  pdlib_w3profsmd_m        5922  w3profsmd_pdlib.F90
4196: fv3.exe            0000000001D3D635  pdlib_w3profsmd_m        2796  w3profsmd_pdlib.F90
4196: fv3.exe            0000000001BE4E68  w3wavemd_mp_w3wav        1843  w3wavemd.F90
4196: fv3.exe            00000000019E9D62  wav_comp_nuopc_mp        1126  wav_comp_nuopc.F90
4196: fv3.exe            0000000000C37EB8  Unknown               Unknown  Unknown
4196: fv3.exe            0000000000C37E27  Unknown               Unknown  Unknown
4196: fv3.exe            0000000000C36A03  Unknown               Unknown  Unknown
4196: fv3.exe            0000000000433182  Unknown               Unknown  Unknown
4196: fv3.exe            000000000205FCDD  Unknown               Unknown  Unknown
4196: fv3.exe            0000000000B36B84  Unknown               Unknown  Unknown

The whole run folder is located on Hercules/Orion at /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/rt_15740_atm_ww/coastal_ian_atm2ww3_intel_1, and the compilation I am using is the coastal_ike_shinnecock_atm2ww3_intel regression test.

I have just resubmitted it, and will let you know when it fails

@yunfangsun
Copy link
Collaborator

Hi @uturuncoglu
I have tried twice by submitting job_card, and it stopped at different times, 36 hours and 37 hours, then I tried to use this same setting, but wind is using wind.ww3 which is interpolated from the exact era5 by using ww3_prnc, and use stand-alone ww3 by submitting (xmodel_slurm.job), and it didn't break, and kept running. all the files are located at /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/rt_15740_atm_ww/coastal_ian_atm2ww3_intel_1

@uturuncoglu
Copy link
Collaborator Author

@yunfangsun It seems that the issue in WW3 side. Maybe it is not getting fields from DATM. Let me check.

@uturuncoglu
Copy link
Collaborator Author

@yunfangsun It seems your are using ınp. format for configuration but in the RT I am using nml and when you couple the WW3 you need to change D to C for the wind but I am not seeing those kind of definition in your configuration file. So, could you use ww3_shel.nml file from RT and modify its simulation date and run again to see what happens.

@yunfangsun
Copy link
Collaborator

@uturuncoglu I have removed the ww3_shel.inp, and replaced it with ww3_shel.nml, the job stopped again after 40 hours.

@uturuncoglu
Copy link
Collaborator Author

@yunfangsun I could run your case on Hercules with Intel. At this point the run is in 55 hours and still running. I just compile the case with latest version of ufs-coastal using following command,

./compile.sh "hercules" "-DAPP=CSTLW -DPDLIB=ON" coastal intel NO NO

except using Intel (I am not sure but maybe you are using GNU), I did not change anything in the configuration. Anyway, I'll let you know if it fails but you might try with Intel indoor and let me know how it goes.

@uturuncoglu
Copy link
Collaborator Author

@yunfangsun The performance of the model is something like 55 hours in 44 min but this includes also initialization. So, roughly you can do 75 hours/hours. It seems that you are tying to run 900 hours. So, I think this will not finish in 8 hours time window. I suggest to increase the number of cores more. Maybe you could try with doubling WW3 resource. Anyway, of course this runs without any issue in your end.

@yunfangsun
Copy link
Collaborator

yunfangsun commented Jan 14, 2024 •
edited
Loading

@uturuncoglu I have download the newest version of ufs-coastal at /work2/noaa/nos-surge/yunfangs/ufs-coastal

and compile it by using ./compile.sh "hercules" "-DAPP=CSTLW -DPDLIB=ON" coastal intel NO NO, and the run folder is in /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/rt_15740_atm_ww/coastal_ian_atm2ww3_intel_3, it stopped after 40 hours again,

and compile it by using regression test ./rt.sh-a coast -l rt_coastal.conf -c -k -n coastal_ike_shinnecock_atm2ww3 intel, run it in the folder /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/rt_1795978_atm_ww3_new/coastal_ian_atm2ww3_intel, it stopped after 39 hours.

@uturuncoglu
Copy link
Collaborator Author

@yunfangsun I am not sure what is wrong in your case but mine is run until 20221012.000000 (started from 20220915.000000) without any issue under my account. It is almost 649 hours. So, we still need to increase the resource. BTW, this is my run directory if you want to compare with yours (/work/noaa/nems/tufuk/COASTAL/coastal_ian_atm2ww3_intel_1). I am still not sure ww3_shel.inp is correct or not. I'll try to run the same case by removing wind file from run directory and use nml to be sure it is getting wind from cdeps.

@yunfangsun
Copy link
Collaborator

@uturuncoglu Could you please change the permission of the folder /work/noaa/nems/tufuk/COASTAL/coastal_ian_atm2ww3_intel_1, I can't get access to it.

@uturuncoglu
Copy link
Collaborator Author

@yunfangsun i did it.

@pvelissariou1
Copy link
Collaborator

@uturuncoglu, @yunfangsun, @saeed-moghimi-noaa Thank you so much for your response on such a short notice and over the weekend. Your help is greatly appreciated. @yunfang Thank you for spending time to resolve all the issues.
Yunfang, could you please document all the steps you followed to setup the Ian application inside your RT folder and UFS-Coastal (configuration, compilation and run). I am doing the same thing with my simulations but configuring them in such a way so they can run outside the RT folder and without running them as a test case within UFS-Coastal. At the end I want to combine all the steps and requirements in one document. I'll report back on the progress and issues from all the simulations with and without waves.

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 Agree. Documenting each step will help us to implement the workflow.

@yunfangsun
Copy link
Collaborator

@uturuncoglu my atm2sch2ww3 configuration is at /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/rt_11601_atm2sch2ww3/coastal_ian_atm2sch2ww3_intel_1
Thank you for your great help!

@uturuncoglu
Copy link
Collaborator Author

uturuncoglu commented Jan 16, 2024 via email

@yunfangsun
Copy link
Collaborator

@uturuncoglu thank you

@uturuncoglu
Copy link
Collaborator Author

@yunfangsun The DATM+WW3 run is finished. It is in /work/noaa/nems/tufuk/COASTAL/coastal_ian_atm2ww3_intel_1. Please check the results and let me know if you need any change. I'll run other job soon.

@yunfangsun
Copy link
Collaborator

@uturuncoglu the results file permission is denied, can you please change the permissions of the files? Thank you.

@pvelissariou1
Copy link
Collaborator

@uturuncoglu Thank you

@uturuncoglu
Copy link
Collaborator Author

@yunfangsun Hercules is down now but maybe you could reach from Orion. I fixed the permission.

@yunfangsun
Copy link
Collaborator

@uturuncoglu thank you

@yunfangsun
Copy link
Collaborator

New configuration of stand alone WW3:
for regression test:
coastal_ike_shinnecock_ww3

WAV_attributes::
Verbosity = 0
DumpFields = false
ProfileMemory = false
merge_import = .false.
mesh_wav = mesh.shinnecock.cdf5.nc
multigrid = false
gridded_netcdfout = true
diro = "."
logfile = wav.log
standalone = true
::

@yunfangsun yunfangsun mentioned this issue Jun 17, 2024
Closed
@janahaddad janahaddad added old_integration and removed enhancement New feature or request labels Sep 23, 2024
@github-project-automation github-project-automation bot moved this from Todo to Done in ufs-coastal project Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@uturuncoglu uturuncoglu

@yunfangsun yunfangsun

Labels
old_integration
Projects
ufs-coastal project
Status: Done
Milestone
No milestone
Development

No branches or pull requests
6 participants
@uturuncoglu @janahaddad @aliabdolali @pvelissariou1 @saeed-moghimi-noaa @yunfangsun