UKMO Cray bugfixes #105

ukmo-ccbunney · 2019-09-26T15:29:00Z

Changes required to run full regtest suite on Cray HPC using the crayftn compiler.

Adds the following fearures:

New comp/link scripts for matrix generation file. (d85e386)

And addresses the following issues:

Erroneous commas in w3uostmd and w3iogrmd #95 Erroneous commas in w3uostmd and w3iogrmd (cf1ad39)
Invalid Fortran unit number in w3uostmd #96 Invalid Fortran unit number in w3uostmd (ad28d89)
regtest ww3_tp2.10 instability #93 regtest ww3_tp2.10 instability (0d17768)
OMP switches missing in w3psmcmd #92 OMP switches missing in w3psmcmd (6a3864d)
Non advancing WRITE statement causing "record too long" error #98 Non advancing WRITE statement causing "record too long" error (5362e11)
Typo in WMINITMD with MPRF switch #94 Typo in WMINITMD with MPRF switch (2b41012)
Removed extraneous brackets from implied DO loop read statment (fdfd93e)

…hanged unit number from 100 to 110 to avoid Cray reserved unit number range.

…voids record length overflow for large dimension grids in WRITE statment with non-advancing I/O.

* Added cray_xc to the cmplr.env and w3_setup scripts. * Added cray_xc.CCE (Cray Compiler Envrionment) specific comp and link scripts. * Explicitly disable OMP in cray comp scripts for non-OMP compilation (Cray compiler enables by default). Note that is still enabled in link script as OMP library is always required by SCRIP code. * Updated matrix_ukmo_cray to run everything on a shared node using mpiexec - compilation is not efficient on compute nodes. * Add -eg switch (allows use of GOTO jumps into DO loops for SEC1 switch * Addition of GNU compilers on Cray architecture.

ajhenrique · 2019-09-26T15:56:55Z

Thanks @ukmo-ccbunney. We are moving away from the platform/compiler-specific comp and link files towards using the cmplr.env and generic comp.tmpl and link.tmpl. Does this PR include changes to those files that would allow usage of the cmplr.env, comp.tmpl and link.tmpl without referring to any other external script/file? This would be a requirement for accepting this PR. In the near future we will eliminate all platform/compiler-specific comps and links from the package.

ukmo-ccbunney · 2019-09-26T18:50:13Z

@ajhenrique - yes the PR does include an update to cmplr.env to include "cray_xc". Would you prefer it if I removed the comp.cray_xc.CCE and link.cray_xc.CCE scripts?

compilers to complain.

ajhenrique · 2019-10-11T15:45:29Z

@ukmo-ccbunney please merge the auth repo develop into the branch associated with this PR. We can then move to get it merged into the NOAA-EMC/WW3 develop. Thanks!

ukmo-ccbunney · 2019-10-11T17:32:24Z

Develop branch has been merged in: 56857cd

Do you want me to rerun the regtests matrix?

aliabdolali · 2019-10-25T13:38:40Z

Hi @ukmo-ccbunney @ajhenrique
I ran the regression tests at our end and came with the following differences:

For ww3_tp2.10, the time step has changed from 450 90 90 90 to 450 60 60 60. Was it on purpose? As a result, we have different answers.
The outputs for ww3_ts1/work_ST6 are different. Do we expect these differences?

The rests look ok to me.
Comparison attached.
Regards
AA
COMP_UKMO.zip

ukmo-ccbunney · 2019-10-25T14:39:31Z

Hi @aliabdolali

The differences in ww3_tp2.10 are expeced as the timestep was too large resulting in the model going unstable (when I ran the test originally, the output was mostly NaN values!)

However, I cannot see reason why the output ww3_ts1 should be different. I have not changed anything to do with the source terms. I notice that the tab*.ww3 files are identical in the work directories, so the differences must be very small. I can't see why only the ww3_ts1 test should be affected. Could it be linked to the fact that the mww3* tests are not bit comparible?

Chris.

ajhenrique · 2019-10-25T16:05:21Z

Hi @aliabdolali could you provide details of the differences you encountered in ww3_ts1?

aliabdolali · 2019-10-25T16:18:45Z

@ukmo-ccbunney @ajhenrique
For ww3_tp2.10, we will go with smaller time steps.
For ww3_ts1, I see the differences in ww3.196806.nc
Please check the end of matrixDiff.out in the zip folder.
Ali

ajhenrique · 2019-10-25T19:24:49Z

@ajhenrique - yes the PR does include an update to cmplr.env to include "cray_xc". Would you prefer it if I removed the comp.cray_xc.CCE and link.cray_xc.CCE scripts?

@ukmo-ccbunney that would be great if it doesn't affect your work. The idea will be to soon remove all individual comp and link files, not adding new ones unless they are essential is a good way to start cleaning up! Thanks.

ukmo-ccbunney · 2019-10-28T09:29:36Z

@aliabdolali, @ajhenrique
Regarding the differences in the ww3_ts1 regtest: I have just run the all the ww3_ts1 tests on the develop branch code and compared them with the results from the bf_ukmo_cray branch and the results are identical.

ukmo-ccbunney · 2019-10-28T10:15:07Z

@ajhenrique - yes the PR does include an update to cmplr.env to include "cray_xc". Would you prefer it if I removed the comp.cray_xc.CCE and link.cray_xc.CCE scripts?

@ukmo-ccbunney that would be great if it doesn't affect your work. The idea will be to soon remove all individual comp and link files, not adding new ones unless they are essential is a good way to start cleaning up! Thanks.

@ajhenrique OK - I will remove the comp and link files for the Cray XC and add the cray_xc.GNU compiler setup to the cmplr.env file (I've already added the cray_xc.CCE one).

One question regarding the compiler setups - why are the Intel and GNU compiler options set up to hardcode the byte order to "big-endian"? I would prefer to keep the native (little-endian in our case) byte order when compiling with the GNU compiler. Are you happy if I modify the GNU section of cmplr.env slightly to make -fconvert=big-endian a system dependent option (native for cray_xc, big-endian for everything else)?

ghost · 2019-10-28T11:05:38Z

Hi @ukmo-ccbunney, from what I remember about the big-endian option, the aim was to avoid mixing the way binary files are created by ww3 on differents file systems to be able to safely exchange binary files. So it would be better to keep it as default except if it slow down your computation time.

ukmo-ccbunney · 2019-10-28T11:42:47Z

@maccensi I note that profiling (-p switch) is always enabled via the "common options". Is this intentional?
We probably would not want that enabled for operational executables (there must be a perfomance hit to having it on?)

ghost · 2019-10-28T12:39:44Z

There is almost no extra overhead using -p switch since it's just camping time functions. At least from I tested on our system and from what I read on the documentation about flat profiling but it could be good to test it again, I will do it

ukmo-ccbunney · 2019-10-28T12:48:35Z

There is almost no extra overhead using -p switch since it's just camping time functions.
Ok - thanks @mickaelaccensi.

ajhenrique · 2019-10-28T14:07:11Z

@ukmo-ccbunney @mickaelaccensi I've read here and there that there may be a performance hit in runs over long periods of time using massive resources. I am currently running a test with our coupled system with and without the profiling flag, and will report back to this thread as well.

ajhenrique · 2019-10-28T14:25:17Z

Hi @ukmo-ccbunney, from what I remember about the big-endian option, the aim was to avoid mixing the way binary files are created by ww3 on differents file systems to be able to safely exchange binary files. So it would be better to keep it as default except if it slow down your computation time.

@ukmo-ccbunney, I agree with @mickaelaccensi . Please indicate if this results in overheads for your system.

ajhenrique · 2019-10-28T17:53:11Z

@mickaelaccensi @ukmo-ccbunney I've completed a first set of tests running our coupled FV3-WW3 ensemble system. This is currently the operational cube sphere for FV3, and a mosaic of 3 grids in WW3 at 1/4 deg resolution (Arctic, core and Antarctic), with 30 perturbed members + 1 control, which are run out to 16 days. I used a Dell machine with 28 cores per node, the run had 14 ppn and 2 OMP threads, and a total of 38 nodes (532 MPI processes per member). With/without the -p flag the control ran ~61min/57min, and the perturbed members averaged ~69min/66min. There is small but noticeable shaving off of time in our system when -p is removed.

JessicaMeixner-NOAA · 2019-10-28T17:55:28Z

@ajhenrique Did you remove the -p just for WW3 or both FV3 and WW3?

ajhenrique · 2019-10-28T18:05:11Z

@ajhenrique Did you remove the -p just for WW3 or both FV3 and WW3?

@JessicaMeixner-NOAA, Only WW3.

JessicaMeixner-NOAA · 2019-10-28T18:11:33Z

Adding more WW3 pets might further reduce the runtime since it seems that WW3 is potentially what's holding back the total timing. Or if you now hit the FV3 time, that -p could even further reduce your runtime from the amount you already saw. Either way it looks like good news for the GEFSv12 timings!

ghost · 2019-10-28T21:29:18Z

that's a great news ! so it would be good to put the '-p-' option in a specific profiling section, with _prof suffix like it's done for debugging option with suffix _debug

ajhenrique · 2019-10-29T00:00:05Z

@mickaelaccensi that would be preferable. I can do that in a commit to our GEFS_v12 implementation branch and merge it back to develop.

ukmo-ccbunney · 2019-10-29T08:15:38Z

Excellent! Sounds like an easy win. :)

ukmo-ccbunney · 2019-10-29T10:11:52Z

I've just pushed up a commit (c134eff ) that removes the Met Office specific comp/link files and updates the cmplr.env with build configs for our Cray HPC using craytfn and gfortran.

ajhenrique · 2019-10-30T00:54:38Z

@aliabdolali what is the final recommendation for this PR? I've added both you and @JessicaMeixner-NOAA as reviewers. I look forward to both reviews.

aliabdolali · 2019-10-30T00:57:34Z

@ajhenrique The suspicious regression test behaves differently on our machine than UKMO's machine. As you can see, Chris has tested that test with the code before and after development and they were identical. So we either need to move on or we investigate more. The rest of the tests are OK.

ajhenrique · 2019-10-30T03:52:01Z

@ukmo-ccbunney @aliabdolali @JessicaMeixner-NOAA I reran the matrix for ww3_ts1 and had the following output in matrixCompSummary.out
test(s) :
ww3_ts1

********************* non-identical cases ****************************

************************ identical cases *****************************
ww3_ts1/./work_ST6
ww3_ts1/./work_ST1_RWND
ww3_ts1/./work_ST4_WRT
ww3_ts1/./work_ST4_GMD
ww3_ts1/./work_ST4_TSA
ww3_ts1/./work_ST4
ww3_ts1/./work_ST1
ww3_ts1/./work_ST2
ww3_ts1/./work_ST3

I'm not sure what happened in the previous matrix run, but I'll consider this a done deal and will be ready to merge asap, given @aliabdolali accepted his review, if @JessicaMeixner-NOAA concurs in her review.

ajhenrique · 2019-10-30T13:47:01Z

Thanks everybody in this thread for your comments. @ukmo-ccbunney please close issues related to this PR, and make sure you back up some of the discussions above that may be useful elsewhere. Merging!

ukmo-ccbunney · 2019-10-30T13:49:45Z

Fantastic. Thanks everyone.

* Removed incorrectly placed commas. * Added UNIT_AB variable to hold unit number for LOAD_ALPHABETA call. Changed unit number from 100 to 110 to avoid Cray reserved unit number range. * Fixed instability in ww3_tp2.10 regtest by decreasing CFL timestep * Added !/OMPG switches to OMP directives. * Added RECL specifier to OPEN statement enabled by switch /O2c. This avoids record length overflow for large dimension grids in WRITE statment with non-advancing I/O. * Changes for compilation and regtesting on UK Met Office Cray HPC: * Added cray_xc to the cmplr.env and w3_setup scripts. * Added cray_xc.CCE (Cray Compiler Envrionment) specific comp and link scripts. * Explicitly disable OMP in cray comp scripts for non-OMP compilation (Cray compiler enables by default). Note that is still enabled in link script as OMP library is always required by SCRIP code. * Updated matrix_ukmo_cray to run everything on a shared node using mpiexec - compilation is not efficient on compute nodes. * Add -eg switch (allows use of GOTO jumps into DO loops for SEC1 switch * Addition of GNU compilers on Cray architecture. * Fixed typo in wminitmd.ftn as rasied in NOAA-EMC#94 * Removed extra brackets around variable list that were causing some compilers to complain. * Removed UKMO comp/link scripts and updated cmplr.env accordingly

ukmo-ccbunney added 7 commits September 24, 2019 20:25

Removed incorrectly placed commas.

cf1ad39

Added UNIT_AB variable to hold unit number for LOAD_ALPHABETA call. C…

ad28d89

…hanged unit number from 100 to 110 to avoid Cray reserved unit number range.

Fixed instability in ww3_tp2.10 regtest by decreasing CFL timestep

0d17768

Added !/OMPG switches to OMP directives.

6a3864d

Added RECL specifier to OPEN statement enabled by switch /O2c. This a…

5362e11

…voids record length overflow for large dimension grids in WRITE statment with non-advancing I/O.

Fixed typo in wminitmd.ftn as rasied in NOAA-EMC#94

2b41012

Removed extra brackets around variable list that were causing some

fdfd93e

compilers to complain.

ukmo-ccbunney mentioned this pull request Oct 7, 2019

Remove incorrectly placed commas #97

Closed

Merge branch 'develop' into bf_ukmo_cray

56857cd

Merge branch 'develop' into bf_ukmo_cray

18206ec

Removed UKMO comp/link scripts and updated cmplr.env accordingly

c134eff

ajhenrique requested review from aliabdolali and JessicaMeixner-NOAA October 30, 2019 00:54

aliabdolali approved these changes Oct 30, 2019

View reviewed changes

JessicaMeixner-NOAA approved these changes Oct 30, 2019

View reviewed changes

ajhenrique merged commit 39d1fb7 into NOAA-EMC:develop Oct 30, 2019

ajhenrique mentioned this pull request Oct 31, 2019

NOAA-EMC/WW3 develop branch significant update message #122

Closed

ukmo-ccbunney deleted the bf_ukmo_cray branch November 27, 2019 10:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UKMO Cray bugfixes #105

UKMO Cray bugfixes #105

ukmo-ccbunney commented Sep 26, 2019 •

edited

Loading

ajhenrique commented Sep 26, 2019 •

edited

Loading

ukmo-ccbunney commented Sep 26, 2019

ajhenrique commented Oct 11, 2019

ukmo-ccbunney commented Oct 11, 2019

aliabdolali commented Oct 25, 2019

ukmo-ccbunney commented Oct 25, 2019 •

edited

Loading

ajhenrique commented Oct 25, 2019

aliabdolali commented Oct 25, 2019

ajhenrique commented Oct 25, 2019

ukmo-ccbunney commented Oct 28, 2019

ukmo-ccbunney commented Oct 28, 2019

ghost commented Oct 28, 2019 •

edited by ghost

Loading

ukmo-ccbunney commented Oct 28, 2019

ghost commented Oct 28, 2019

ukmo-ccbunney commented Oct 28, 2019

ajhenrique commented Oct 28, 2019

ajhenrique commented Oct 28, 2019

ajhenrique commented Oct 28, 2019

JessicaMeixner-NOAA commented Oct 28, 2019

ajhenrique commented Oct 28, 2019 •

edited

Loading

JessicaMeixner-NOAA commented Oct 28, 2019

ghost commented Oct 28, 2019

ajhenrique commented Oct 29, 2019

ukmo-ccbunney commented Oct 29, 2019

ukmo-ccbunney commented Oct 29, 2019

ajhenrique commented Oct 30, 2019 •

edited

Loading

aliabdolali commented Oct 30, 2019

ajhenrique commented Oct 30, 2019 •

edited

Loading

ajhenrique commented Oct 30, 2019 •

edited

Loading

ukmo-ccbunney commented Oct 30, 2019

UKMO Cray bugfixes #105

UKMO Cray bugfixes #105

Conversation

ukmo-ccbunney commented Sep 26, 2019 • edited Loading

ajhenrique commented Sep 26, 2019 • edited Loading

ukmo-ccbunney commented Sep 26, 2019

ajhenrique commented Oct 11, 2019

ukmo-ccbunney commented Oct 11, 2019

aliabdolali commented Oct 25, 2019

ukmo-ccbunney commented Oct 25, 2019 • edited Loading

ajhenrique commented Oct 25, 2019

aliabdolali commented Oct 25, 2019

ajhenrique commented Oct 25, 2019

ukmo-ccbunney commented Oct 28, 2019

ukmo-ccbunney commented Oct 28, 2019

ghost commented Oct 28, 2019 • edited by ghost Loading

ukmo-ccbunney commented Oct 28, 2019

ghost commented Oct 28, 2019

ukmo-ccbunney commented Oct 28, 2019

ajhenrique commented Oct 28, 2019

ajhenrique commented Oct 28, 2019

ajhenrique commented Oct 28, 2019

JessicaMeixner-NOAA commented Oct 28, 2019

ajhenrique commented Oct 28, 2019 • edited Loading

JessicaMeixner-NOAA commented Oct 28, 2019

ghost commented Oct 28, 2019

ajhenrique commented Oct 29, 2019

ukmo-ccbunney commented Oct 29, 2019

ukmo-ccbunney commented Oct 29, 2019

ajhenrique commented Oct 30, 2019 • edited Loading

aliabdolali commented Oct 30, 2019

ajhenrique commented Oct 30, 2019 • edited Loading

ajhenrique commented Oct 30, 2019 • edited Loading

ukmo-ccbunney commented Oct 30, 2019

ukmo-ccbunney commented Sep 26, 2019 •

edited

Loading

ajhenrique commented Sep 26, 2019 •

edited

Loading

ukmo-ccbunney commented Oct 25, 2019 •

edited

Loading

ghost commented Oct 28, 2019 •

edited by ghost

Loading

ajhenrique commented Oct 28, 2019 •

edited

Loading

ajhenrique commented Oct 30, 2019 •

edited

Loading

ajhenrique commented Oct 30, 2019 •

edited

Loading

ajhenrique commented Oct 30, 2019 •

edited

Loading