Compiler error fms 2023.03 with gnu 12.2.0 (coupler_types.F90 - fms_io_register_restart_field)

[climbfuji]

Is your question related to a problem? Please describe.
I am having trouble building tag 2023.03 on the new NCAR Supercomputer Derecho with gnu 12.2.0. The config options are:

-DGFS_PHYS:BOOL=ON
-DOPENMP:BOOL=ON
-DENABLE_QUAD_PRECISION:BOOL=ON
-DWITH_YAML:BOOL=OFF
-DCONSTANTS:STRING=GFS
-D32BIT:BOOL=ON
-D64BIT:BOOL=ON
-DFPIC:BOOL=ON
-DUSE_DEPRECATED_IO:BOOL=ON
-DCMAKE_C_COMPILER:STRING=/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/bin/mpicc
-DCMAKE_CXX_COMPILER:STRING=/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/bin/mpicxx
-DCMAKE_Fortran_COMPILER:STRING=/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/bin/mpif90
-DCMAKE_Fortran_FLAGS:STRING=-fallow-argument-mismatch

(I believe we don't have to worry about the 9.1 in the cray-mpich compiler wrapper path). The error is:

/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/coupler/coupler_types.F90:4207:37:

 4207 |         var%bc(n)%field(m)%id_rest = fms_io_register_restart_field(rest_file,&
      |                                     1
Error: There is no specific function for the generic 'register_restart_field' at (1)

and the corresponding compile command is

[93%] Building Fortran object CMakeFiles/fms_r8_f.dir/coupler/coupler_types.F90.o /opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/bin/mpif90
-DENABLE_QUAD_PRECISION -DGFS_CONSTANTS -DGFS_PHYS -DHAVE_GETTID -DINTERNAL_FILE_NML -Duse_deprecated_io -Duse_libMPI -Duse_netCDF
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/fms
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/fms/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/fms2_io/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/string_utils/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/mpp/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/column_diagnostics/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/monin_obukhov/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/sat_vapor_pres/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/horiz_interp/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/diag_integral/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/random_numbers/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/diag_manager/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/constants4
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/topography/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/axis_utils/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/mosaic2/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/constants
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/astronomy/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/field_manager/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/time_interp/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/tracer_manager/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/tridiagonal/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/interpolator/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/coupler/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/data_override/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/amip_interp/include
-I/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/envs/unified-env/install/gcc/12.2.0/netcdf-fortran-4.6.0-sncwq43/include
-fallow-argument-mismatch -fcray-pointer -fconvert=big-endian -ffree-line-length-none
-fno-range-check -fbacktrace -O3 -funroll-all-loops -finline-functions -Jinclude_r8 -fPIC -fallow-argument-mismatch -w -fopenmp
-c /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/cache/build_stage/spack-stage-fms-2023.03-jg2y5u4wop6gixbep4secdjqm64r4ri2/spack-src/coupler/coupler_types.F90
-o CMakeFiles/fms_r8_f.dir/coupler/coupler_types.F90.o

I looked for open and closed issues about this error, but couldn't find anything. Before I spend a lot of time debugging this, is this something you've encountered and/or do you have pointers to fix that?

Describe what you have tried
See above

[bensonr]

@climbfuji - what is your default real being defined as here? It looks like it is a default 32bit (kind=4) but I can't tell for sure.

[climbfuji]

@climbfuji - what is your default real being defined as here? It looks like it is a default 32bit (kind=4) but I can't tell for sure.

Yes, that's what it looks like. I'll try again manually with only one make job to get better output, but it looks like the r4 version of coupler_types.F90 compiles ok, but the r8 does not. And the compile command cited above does not promote reals to r8 (it does have -Jinclude_r8 but I am not entirely sure if that is meant to do the trick).

[bensonr]

With the latest FMS, we went to mixed-precision support throughout FMS which means you only have to compile up one version of the library and it should support your mixed-precision needs. The one place that wasn't touched was the deprecated mpp/fms I/O. Because you chose to use default real(kind=r4), you'll need to still include the -DOVERLAD_R8 for the older I/O to work properly. If you compile with -r8 by default, the logic I was looking at suggests there is no real(kind=4) support within the deprecated I/O and that case may cause issues. I'll have to see if anyone in the team tested that combination.

[climbfuji]

Hmm, very strange. We use exactly the same code and options to install fms 2023.03 across platforms, the only one that failed so far is Derecho.

[bensonr]

You're right, that is strange as I'd expect it to behave the same across all platforms. I'd suggest double checking the other versions to see what exact compiler options (not CMake option sets) were used during the build. If you've used gnu 12.2 elsewhere, then the only thing I can think of is an issue with the v12.2 install on Derecho.

[climbfuji]

Sure enough, we compiled it successfully on Hercules with the same version of gnu. But the compile command for the particular file is different. On Derecho (see above):

...
-fallow-argument-mismatch -fcray-pointer -fconvert=big-endian -ffree-line-length-none
-fno-range-check -fbacktrace -O3 -funroll-all-loops -finline-functions -Jinclude_r8 -fPIC -fallow-argument-mismatch -w -fopenmp
...

On Hercules:

...
-fallow-argument-mismatch -fcray-pointer -fconvert=big-endian -ffree-line-length-none
-fno-range-check -fbacktrace -O3 -funroll-all-loops -finline-functions -Jinclude_r8 -fPIC -fdefault-real-8 -fdefault-double-8 -fopenmp
...

Somehow, -fdefault-real-8 -fdefault-double-8 is replaced with -fallow-argument-mismatch -w on Derecho. I wonder if that's one of the hidden features that the sysadmins have baked into their version of the compiler wrappers. At least I know where to look next!

[bensonr]

Looks like you are on track to figuring this one out. Should we close out the issue?

[climbfuji]

I'd prefer to wait until I've figured it out, but it's up to you. Thanks for your help by the way!

[thomas-robinson]

Do you need to use the deprecated IO? -DUSE_DEPRECATED_IO:BOOL=ON

[climbfuji]

Do you need to use the deprecated IO? -DUSE_DEPRECATED_IO:BOOL=ON

Yes, unfortunately we'll need this for a while for UFS and also for JEDI

[climbfuji]

@bensonr @thomas-robinson I am glad we kept this issue open. I found the culprit, it's in FMS' very own top-level CMakeLists.txt:

FMS/CMakeLists.txt

Lines 348 to 354 in 527a42f

	# Check if gnu 10 or higher with mpich
	if ( CMAKE_Fortran_COMPILER_VERSION MATCHES "1[0-9]\.[0-9]*\.[0-9]*" AND CMAKE_Fortran_COMPILER_ID MATCHES "GNU")
	if(MPI_C_COMPILER MATCHES ".*mpich.*" )
	message(STATUS "Adding -fallow-argument-mismatch flag to compile with GCC >=10 and MPICH")
	set_target_properties(${libTgt}_f PROPERTIES COMPILE_FLAGS "-fallow-argument-mismatch -w")
	endif()
	endif()

Setting this flags overwrites the previous compile flags that contain -fdefault-real-8 -fdefault-double-8. In my opinion, this block shouldn't be there. The user's build system should take care of adding -fallow-argument-mismatch if needed (and spack does that for us). But if you want it to be here, then it needs to append to the existing flags, not overwrite them. I removed that block of code and compiled on Derecho without problems. On Hercules, this block didn't trigger because we use [email protected] with [email protected] (which is basically mpich under the hood, but not recognized as such).

[bensonr]

@climbfuji - thanks for tracking this down. Since not every build system adds this option by default, could you make the change to append the argument, as was intended, and submit a PR?

[climbfuji]

@bensonr Please see #1418. It would be good if someone other than me tested this change.

[bensonr]

@climbfuji - appreciate the quick turnaround. We'll test the updates on a few systems we have here to ensure things work as expected.