How GPU works with Fortran interface with pgi compiler

[pyrimidine]

Dear AMReX developers,

Sorry for disturbing you. I got several questions and a few comments based on the lasted version of AMReX.

As the lastest version of the hpc-sdk( previously named PGI) has already remove the flag name '-Mcuda', the previous GNUmake system cannot compile the code. Therefore, it is possible to handle this issue by modify the pgi.mak and nvhpc.mak file by changing:
-Mcuda to -cuda -gpu

However, this still have some potential problem with related to the combination of fortran compiler. The error message shows as:

Compiling AMReX_fi_mpi_mod.F90 ... mpif90 -O2 -fast -cuda -gpu=cc80 -cuda -gpu=lineinfo -cuda -gpu=ptxinfo CUDA_HOME=:/usr/local/cuda -cuda -gpu=maxregcount:255 -Mdclchk -noacc -module tmp_build_dir/o/2d.pgi.MPI.CUDA.EXE -Itmp_build_dir/o/2d.pgi.MPI.CUDA.EXE -DBL_LANG_FORT -DAMREX_LANG_FORT -DBL_USE_MPI -DAMREX_USE_MPI -DAMREX_USE_CUDA -DAMREX_USE_GPU -DBL_COALESCE_FABS -DAMREX_GPU_MAX_THREADS=256 -DAMREX_USE_GPU_RDC -DBL_SPACEDIM=2 -DAMREX_SPACEDIM=2 -DBL_FORT_USE_UNDERSCORE -DAMREX_FORT_USE_UNDERSCORE -DBL_Linux -DAMREX_Linux -DAMREX_PARTICLES -DNDEBUG -DOMPI_SKIP_MPICXX -DBL_USE_F_INTERFACES -Itmp_build_dir/s/2d.pgi.MPI.CUDA.EXE -I. -I. -I../lapp-wenq/src -I../amrex/Src/Base -I../amrex/Src/Base/Parser -I../amrex/Src/Boundary -I../amrex/Src/AmrCore -I../amrex/Src/Particle -I../amrex/Src/F_Interfaces/Base -I../amrex/Src/F_Interfaces/AmrCore -I../amrex/Src/F_Interfaces/Particle -I :/usr/local/cuda/include -c ../amrex/Src/F_Interfaces/Base/AMReX_fi_mpi_mod.F90 -o tmp_build_dir/o/2d.pgi.MPI.CUDA.EXE/AMReX_fi_mpi_mod.o pgfortran-Warning-unknown driver variable assigned on the command line: CUDA_HOME=:/usr/local/cuda NVFORTRAN-F-0004-Corrupt or Old Module file /usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi/mpi.mod (../amrex/Src/F_Interfaces/Base/AMReX_fi_mpi_mod.F90: 3) NVFORTRAN/x86-64 Linux 24.11-0: compilation aborted make: *** [../amrex/Tools/GNUMake/Make.rules:329：tmp_build_dir/o/2d.pgi.MPI.CUDA.EXE/AMReX_fi_mpi_mod.o] error 2

[pkufourier]

As II know, the cuda compilers are the most compatible with GCC series (used by default). Changing to other compilers (I have tested with Intel compiler) seems to benefit little on performance. Thus I am not sure why you want to use PGI for cuda applications. Waiting for @WeiqunZhang for a declaration.

[WeiqunZhang]

We do not use CUDA Fortran in AMReX. In our Fortran interface, the GPU kernels are in C++.

If you want to use CUDA Fortran, I think it should still be compatible, though the make system may need to be updated. If you have a working setup, please feel free to submit a PR.

This is in your error message mpif90 -O2 -fast -cuda -gpu=cc80 -cuda -gpu=lineinfo -cuda -gpu=ptxinfo CUDA_HOME=:/usr/local/cuda. Not sure how CUDA_HOME=:/usr/local/cuda gets there. What's =:? Is that something for GNU Make or nvfortran?