How to save and load model output run on GPU?

[iuryt]

I couldn't see an example from the documentation that uses GPU and write/load model outputs. I just realized that I was getting error while loading the jld2 output from a GPU run, but the same worked for CPU. I understand that the matrices are differently defined for each architecture and thought that maybe the way to write, or load the data might need some tweak. Is that any special advices for writing and loading GPU-run outputs?

[navidcy]

I run on GPUs and save using JLD2 in a "normal" fashion.
Can you post your example here?

[iuryt]

From here it looks like it occurs when

a kernel is being invoked with CPU arrays as arguments

But all initial and boundary conditions are defined using functions. Not sure this is the problem.

[iuryt]

Wait, could it be related to the fact Oceananigans gives this warning at the import?

┌ Warning: You appear to be using MPI.jl with the default MPI binary on a cluster.
│ We recommend using the system-provided MPI, see the Configuration section of the MPI.jl docs.
└ @ MPI /home/isimoesdesousa/.julia/packages/MPI/08SPr/deps/deps.jl:15
┌ Info: Oceananigans will use 4 threads
└ @ Oceananigans /home/isimoesdesousa/.julia/packages/Oceananigans/CuznF/src/Oceananigans.jl:243
┌ Warning: This version of CUDA.jl only supports NVIDIA drivers for CUDA 10.1 or higher (yours is for CUDA 10.0.0)
└ @ CUDA /home/isimoesdesousa/.julia/packages/CUDA/DL5Zo/src/initialization.jl:38
┌ Warning: The NVIDIA driver on this system only supports up to CUDA 10.0.0.
│ For performance reasons, it is recommended to upgrade to a driver that supports CUDA 11.2 or higher.
└ @ CUDA /home/isimoesdesousa/.julia/packages/CUDA/DL5Zo/src/initialization.jl:42
┌ Warning: This version of CUDA.jl only supports CUDA 10.1 or higher (your toolkit provides CUDA 10.0.0)
└ @ CUDA /home/isimoesdesousa/.julia/packages/CUDA/DL5Zo/src/initialization.jl:68

[navidcy]

not sure if this CUDA-related warning is related.
But what version Oceananigans you using? v0.72.4?

[iuryt]

You are right! I was using v72.0. I updated to v72.4 and now it's working. How should I frame an answer to this post?
"There is no pre/post-processing needed and CPU/GPU conversion should work right away. If it's not working, this is either an outdated version of the code or a bug."?

[navidcy]

v0.72.0...v0.72.4

I believe the critical change in between these two versions was via #2346