Dynamic load of nvidia libs

[WilliamTambellini]

Hi all
As today, if ngraph is compiled with NGRAPH_GPU_ENABLE=TRUE (which ofcourse needs a nvidia ecosystem installed) then libngraph.so is hardly linked with few nvidia libs:
`ldd /tmp/ngraph/lib/libngraph.so

libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1
libnvrtc.so.8.0 => /usr/local/cuda-8.0/targets/x86_64-linux/lib/libnvrtc.so.8.0
libcublas.so.8.0 => /usr/local/cuda-8.0/targets/x86_64-linux/lib/libcublas.so.8.0
libcudnn.so.7 => /usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.7
libnvidia-fatbinaryloader.so.375.74 => /usr/lib/nvidia-375/libnvidia-fatbinaryloader.so.375.74
...
`

For production & deployment purpose, this is not practical because such a libngraph build wont be able to be loaded at all on machines without the nvidia gpu ecosystem, ie cpu only.

Thanks to the fact that the ngraph arch is pretty clean, this is only needed for some C function calls needed in the gpu backend only (ngraph/src/ngraph/runtime/gpu) : cublasCreate, cudnnCreate, cublasSetPointerMode, cublasDestroy, cuInit, cuDeviceGet, cuCtxCreate, nvrtcCreateProgram, nvrtcCompileProgram, nvrtcGetPTXSize, nvrtcGetPTX, nvrtcDestroyProgram, cuLaunchKernel, ...

It could be possible to dynamically load these libs using standard posix 'dl' :
http://man7.org/linux/man-pages/man3/dlopen.3.html

The PyTorch team seems to also try to do such :
pytorch/pytorch#3395

Before moving forward, would you be open to this idea or do you see any red flags ?

Kind
W.

[diyessi]

Thankyou for your suggestion. We think we will do something like this when our GPU support is further along.

At this time we do not yet have a contributor license agreement (CLA) procedure in place. Until that is resolved, we won't be able to accept outside contributions.

[WilliamTambellini]

Thank you, you are welcome.
FYI, it seems that the CUEW library has implemented the dynamic loading wrapping of most of the nvidia API (nvrtc, cuda, ...) :
https://github.com/CudaWrangler/cuew

[WilliamTambellini]

Hi @diyessi
I have seen a minimalist POC on devtalk :
main.cpp.txt
Could you tell at least if you would prefer :

• an implementation using CudaWrangler (cuew)
• an implementation not using any new external libraries

Kind

[diyessi]

External libraries need to go through an approval process that looks into licensing, IP, security, etc.
We will need to do the dynamic loading before we can do binary releases.
@csullivan can you give an update on dynamic loading?

[WilliamTambellini]

If I may emit 2 warnings/advices :

• cuew does not wrap cudnn (yet ?) but does provide a mature cross platform dynamic loading framework (Linux, Mac and even Windows)
• most cudaXxx from cudart (cuda runtime) should not need to be wrapped IF linking ngraph with cudart static (${CUDA_cudart_static_LIBRARY}, usually used if CUDA_USE_STATIC_CUDA_RUNTIME=ON)

[csullivan]

Thanks @WilliamTambellini. As you suggest, we'd like to dynamically load nvrtc, cuda, cublas, and cudnn all in one go. CUEW's cross-platform support is attractive and may be a good starting point if cudnn can be added in a straightforward way. Per your original question, we are still awaiting information on contributors licensing but hope to have that good to go soon. If you want to take a stab at it in the mean time, I'm more than happy to review / merge once we have our contributors license up. Otherwise, it's on our roadmap and I will provide updates here when this is underway internally.

[WilliamTambellini]

Thank you @csullivan : I see enough reasons to wait a little :

• the cuew team is not sure yet how to handle cudnn in cuew : the fact that cudnn has its own version in parallell to cuda is adding some complexity :
  Wrap the main cudnn functions CudaWrangler/cuew#7
• I see cuew is targeting cuda 9 but looks like ngraph is on cuda 8 atm iirc :
  CUDA 9 support #947
• ngraph CLA not online yet

To be continued.

[WilliamTambellini]

Early result from the cudnn support in cuew :
https://github.com/Nazg-Gul/cuew/blob/cudnn_experiment/include/cuew.h
this header is generated so all the cudnn api should be in it.
@csullivan @diyessi : could you check if at least you see all the api ngraph needs ?

[csullivan]

@WilliamTambellini the update looks promising, I'll give it a review, thanks