Model loading is too slow with onnxruntime-gpu

[korhun]

Using the cpu version, I have no problem. Loading onnx models using "InferenceSession" with onnxruntime-gpu takes >102 seconds for the first model. If we load more than one models, the others take no time at all; we only wait for the first call of "onnxruntime.InferenceSession(onnx_fn)" I have tried with yolov3, yolov3-tiny, yolov4, yolov5, ssd_mobilenet.

System information

• OS: Windows 10 x64
• ONNX Runtime installed with pip using: "pip3 install onnxruntime-gpu"
• ONNX Runtime version: onnxruntime-gpu==1.5.2
• Python version: 3.7.9
• Using latest version of pycharm
• CUDA 10.2 and cuDNN 8.0.3
• GPU: GeForce RTX 3070 8GB
• CPU/Ram/hard drive: AMD Ryzen 7 3700x / 64GB / ~3000mb/sec io capable SSD

Is this normal? Should I try other versions of CUDA or cuDNN?

[korhun]

sess = rt.InferenceSession(onnx_fn) --> this waits for 50 - 200 seconds at the first call.
sess.run(output_names, input_item) --> this waits for 50 - 200 seconds at the first call.

With the same model, the amount of time varies even for the sequential tries. Sometimes I have to wait for 5 minutes and that makes the debuging process very not enjoyable. Any help is appreciated.

[korhun]

If:

• first --> pip3 install onnxruntime
• then --> pip3 install onnxruntime-gpu

yolov5x model took only 2 seconds to load. I was always uninstalling onnxruntime before installing onnxruntime-gpu.

If this was specified in the documents, I'm sorry guys :) If not, I have no idea if this is an important issue to be resolved or not. You decide, and fell free to close this issue.

Keep up the good work..

[korhun]

My previous comment is wrong. Uninstalling-installing has nothing to do with this issue. I have tried with another computer which has RXT 3080 GPU and Intel(R) Core(TM) i7-10700F CPU @ 2.90GHz, same issue occurred. I have no idea what made the other computer to load models in 0-1 seconds.

[korhun]

I'm not sure this issue is related to onnxruntime or not. These are the steps that resolved this issue in my tries on two computers:

1. install Cuda 10.2 and use latest cudnn for 10.2
2. pip install onnxruntime==1.5.2 --upgrade
3. pip install onnxruntime-gpu==1.5.2 --upgrade
4. install Cuda 11.0 and use latest cudnn for 11.0 --> resolved this issue for Geforce RTX 3070
5. install Cuda 11.1 and use latest cudnn for 11.1
6. install Cuda 11.2 --> issue seems to be resolved for Geforce RTX 3080

[apache2046]

I encounter the same problem

[pranavsharma]

@korhun is the issue resolved for you now (based on your earlier comment)? We'll be upgrading to cuda11 in the next release (coming soon).

[korhun]

@pranavsharma yes the issue had been resolved in 2 different computers after trying the steps I've mentioned on my previous comment.

Thanks for your interest. I'm looking forward for your next release; keep up the good work.

[SystemErrorWang]

@pranavsharma yes the issue had been resolved in 2 different computers after trying the steps I've mentioned on my previous comment.

Thanks for your interest. I'm looking forward for your next release; keep up the good work.

Thank you for providing the solution, I would like to know how should I solve this if my computer has no Nvidia GPU (on MacBook Pro)?
I tried to run pip install onnxruntime==1.5.2 --upgrade, but didn't help

[korhun]

@pranavsharma yes the issue had been resolved in 2 different computers after trying the steps I've mentioned on my previous comment.
Thanks for your interest. I'm looking forward for your next release; keep up the good work.

Thank you for providing the solution, I would like to know how should I solve this if my computer has no Nvidia GPU (on MacBook Pro)? I tried to run pip install onnxruntime==1.5.2 --upgrade, but didn't help

@SystemErrorWang there was no issue when working on CPU in any of my tries. I recommend, you uninstall onnxruntime-gpu and install the latest version of onnxruntime.

[SystemErrorWang]

@pranavsharma yes the issue had been resolved in 2 different computers after trying the steps I've mentioned on my previous comment.
Thanks for your interest. I'm looking forward for your next release; keep up the good work.

Thank you for providing the solution, I would like to know how should I solve this if my computer has no Nvidia GPU (on MacBook Pro)? I tried to run pip install onnxruntime==1.5.2 --upgrade, but didn't help

@SystemErrorWang there was no issue when working on CPU in any of my tries. I recommend, you uninstall onnxruntime-gpu and install the latest version of onnxruntime.

Thank You korhun, I finally found the problem, I used onnx simplifier and caused this error, the raw onnx works fine