Creating TRT Cache much slower on Linux than on Windows

[BengtGustafsson]

Describe the issue

When we create .engine files on Windows we see a run time of 15-30 s and it is not very dependent on the input sizes we use.

On Linux the same networks on the same hardware can take up to 10 minutes to optimize. The hardware includes a T1000 GPU and a recent Intel CPU.

Any ideas on this? Could it be that we run Linux via Docker?

To reproduce

Set up to create .engine files on Windows and Linux. check the time it takes.

Urgency

No response

Platform

Linux

OS Version

Ubuntu 20.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.20

ONNX Runtime API

C++

Architecture

X64

Execution Provider

TensorRT

Execution Provider Library Version

CUDA 11.6, TrT 10.4.0.26

[jywu-msft]

@yf711 can you confirm if this behavior repros for you?

[yf711]

Is your Linux env a dockerenv under your windows host env? Or it’s under another Linux host env

Similar thing happened to me when I ran Linux using Docker within Windows system and path to model/engine were projected from windows file system.