Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] The pointPillars model got wrong output when I use TensorRT acceleration #1520

Closed
3 tasks done
ykqyzzs opened this issue Dec 12, 2022 · 13 comments
Closed
3 tasks done

[Bug] The pointPillars model got wrong output when I use TensorRT acceleration #1520

ykqyzzs opened this issue Dec 12, 2022 · 13 comments
Assignees
@tpoisonooo
Labels
awaiting response Stale

Comments

@ykqyzzs
Copy link

ykqyzzs commented Dec 12, 2022

Checklist

  • I have searched related issues but cannot get the expected help.
  • 2. I have read the FAQ documentation but cannot get the expected help.
  • 3. The bug has not been fixed in the latest version.

Describe the bug

Hi,
I used mmdeploy/tools/test.py to test the converted pointpillars onnx model, it successfully finished but the AP result is abnormal:

----------- AP11 Results ------------

Pedestrian [email protected], 0.50, 0.50:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Pedestrian [email protected], 0.25, 0.25:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Cyclist [email protected], 0.50, 0.50:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Cyclist [email protected], 0.25, 0.25:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Car [email protected], 0.70, 0.70:
bbox AP11:0.0000, 9.0909, 9.0909
bev AP11:0.0000, 9.0909, 9.0909
3d AP11:0.0000, 9.0909, 9.0909
aos AP11:0.00, 9.09, 9.09
Car [email protected], 0.50, 0.50:
bbox AP11:0.0000, 9.0909, 9.0909
bev AP11:0.0000, 9.0909, 9.0909
3d AP11:0.0000, 9.0909, 9.0909
aos AP11:0.00, 9.09, 9.09

Overall AP11@easy, moderate, hard:
bbox AP11:0.0000, 3.0303, 3.0303
bev AP11:0.0000, 3.0303, 3.0303
3d AP11:0.0000, 3.0303, 3.0303
aos AP11:0.00, 3.03, 3.03

----------- AP40 Results ------------

Pedestrian [email protected], 0.50, 0.50:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Pedestrian [email protected], 0.25, 0.25:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Cyclist [email protected], 0.50, 0.50:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Cyclist [email protected], 0.25, 0.25:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Car [email protected], 0.70, 0.70:
bbox AP40:0.0000, 2.5000, 2.5000
bev AP40:0.0000, 2.5000, 2.5000
3d AP40:0.0000, 2.5000, 2.5000
aos AP40:0.00, 2.50, 2.50
Car [email protected], 0.50, 0.50:
bbox AP40:0.0000, 2.5000, 2.5000
bev AP40:0.0000, 2.5000, 2.5000
3d AP40:0.0000, 2.5000, 2.5000
aos AP40:0.00, 2.50, 2.50

Overall AP40@easy, moderate, hard:
bbox AP40:0.0000, 0.8333, 0.8333
bev AP40:0.0000, 0.8333, 0.8333
3d AP40:0.0000, 0.8333, 0.8333
aos AP40:0.00, 0.83, 0.83

Then I checked the data_loader, the inputs seems correct，and I have checked the onnx model, it seems ok and visualization by netron is almost same as the pointpillars onnx file from this link(which I found from this issue，NVIDIA/TensorRT#2338):
https://drive.google.com/file/d/1FuZJWLIsJyUsUk_lM1euXzyPgagu-tXj/view?usp=sharing
I have tried the both onnx file, converted into .engine file and test, but got the same results.

So I print the outputs then found that, outputs = task_processor.single_gpu_test(model, data_loader, args.show, args.show_dir) returned the empty outputs such as:

{'boxes_3d': LiDARInstance3DBoxes(
tensor([], size=(0, 7))), 'scores_3d': tensor([]), 'labels_3d': tensor([], dtype=torch.int64)}, {'boxes_3d': LiDARInstance3DBoxes(
tensor([], size=(0, 7))), 'scores_3d': tensor([]), 'labels_3d': tensor([], dtype=torch.int64)}, {'boxes_3d': LiDARInstance3DBoxes(
tensor([], size=(0, 7))), 'scores_3d': tensor([]), 'labels_3d': tensor([], dtype=torch.int64)}, {'boxes_3d': LiDARInstance3DBoxes(
tensor([], size=(0, 7))), 'scores_3d': tensor([]), 'labels_3d': tensor([], dtype=torch.int64)}, {'boxes_3d': LiDARInstance3DBoxes(
tensor([], size=(0, 7))), 'scores_3d': tensor([]), 'labels_3d': tensor([], dtype=torch.int64)}, {'boxes_3d': LiDARInstance3DBoxes(
tensor([], size=(0, 7))), 'scores_3d': tensor([]), 'labels_3d': tensor([], dtype=torch.int64)}, ... ...

So, I would like to ask, what may be the cause of the the wrong test results? and how to solve it?
Thanks very much!

Reproduction

I use this command to convert model:
python mmdeploy/tools/deploy.py mmdeploy/configs/mmdet3d/voxel-detection/voxel-detection_tensorrt_dynamic-kitti-32x4.py mmdetection3d/configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py checkpoints/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class_20220301_150306-37dc2420.pth mmdetection3d/demo/data/kitti/kitti_000008.bin --work-dir work-dir2 --device cuda:0 --show

I use this command to test the converted model:
python ../mmdeploy/tools/test.py ../mmdeploy/configs/mmdet3d/voxel-detection/voxel-detection_tensorrt_dynamic-kitti-32x4.py ./configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py --model ../mmdeploy/work-dir2/end2end.engine --metrics bbox --device cuda:0

Environment

2022-12-12 18:08:33,002 - mmdeploy - INFO - 

2022-12-12 18:08:33,002 - mmdeploy - INFO - **********Environmental information**********
2022-12-12 18:08:33,416 - mmdeploy - INFO - sys.platform: linux
2022-12-12 18:08:33,416 - mmdeploy - INFO - Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0]
2022-12-12 18:08:33,416 - mmdeploy - INFO - CUDA available: True
2022-12-12 18:08:33,416 - mmdeploy - INFO - GPU 0,1,2,3,4,5,6,7,8,9: NVIDIA GeForce RTX 3090
2022-12-12 18:08:33,416 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda
2022-12-12 18:08:33,416 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.3, V11.3.109
2022-12-12 18:08:33,416 - mmdeploy - INFO - GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
2022-12-12 18:08:33,416 - mmdeploy - INFO - PyTorch: 1.11.0
2022-12-12 18:08:33,416 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.2
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.11.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

2022-12-12 18:08:33,416 - mmdeploy - INFO - TorchVision: 0.12.0
2022-12-12 18:08:33,416 - mmdeploy - INFO - OpenCV: 4.5.5
2022-12-12 18:08:33,416 - mmdeploy - INFO - MMCV: 1.5.2
2022-12-12 18:08:33,416 - mmdeploy - INFO - MMCV Compiler: GCC 7.5
2022-12-12 18:08:33,416 - mmdeploy - INFO - MMCV CUDA Compiler: 11.3
2022-12-12 18:08:33,416 - mmdeploy - INFO - MMDeploy: 0.10.0+99040d5
2022-12-12 18:08:33,416 - mmdeploy - INFO - 

2022-12-12 18:08:33,417 - mmdeploy - INFO - **********Backend information**********
2022-12-12 18:08:34,080 - mmdeploy - INFO - onnxruntime: None	ops_is_avaliable : False
2022-12-12 18:08:34,108 - mmdeploy - INFO - tensorrt: 8.5.1.7	ops_is_avaliable : True
2022-12-12 18:08:34,124 - mmdeploy - INFO - ncnn: None	ops_is_avaliable : False
2022-12-12 18:08:34,125 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-12-12 18:08:34,126 - mmdeploy - INFO - openvino_is_avaliable: False
2022-12-12 18:08:34,142 - mmdeploy - INFO - snpe_is_available: False
2022-12-12 18:08:34,143 - mmdeploy - INFO - ascend_is_available: False
2022-12-12 18:08:34,144 - mmdeploy - INFO - coreml_is_available: False
2022-12-12 18:08:34,144 - mmdeploy - INFO - 

2022-12-12 18:08:34,144 - mmdeploy - INFO - **********Codebase information**********
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmdet:	2.24.1
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmseg:	0.24.1
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmcls:	0.23.0
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmocr:	None
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmedit:	None
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmdet3d:	1.0.0rc4
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmpose:	None
2022-12-12 18:08:34,146 - mmdeploy - INFO - mmrotate:	None

Error traceback

Nothing
@ykqyzzs
Copy link
Author

ykqyzzs commented Dec 12, 2022

Add a little infomation, there are some worning when I run tools/test.py:

2022-12-12 16:14:11,925 - mmdeploy - INFO - Successfully loaded tensorrt plugins from /home/yangkang/anaconda3/envs/ri_fusion/lib/python3.7/site-packages/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
2022-12-12 16:14:11,925 - mmdeploy - INFO - Successfully loaded tensorrt plugins from /home/yangkang/anaconda3/envs/ri_fusion/lib/python3.7/site-packages/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
[12/12/2022-16:14:13] [TRT] [W] TensorRT was linked against cuDNN 8.6.0 but loaded cuDNN 8.2.1
[12/12/2022-16:14:13] [TRT] [W] TensorRT was linked against cuDNN 8.6.0 but loaded cuDNN 8.2.1
[12/12/2022-16:14:14] [TRT] [W] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See CUDA_MODULE_LOADING in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
[ ] 0/3769, elapsed: 0s, ETA:/home/yangkang/ri_fusion/rifusion_test/mmdet3d/models/dense_heads/anchor3d_head.py:85: UserWarning: dir_offset and dir_limit_offset will be depressed and be incorporated into box coder in the future
'dir_offset and dir_limit_offset will be depressed and be '
2022-12-12 16:14:19,713 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied
/home/yangkang/anaconda3/envs/ri_fusion/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1646755953518/work/aten/src/ATen/native/TensorShape.cpp:2228.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

[ ] 1/3769, 0.5 task/s, elapsed: 2s, ETA: 7918s2022-12-12 16:14:20,237 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied

[ ] 2/3769, 0.8 task/s, elapsed: 3s, ETA: 4826s2022-12-12 16:14:20,700 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied
... ...
... ...
... ...
... ...
[>>>>>>>>>>>>>>>>>>>>>>>>> ] 3767/3769, 6276.0 task/s, elapsed: 1s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>> ] 3768/3769, 6276.4 task/s, elapsed: 1s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>>>] 3769/3769, 6276.8 task/s, elapsed: 1s, ETA: 0s
Result is saved to /tmp/tmpaslh9sjb/results.pkl.
OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.

----------- AP11 Results ------------

Pedestrian [email protected], 0.50, 0.50:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Pedestrian [email protected], 0.25, 0.25:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Cyclist [email protected], 0.50, 0.50:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Cyclist [email protected], 0.25, 0.25:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
Car [email protected], 0.70, 0.70:
bbox AP11:0.0000, 9.0909, 9.0909
bev AP11:0.0000, 9.0909, 9.0909
3d AP11:0.0000, 9.0909, 9.0909
aos AP11:0.00, 9.09, 9.09
Car [email protected], 0.50, 0.50:
bbox AP11:0.0000, 9.0909, 9.0909
bev AP11:0.0000, 9.0909, 9.0909
3d AP11:0.0000, 9.0909, 9.0909
aos AP11:0.00, 9.09, 9.09

Overall AP11@easy, moderate, hard:
bbox AP11:0.0000, 3.0303, 3.0303
bev AP11:0.0000, 3.0303, 3.0303
3d AP11:0.0000, 3.0303, 3.0303
aos AP11:0.00, 3.03, 3.03

----------- AP40 Results ------------

Pedestrian [email protected], 0.50, 0.50:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Pedestrian [email protected], 0.25, 0.25:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Cyclist [email protected], 0.50, 0.50:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Cyclist [email protected], 0.25, 0.25:
bbox AP40:0.0000, 0.0000, 0.0000
bev AP40:0.0000, 0.0000, 0.0000
3d AP40:0.0000, 0.0000, 0.0000
aos AP40:0.00, 0.00, 0.00
Car [email protected], 0.70, 0.70:
bbox AP40:0.0000, 2.5000, 2.5000
bev AP40:0.0000, 2.5000, 2.5000
3d AP40:0.0000, 2.5000, 2.5000
aos AP40:0.00, 2.50, 2.50
Car [email protected], 0.50, 0.50:
bbox AP40:0.0000, 2.5000, 2.5000
bev AP40:0.0000, 2.5000, 2.5000
3d AP40:0.0000, 2.5000, 2.5000
aos AP40:0.00, 2.50, 2.50

Overall AP40@easy, moderate, hard:
bbox AP40:0.0000, 0.8333, 0.8333
bev AP40:0.0000, 0.8333, 0.8333
3d AP40:0.0000, 0.8333, 0.8333
aos AP40:0.00, 0.83, 0.83

@tpoisonooo
Copy link
Collaborator

Let me try to reproduce it with TRT8.5 later.

@tpoisonooo
Copy link
Collaborator

tpoisonooo commented Dec 14, 2022 •
edited
Loading

And I have tested pointpillars+centerpoint on cu102+TRT8.4 and passed.

@ykqyzzs
Copy link
Author

ykqyzzs commented Dec 17, 2022

Thanks for your reply,
I have tried cuda11.3+TRT8.4.3.1 @yestoday, got the same problem T_T. I used the kitti dataset to test, and then I looked at the outputs carefully, found not all the outputs were null, Only one frame of outputs got reasonable results:
image
I guess it's my envs or commands have something wrong... Maybe some important warning caused my situation? Would you please help me to check my command or running logs to solve the problem? @tpoisonooo
Thank you very much!

@ykqyzzs
Copy link
Author

ykqyzzs commented Dec 17, 2022

TRT8.4.3.1+CUDA11.3, running log is:

2022-12-17 10:19:01,704 - mmdeploy - INFO - Successfully loaded tensorrt plugins from /home/yangkang/anaconda3/envs/ri_fusion/lib/python3.7/site-packages/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
2022-12-17 10:19:01,704 - mmdeploy - INFO - Successfully loaded tensorrt plugins from /home/yangkang/anaconda3/envs/ri_fusion/lib/python3.7/site-packages/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
[ ] 0/3769, elapsed: 0s, ETA:/home/yangkang/ri_fusion/rifusion_test/mmdet3d/models/dense_heads/anchor3d_head.py:85: UserWarning: dir_offset and dir_limit_offset will be depressed and be incorporated into box coder in the future
'dir_offset and dir_limit_offset will be depressed and be '
2022-12-17 10:19:09,987 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied
/home/yangkang/anaconda3/envs/ri_fusion/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1646755953518/work/aten/src/ATen/native/TensorShape.cpp:2228.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

[ ] 1/3769, 0.3 task/s, elapsed: 4s, ETA: 14365s2022-12-17 10:19:10,382 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied

[ ] 2/3769, 0.5 task/s, elapsed: 4s, ETA: 7894s2022-12-17 10:19:10,766 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied

[ ] 3/3769, 0.7 task/s, elapsed: 5s, ETA: 5740s2022-12-17 10:19:11,143 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied

[ ] 4/3769, 0.8 task/s, elapsed: 5s, ETA: 4658s2022-12-17 10:19:11,521 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied
... ...
... ...
... ...
Converting prediction to KITTI format
[ ] 0/3769, elapsed: 0s, ETA:
[ ] 1/3769, 3231.4 task/s, elapsed: 0s, ETA: 1s
[ ] 2/3769, 4082.0 task/s, elapsed: 0s, ETA: 1s
[ ] 3/3769, 4782.6 task/s, elapsed: 0s, ETA: 1s
[ ] 4/3769, 5304.2 task/s, elapsed: 0s, ETA: 1s
[ ] 5/3769, 5714.3 task/s, elapsed: 0s, ETA: 1s
... ...
... ...
... ...
[>>>>>>>>>>>>>>>>>>>>>>>>> ] 3767/3769, 7688.5 task/s, elapsed: 0s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>> ] 3768/3769, 7688.9 task/s, elapsed: 0s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>>>] 3769/3769, 7689.2 task/s, elapsed: 0s, ETA: 0s
Result is saved to /tmp/tmp_j1lcqev/results.pkl.
OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.

----------- AP11 Results ------------

Pedestrian [email protected], 0.50, 0.50:
bbox AP11:0.0000, 0.0000, 0.0000
bev AP11:0.0000, 0.0000, 0.0000
3d AP11:0.0000, 0.0000, 0.0000
aos AP11:0.00, 0.00, 0.00
... ...
... ...
... ...

@ykqyzzs
Copy link
Author

ykqyzzs commented Dec 19, 2022

I tested again on another server with CUDA10.2 + TensorRT8.4.3.1. The problem was gone, I got the correct results. thanks alot.
but I still want to know, what's the wrong reason on my envs in this issue...

@tpoisonooo
Copy link
Collaborator

I have no TRT source code, just some assumptions.

  1. Open your .onnx file, you can get a Scatter or ScatterND operator
  2. TRT has some bug when producing this operator with dynamic input shape

@tpoisonooo tpoisonooo pinned this issue Dec 20, 2022
@kunnalparihar
Copy link

kunnalparihar commented Feb 19, 2023 •
edited
Loading

I tested again on another server with CUDA10.2 + TensorRT8.4.3.1. The problem was gone, I got the correct results. thanks alot. but I still want to know, what's the wrong reason on my envs in this issue...

Hi @ykqyzzs , I need your help on this.
What was Torch version you used?
Have you used docker ?

@ykqyzzs
Copy link
Author

ykqyzzs commented Feb 19, 2023

I tested again on another server with CUDA10.2 + TensorRT8.4.3.1. The problem was gone, I got the correct results. thanks alot. but I still want to know, what's the wrong reason on my envs in this issue...

Hi @ykqyzzs , I need your help on this. What was Torch version you used? Have you used docker ?

the pytorch version I used is 1.10.1;
without using docker, I deployed it directly in a new virtual environment, following the steps of the official document.
wish this information above can help you.

@WYYAHYT
Copy link

WYYAHYT commented Feb 20, 2023

I tested again on another server with CUDA10.2 + TensorRT8.4.3.1. The problem was gone, I got the correct results. thanks alot. but I still want to know, what's the wrong reason on my envs in this issue...

I met the same error with CUDA11.4+TensorRT8.4.0.11 , and have you found the reason?

@ykqyzzs
Copy link
Author

ykqyzzs commented Feb 20, 2023

I tested again on another server with CUDA10.2 + TensorRT8.4.3.1. The problem was gone, I got the correct results. thanks alot. but I still want to know, what's the wrong reason on my envs in this issue...

I met the same error with CUDA11.4+TensorRT8.4.0.11 , and have you found the reason?

sorry I have no idea. I have cross-verified some cuda and trt versions, cuda11.3+trt8.4 also failed, cuda102+trt 8.4.3.1 passed. maybe you can have a try.

@tpoisonooo tpoisonooo mentioned this issue Mar 6, 2023
Closed
3 tasks
@github-actions
Copy link

This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.

@github-actions github-actions bot added the Stale label Mar 24, 2023
@github-actions
Copy link

This issue is closed because it has been stale for 5 days. Please open a new issue if you have similar issues or you have any new updates now.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Mar 30, 2023
@RunningLeon RunningLeon unpinned this issue Apr 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tpoisonooo tpoisonooo

Labels
awaiting response Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tpoisonooo @kunnalparihar @WYYAHYT @ykqyzzs