Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TypeError: can't convert cuda:0 device type tensor to numpy. #506

Closed
421psh opened this issue May 20, 2022 · 1 comment · Fixed by #511
Closed

TypeError: can't convert cuda:0 device type tensor to numpy. #506

421psh opened this issue May 20, 2022 · 1 comment · Fixed by #511
Assignees
@RunningLeon

Comments

@421psh
Copy link

421psh commented May 20, 2022

Describe the bug

This bug was already mentioned here #339, but it is still occurring.

Reproduction

  1. What command or script did you run?
python mmdeploy/tools/test.py ./mmdeploy/configs/mmdet/instance-seg/instance-seg_tensorrt-fp16_dynamic-320x320-1344x1344.py ./experiments/experiment_13/cascade_mask_rcnn_r50_fpn_20e_coco.py --model ./experiments/experiment_13/tensorrt/end2end.engine --out ./experiments/experiment_13/tensorrt/evaluation_log.pkl --metrics bbox segm --speed-test  --warmup 10 --log2file ./experiments/experiment_13/tensorrt/log.pkl --device cuda:0
  1. Did you make any modifications on the code or config? Did you understand what you have modified? - No modifications from my side.

Environment

2022-05-20 19:09:04,047 - mmdeploy - INFO - **********Environmental information**********
2022-05-20 19:09:04,209 - mmdeploy - INFO - sys.platform: linux
2022-05-20 19:09:04,209 - mmdeploy - INFO - Python: 3.8.5 (default, May 27 2021, 13:30:53) [GCC 9.3.0]
2022-05-20 19:09:04,209 - mmdeploy - INFO - CUDA available: True
2022-05-20 19:09:04,209 - mmdeploy - INFO - GPU 0: NVIDIA GeForce RTX 3070 Ti
2022-05-20 19:09:04,209 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda
2022-05-20 19:09:04,209 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.3, V11.3.109
2022-05-20 19:09:04,209 - mmdeploy - INFO - GCC: x86_64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
2022-05-20 19:09:04,209 - mmdeploy - INFO - PyTorch: 1.11.0+cu113
2022-05-20 19:09:04,209 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.2
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.11.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

2022-05-20 19:09:04,210 - mmdeploy - INFO - TorchVision: 0.12.0+cu113
2022-05-20 19:09:04,210 - mmdeploy - INFO - OpenCV: 4.5.5
2022-05-20 19:09:04,210 - mmdeploy - INFO - MMCV: 1.5.1
2022-05-20 19:09:04,210 - mmdeploy - INFO - MMCV Compiler: GCC 7.3
2022-05-20 19:09:04,210 - mmdeploy - INFO - MMCV CUDA Compiler: 11.3
2022-05-20 19:09:04,210 - mmdeploy - INFO - MMDeploy: 0.4.0+8cd17f4
2022-05-20 19:09:04,210 - mmdeploy - INFO - 

2022-05-20 19:09:04,210 - mmdeploy - INFO - **********Backend information**********
[2022-05-20 19:09:04.262] [mmdeploy] [info] [model.cpp:95] Register 'DirectoryModel'
2022-05-20 19:09:04,267 - mmdeploy - INFO - onnxruntime: 1.8.1	ops_is_avaliable : True
2022-05-20 19:09:04,268 - mmdeploy - INFO - tensorrt: 7.2.3.4	ops_is_avaliable : True
2022-05-20 19:09:04,269 - mmdeploy - INFO - ncnn: None	ops_is_avaliable : False
2022-05-20 19:09:04,270 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-05-20 19:09:04,271 - mmdeploy - INFO - openvino_is_avaliable: False
2022-05-20 19:09:04,271 - mmdeploy - INFO - 

2022-05-20 19:09:04,271 - mmdeploy - INFO - **********Codebase information**********
2022-05-20 19:09:04,273 - mmdeploy - INFO - mmdet:	2.24.1
2022-05-20 19:09:04,273 - mmdeploy - INFO - mmseg:	None
2022-05-20 19:09:04,273 - mmdeploy - INFO - mmcls:	None
2022-05-20 19:09:04,273 - mmdeploy - INFO - mmocr:	None
2022-05-20 19:09:04,273 - mmdeploy - INFO - mmedit:	None
2022-05-20 19:09:04,273 - mmdeploy - INFO - mmdet3d:	None
2022-05-20 19:09:04,273 - mmdeploy - INFO - mmpose:	None

Docker container

# NVIDIA DRIVER = 465.19.01, CUDA = 1.3.1.005, TensorRT = 7.2.3.4
FROM nvcr.io/nvidia/tensorrt:21.06-py3

# Install dependencies
ENV TZ=Europe/Moscow
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get update && apt-get install -y build-essential
RUN apt-get update --fix-missing
RUN apt-get install -y libgtk-3-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev \
    libxvidcore-dev libx264-dev libjpeg-dev libpng-dev libtiff-dev \
    gfortran openexr libatlas-base-dev python3-dev python3-numpy \
    libtbb2 libtbb-dev libdc1394-22-dev libopenexr-dev \
    libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev  \
    libsm6 libxext6 libxrender-dev libgl1-mesa-glx git wget libssl-dev libopencv-dev libspdlog-dev python3-tk

# Workspace directory
WORKDIR /workspace

# Enviroment variables
ENV TENSORRT_DIR=/workspace/tensorrt
ENV LD_LIBRARY_PATH=$TENSORRT_DIR/lib:$LD_LIBRARY_PATH
ENV CUDNN_DIR=/usr/local/cuda
ENV CUDA_HOME=/usr/local/cuda
ENV LD_LIBRARY_PATH=$CUDNN_DIR/lib64:$LD_LIBRARY_PATH
ENV BACKUP_LD_LIBRARY_PATH=$LD_LIBRARY_PATH

# Upgrade pip and install cmake
RUN pip install pip -U
RUN pip install cmake

# OpenCV
RUN mkdir opencv_build
WORKDIR /workspace/opencv_build
RUN git clone https://github.com/opencv/opencv.git --branch 4.5.5
RUN git clone https://github.com/opencv/opencv_contrib.git --branch 4.5.5
WORKDIR /workspace/opencv_build/opencv
RUN mkdir build
WORKDIR /workspace/opencv_build/opencv/build
RUN cmake -D CMAKE_BUILD_TYPE=RELEASE \
    -D CMAKE_INSTALL_PREFIX=/usr/local \
    -D INSTALL_C_EXAMPLES=ON \
    -D INSTALL_PYTHON_EXAMPLES=ON \
    -D OPENCV_GENERATE_PKGCONFIG=ON \
    -D OPENCV_EXTRA_MODULES_PATH=/workspace/opencv_build/opencv_contrib/modules \
    -D BUILD_EXAMPLES=ON ..
RUN make -j$(nproc) && make install
WORKDIR /workspace

# PPL CV
RUN git clone https://github.com/openppl-public/ppl.cv.git --branch v0.6.2
WORKDIR /workspace/ppl.cv
RUN ./build.sh cuda
ENV PPLCV_DIR=/workspace/ppl.cv
WORKDIR /workspace

# ONNXRUNTIME
ARG ONNXRUNTIME_VERSION=1.8.1
RUN wget https://github.com/microsoft/onnxruntime/releases/download/v${ONNXRUNTIME_VERSION}/onnxruntime-linux-x64-${ONNXRUNTIME_VERSION}.tgz \
    && tar -zxvf onnxruntime-linux-x64-${ONNXRUNTIME_VERSION}.tgz &&\
    pip install onnxruntime-gpu==${ONNXRUNTIME_VERSION}
ENV ONNXRUNTIME_DIR=/workspace/onnxruntime-linux-x64-${ONNXRUNTIME_VERSION}
WORKDIR /workspace

# Install MMDeploy
RUN git clone https://github.com/open-mmlab/mmdeploy.git
WORKDIR /workspace/mmdeploy
RUN git checkout a4de9f3

## Add MMDeploy third party submodules
WORKDIR /workspace/mmdeploy/third_party
RUN git clone https://github.com/NVIDIA/cub.git
WORKDIR /workspace/mmdeploy/third_party/cub
RUN git checkout c3cceac115
WORKDIR /workspace/mmdeploy/third_party
RUN git clone https://github.com/pybind/pybind11.git
WORKDIR /workspace/mmdeploy/third_party/pybind11
RUN git checkout 70a58c5

WORKDIR /workspace/mmdeploy
RUN mkdir build
WORKDIR /workspace/mmdeploy/build
RUN cmake -DMMDEPLOY_TARGET_BACKENDS="ort;trt" ..
RUN make -j$(nproc)
WORKDIR /workspace/mmdeploy
RUN pip install -e .

RUN rm -rf build/CM* build/cmake-install.cmake build/Makefile build/csrc
WORKDIR /workspace/mmdeploy/build
RUN cmake ..  \
    -DMMDEPLOY_BUILD_SDK=ON \
    -DCMAKE_CXX_COMPILER=g++ \
    -Dpplcv_DIR=${PPLCV_DIR}/cuda-build/install/lib/cmake/ppl \
    -DTENSORRT_DIR=${TENSORRT_DIR} \
    -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} \
    -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON \
    -DMMDEPLOY_TARGET_DEVICES="cuda;cpu" \
    -DMMDEPLOY_TARGET_BACKENDS="ort;trt" \
    -DMMDEPLOY_CODEBASES=all
RUN cmake --build . -- -j$(nproc)
RUN cmake --install .
ENV MMDEPLOY_DIR=/workspace/mmdeploy

# Pytorch, MMCV, MMDetection
RUN pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
RUN pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11.0/index.html
RUN pip install mmdet==2.24.1
RUN pip install instaboostfast
RUN pip install git+https://github.com/cocodataset/panopticapi.git
RUN pip install git+https://github.com/lvis-dataset/lvis-api.git
RUN pip install albumentations>=0.3.2 --no-binary imgaug,albumentations
RUN pip install future==0.18.2
RUN pip install tensorboard==2.9.0

ADD https://download.pytorch.org/models/resnet50-0676ba61.pth /root/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth

# App directory
WORKDIR /workspace/app
COPY . /workspace/app

Error traceback

Traceback (most recent call last):
  File "mmdeploy/tools/test.py", line 137, in <module>
    main()
  File "mmdeploy/tools/test.py", line 126, in main
    outputs = task_processor.single_gpu_test(model, data_loader,
  File "/workspace/mmdeploy/mmdeploy/codebase/base/task.py", line 137, in single_gpu_test
    return self.codebase_class.single_gpu_test(model, data_loader, show,
  File "/workspace/mmdeploy/mmdeploy/codebase/mmdet/deploy/mmdetection.py", line 142, in single_gpu_test
    outputs = single_gpu_test(model, data_loader, show, out_dir, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/mmdet/apis/test.py", line 65, in single_gpu_test
    result = [(bbox_results, encode_mask_results(mask_results))
  File "/usr/local/lib/python3.8/dist-packages/mmdet/apis/test.py", line 65, in <listcomp>
    result = [(bbox_results, encode_mask_results(mask_results))
  File "/usr/local/lib/python3.8/dist-packages/mmdet/core/mask/utils.py", line 59, in encode_mask_results
    np.array(
  File "/usr/local/lib/python3.8/dist-packages/torch/_tensor.py", line 734, in __array__
    return self.numpy().astype(dtype, copy=False)
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
@RunningLeon
Copy link
Collaborator

@421psh Hi thanks for your feedback. We'll fix it asap.

@RunningLeon RunningLeon mentioned this issue May 23, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RunningLeon RunningLeon

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix mask test with mismatched device RunningLeon/mmdeploy
2 participants
@RunningLeon @421psh