Failed to import MXNet built with TensorRT

[Faldict]

I pulled the latest source code from the master branch and built MXNet successfully with USE_TENSORRT = 1. But I failed to import mxnet:

python3 -E -v -c "import mxnet as mx"

Here is the error log:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/faldict/incubator-mxnet/python/mxnet/__init__.py", line 24, in <module>
    from .context import Context, current_context, cpu, gpu, cpu_pinned
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/faldict/incubator-mxnet/python/mxnet/context.py", line 24, in <module>
    from .base import classproperty, with_metaclass, _MXClassPropertyMetaClass
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/faldict/incubator-mxnet/python/mxnet/base.py", line 217, in <module>
    _LIB = _load_lib()
  File "/home/faldict/incubator-mxnet/python/mxnet/base.py", line 208, in _load_lib
    lib = ctypes.CDLL(lib_path[0], ctypes.RTLD_LOCAL)
  File "/usr/lib/python3.5/ctypes/__init__.py", line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/faldict/incubator-mxnet/python/mxnet/../../lib/libmxnet.so: undefined symbol: _ZNK6google8protobuf7Message9SpaceUsedEv

I use protobuf 3.5.1.
@mkolod Could you please take a look at this?

[lanking520]

Hi @Faldict thanks for your reported issue.

@haojin2 could you please take a look at here, I remember somebody is already working on this, can you point this issue to that PR?

@mxnet-label-bot could you please label this as [build, backend]?

[marcoabreu]

@KellenSunderland

[KellenSunderland]

Hey @Faldict. The problem is that you don't have protobuf on your LD_LIBRARY_PATH. I'd recommend setting your path like the following:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MXNET_PATH/incubator-mxnet/3rdparty/onnx-tensorrt/build:$MXNET_PATH/3rdparty/onnx-tensorrt/third_party/onnx/build:$PROTOBUF_PATH/src/.libs

Where MXNET_PATH is the root director of you MXNet folder, and PROTOBUF_PATH is the root directory of your protobuf files.

In general when you run into these runtime issues one step that usually helps me is to run

ldd /home/faldict/incubator-mxnet/python/mxnet/../../lib/libmxnet.so

which should show all the libraries that have been referenced during compilation, but which are not currently on my library path. I then search my filesystem for those libs and append their folders into my LD_LIBRARY_PATH.

Note: Right now building from source for this feature is admittedly quite complicated. Thank you very much for being an early adopter. We're working together this week to provide some detailed information about how to install and run this feature. Those docs will hopefully make the process a bit easier.

[Faldict]

@KellenSunderland Thanks for your useful reply! I run ldd libmxnet.so and get the following information:

	linux-vdso.so.1 =>  (0x00007fffa4d98000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f31c96a5000)
	libcudart.so.9.0 => /usr/local/cuda-9.0/lib64/libcudart.so.9.0 (0x00007f31c9438000)
	libcublas.so.9.0 => /usr/local/cuda-9.0/lib64/libcublas.so.9.0 (0x00007f31c5b33000)
	libcurand.so.9.0 => /usr/local/cuda-9.0/lib64/libcurand.so.9.0 (0x00007f31c1bcf000)
	libcusolver.so.9.0 => /usr/local/cuda-9.0/lib64/libcusolver.so.9.0 (0x00007f31bcfd4000)
	libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007f31baf40000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f31bad38000)
	libprotobuf.so.15 => /usr/local/lib/libprotobuf.so.15 (0x00007f31ba8b0000)
	libnvonnxparser.so.0 => /usr/local/lib/libnvonnxparser.so.0 (0x00007f31ba591000)
	libnvonnxparser_runtime.so.0 => /usr/local/lib/libnvonnxparser_runtime.so.0 (0x00007f31ba2dc000)
	libnvinfer.so.4 => /usr/lib/x86_64-linux-gnu/libnvinfer.so.4 (0x00007f31b45b4000)
	libnvinfer_plugin.so.4 => /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.4 (0x00007f31b40ed000)
	liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00007f31b390a000)
	libcudnn.so.7 => /usr/local/cuda-9.0/lib64/libcudnn.so.7 (0x00007f31a2403000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f31a2081000)
	libomp.so.5 => /usr/lib/x86_64-linux-gnu/libomp.so.5 (0x00007f31a1db5000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f31a1b9f000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f31a1982000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f31a15b8000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f31ccce9000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f31a13b4000)
	libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f31a1089000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f31a0e6f000)
	libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f31a0c30000)

In fact, I have built protobuf, onnx and onnx-tensorrt from source separately and add them to the LD_LIBRARY_PATH. Currently I doubt this problem is because of the compatibility of different protobuf versions. Run ldconfig -p | grep -i protobuf:

	libprotobuf.so.15 (libc6,x86-64) => /usr/local/lib/libprotobuf.so.15
	libprotobuf.so.9 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libprotobuf.so.9
	libprotobuf.so (libc6,x86-64) => /usr/local/lib/libprotobuf.so
	libprotobuf.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libprotobuf.so
	libprotobuf-lite.so.15 (libc6,x86-64) => /usr/local/lib/libprotobuf-lite.so.15
	libprotobuf-lite.so.9 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libprotobuf-lite.so.9
	libprotobuf-lite.so (libc6,x86-64) => /usr/local/lib/libprotobuf-lite.so
	libprotobuf-lite.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libprotobuf-lite.so
	libmirprotobuf.so.3 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libmirprotobuf.so.3

libprotobuf.so.9 is the default version of Ubuntu 16.04, while libprotobuf.so.15 is installed after building protobuf 3.5.1. Do you have any further ideas?

[KellenSunderland]

@Faldict It looks like everything is linked and resolved correctly to me. That is a little strange. I'd like to statically link the protobuf lib in the future which should solve this. The only advice I could give at this point would be to try to closely copy the installation process the CI is taking. I'll hope to have docker images and/or pip packages next week if you're ok with either of those solutions.

Edit: one thing you could try and do is ensure you only have a single version of protobuf on your machine (i.e. uninstall any that may have been included from package managers), then clean and rebuild.

[Faldict]

@KellenSunderland I uninstalled protobuf 3.5.1 and rebuild the whole toolchain. At present, MXNet could be imported successfully. It seems that you should constrain the protobuf version strictly.

Further more, I tried to run a tensorrt baseline. I used the test code incubator-mxnet/tests/python/tensorrt/test_tensorrt_lenet5.py but got an unexpected error:

“python3 test_tensorrt_lenet5.py” terminated by signal SIGSEGV (Address boundary error)

As I set some breakpoints, I found this error occurs when executing this line:

        executor = mx.contrib.tensorrt.tensorrt_bind(sym, ctx=mx.gpu(0), all_params=all_params,
                                                     data=data_size,
                                                     softmax_label=(batch_size,),
                                                     grad_req='null',
                                                     force_rebind=True)

where the symbol and parameters are trained by running python3 lenet5_train.py. So how to solve this problem?

EDIT: when I dig deeper, the error occurs during the execution of _LIB.MXExecutorSimpleBind().

[KellenSunderland]

@Faldict: I suspect it's still something to do with the build, but it could be some missing validation. Do the other tests run properly?

I'd like to do two things to help troubleshoot the problem. First use a pre-built package to rule out build issues. Second lets gather some more information by getting a full stack dump.

Would you be able to run the diagnose script so I can see what OS distro you're running? I'm working on an installer package for TRT at the moment. Since you're one of the earlier adopters maybe you can give it a shot and see if it fixes your issues?

What to do:

1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using python diagnose.py and paste its output here.

Could you also try to run the test using gdb? You would need to run something like:

gdb python3 incubator-mxnet/tests/python/tensorrt/test_tensorrt_lenet5.py

then from within gdb
c 
# to continue, it should then crash and allow you to enter this command:
thread apply all bt
# dumps the stack of all threads

If you could then paste the results here that would help me understand where the crash is coming from.

[Faldict]

@KellenSunderland I'm glad to do something that would benefit your work. Firstly, I ran the diagnosis and paste it below:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                12
On-line CPU(s) list:   0-11
Thread(s) per core:    2
Core(s) per socket:    6
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 158
Model name:            Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
Stepping:              10
CPU MHz:               1579.386
CPU max MHz:           4600.0000
CPU min MHz:           800.0000
BogoMIPS:              6384.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              12288K
NUMA node0 CPU(s):     0-11
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
----------Python Info----------
Version      : 3.5.2
Compiler     : GCC 5.4.0 20160609
Build        : ('default', 'Nov 23 2017 16:37:01')
Arch         : ('64bit', 'ELF')
------------Pip Info-----------
Version      : 18.0
Directory    : /usr/local/lib/python3.5/dist-packages/pip
----------MXNet Info-----------
Version      : 1.3.0
Directory    : /home/faldict/incubator-mxnet/python/mxnet
Hashtag not found. Not installed from pre-built package.
----------System Info----------
Platform     : Linux-4.15.0-30-generic-x86_64-with-Ubuntu-16.04-xenial
system       : Linux
node         : Nexus
release      : 4.15.0-30-generic
version      : #32~16.04.1-Ubuntu SMP Thu Jul 26 20:25:39 UTC 2018
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0054 sec, LOAD: 2.0901 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0030 sec, LOAD: 0.5963 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0046 sec, LOAD: 1.1451 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0033 sec, LOAD: 4.9819 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0083 sec, LOAD: 1.0066 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0044 sec, LOAD: 2.1012 sec.

What's more, as I mentioned here, my PC has a GTX 1060 GPU.

Then I used gdb to run the test, crashed with following message:

[New Thread 0x7fff1a8ba700 (LWP 20550)]
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid

Thread 1 "python3" received signal SIGABRT, Aborted.
0x00007ffff7825428 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/unix/sysv/linux/raise.c:54
54	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

Next, I entered the dump command and selected the most important segment, which is Thread 1 in this case, and paste here:

Thread 1 (Thread 0x7ffff7fce700 (LWP 20469)):
#0  0x00007ffff7825428 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007ffff782702a in __GI_abort () at abort.c:89
#2  0x00007fffeb49984d in __gnu_cxx::__verbose_terminate_handler() ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
---Type <return> to continue, or q <return> to quit---
#3  0x00007fffeb4976b6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007fffeb497701 in std::terminate() ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007fffeb497969 in __cxa_rethrow ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff2a3510f in std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, onnxTensorDescriptorV1 const*>, false, true>, bool> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, onnxTensorDescriptorV1 const*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, onnxTensorDescriptorV1 const*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<char const* const&, onnxTensorDescriptorV1 const*&>(std::integral_constant<bool, true>, char const* const&, onnxTensorDescriptorV1 const*&)
    () from /usr/local/lib/libnvonnxparser.so.0
#7  0x00007ffff2a2c8b9 in onnx2trt::importInputs(onnx2trt::ImporterContext*, onnx2trt_onnx::GraphProto const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, onnx2trt::TensorOrWeights, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, onnx2trt::TensorOrWeights> > >*, unsigned int, onnxTensorDescriptorV1 const*) ()
   from /usr/local/lib/libnvonnxparser.so.0
#8  0x00007ffff2a2dde9 in onnx2trt::ModelImporter::importModel(onnx2trt_onnx::ModelProto const&, unsigned int, onnxTensorDescriptorV1 const*) ()
   from /usr/local/lib/libnvonnxparser.so.0
#9  0x00007ffff2a31254 in onnx2trt::ModelImporter::parseWithWeightDescriptors(void const*, unsigned long, unsigned int, onnxTensorDescriptorV1 const*) ()
   from /usr/local/lib/libnvonnxparser.so.0
#10 0x00007fffc7b9ce42 in onnx_to_tensorrt::onnxToTrtCtx(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, unsigned long, nvinfer1::ILogger::Severity, bool) ()
   from /home/faldict/incubator-mxnet/python/mxnet/../../lib/libmxnet.so
#11 0x00007fffc778a61e in mxnet::op::TRTCreateState(nnvm::NodeAttrs const&, mxnet::Context, std::vector<nnvm::TShape, std::allocator<nnvm::TShape> > const&, std::vector<int, std::allocator<int> > const&) ()
   from /home/faldict/incubator-mxnet/python/mxnet/../../lib/libmxnet.so

At this time I could clearly affirm that this crash occurs during the execution of importInputs(), sourced from the thirdparty onnx-tensorrt. However, I could use onnx2trt binary alone normally. So I guess it passes a null object somewhere. That's all the information I could provide.

[KellenSunderland]

Hey @Faldict I've updated the version of onnx-trt in our repo. I don't think it'll address your issue yet, but you can give the new version a shot.

[KellenSunderland]

Hey @Faldict. (1) Nice machine. (2) I was wondering if you'd be able to test a pre-release version of MXNet 1.3 from a pip package? Could you try a pip install mxnet-tensorrt-cu90 ?

[Faldict]

Hi @KellenSunderland . I have installed mxnet-tensorrt-cu90, but failed to utilize gpu. While running code with gpu context, I meet some errors:

mxnet.base.MXNetError: [10:57:16] /work/mxnet/3rdparty/mshadow/mshadow/././././cuda/tensor_gpu-inl.cuh:110: Check failed: err == cudaSuccess (48 vs. 0) Name: MapPlanKernel ErrStr:no kernel image is available for execution on the device

The cuda version is indeed 9.0. So I wonder what cudnn version it builds on?

By the way, this pip package depends on protobuf 3.5. I wish you could point out critical dependencies. (I reinstalled protobuf 3.5.1 again.)

[KellenSunderland]

I'm trying to link as many packages as possible statically, but have been unable to do so with protobuf yet.

The no kernel image available is a CUDA warning, but it's not specific to CUDA versions. It's actually saying the package doesn't include object code compatible with your GPU (which should be compute capability 6.1).

[lanking520]

@KellenSunderland +1, I currently holds a decent code that can build all binaries but not protobuf...

[lanking520]

@Faldict for CU90 build:

CUDA_VERSION='9.0.176-1'
LIBCUDA_VERSION='396.26-0ubuntu1'
LIBCUDNN_VERSION='7.2.1.38-1+cuda9.0'
LIBNCCL_VERSION='2.2.12-1+cuda9.0'

[KellenSunderland]

Alright, I'm a little limited in what I can ship at the moment due to maximum file sizes in PyPi. I just pushed a version with static protobuf and JIT compilable GPU operators for Pascal cards. This may introduce a small delay when you first load the library as CUDA kernels are JIT'd. This should get you passed the errors you're currently seeing though, so give it an update.

A regular pip upgrade should work, but if not try:
pip install --upgrade --force-reinstall mxnet-tensorrt-cu90

I'm working with the PyPi maintainers to up our limits there, and then I'll be able to make the package more portable. pypi/warehouse#4686

[KellenSunderland]

The diligent PyPi maintainers have enabled extra storage space for our two packages, and I've uploaded a version that has both Pascal and Volta support included. I've also statically compiled a number of libraries to make the lib more portable. Give the new version a shot and see if it addresses your issues.

[Faldict]

@KellenSunderland That problem was probably due to the mismatch of nvidia driver versions. After I fixed the problems and installed the latest pip package mxnet-tensorrt-cu90, TensorRT is usable now. I tried the lenet test case and measured the inference time:

MXNet costs time: 1.380867, TensorRT costs time: 1.026270.

Seems that it works fine! Thanks for your awesome efforts!

[Faldict]

Hi @KellenSunderland, sorry to bother you again.
Could you plz provide a copy of your tensorrt makefile? I wish to build from source on a Jetson TX2 board.

[fighting-liu]

@Faldict @KellenSunderland
I encounter similar problems with yours, i just pull mxnet/tensorrt from official docker hub. But it crashes when i run

import mxnet as mx
a = mx.nd.ones((2, 3), mx.gpu())

Following is the error message

mxnet mxnet_op.h:622: Check failed: (err) == (cudaSuccess) Name: mxnet_generic_kernel ErrStr:no kernel image is available for execution on the device

[KellenSunderland]

FYI build is quite close to what's in CI under the ci/docker/runtime_function.sh file. Hope it helps.