Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug][TVMC] Map is not supported by RPC #11707

Open
lhutton1 opened this issue Jun 14, 2022 · 2 comments
Open

[Bug][TVMC] Map is not supported by RPC #11707

lhutton1 opened this issue Jun 14, 2022 · 2 comments

Comments

@lhutton1
Copy link
Contributor

lhutton1 commented Jun 14, 2022

Hi, I'm attempting to run a model via RPC and TVMC but I'm running into an issue:

ValueError: Cannot pass argument 1, type Map is not supported by RPC

The TVMC command I'm using is similar to:

tvmc run --repeat=1 --print-top=5 --rpc-tracker=mytracker.com mymodel.tar

I seem to be able to get the model to run again when I revert #9889 which introduced get_input_info in the graph executor. Is this a case of RPC missing an implementation for Map, or should we do something in TVMC to handle get_input_info differently?

Full backtrace:

tvm.error.RPCError: Traceback (most recent call last):
  8: TVMFuncCall
        at ../src/runtime/c_runtime_api.cc:477
  7: tvm::runtime::PackedFuncObj::CallPacked(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
        at ../include/tvm/runtime/packed_func.h:1217
  6: tvm::runtime::RPCWrappedFunc::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
        at ../src/runtime/rpc/rpc_module.cc:127
  5: tvm::runtime::RPCClientSession::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)> const&)
        at ../src/runtime/rpc/rpc_endpoint.cc:1009
  4: tvm::runtime::RPCEndpoint::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)>)
        at ../src/runtime/rpc/rpc_endpoint.cc:800
  3: tvm::runtime::RPCEndpoint::HandleUntilReturnEvent(bool, std::function<void (tvm::runtime::TVMArgs)>)
        at ../src/runtime/rpc/rpc_endpoint.cc:644
  2: tvm::runtime::RPCEndpoint::EventHandler::HandleNextEvent(bool, bool, std::function<void (tvm::runtime::TVMArgs)>)
        at ../src/runtime/rpc/rpc_endpoint.cc:135
  1: tvm::runtime::RPCEndpoint::EventHandler::HandleProcessPacket(std::function<void (tvm::runtime::TVMArgs)>)
        at ../src/runtime/rpc/rpc_endpoint.cc:315
  0: tvm::runtime::RPCEndpoint::EventHandler::HandleReturn(tvm::runtime::RPCCode, std::function<void (tvm::runtime::TVMArgs)>)
        at ../src/runtime/rpc/rpc_endpoint.cc:376
  [bt] (8) libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::HandleNextEvent(bool, bool, std::function<void (tvm::runtime::TVMArgs)>)+0x464) [0xffffacf33c54]
  [bt] (7) libtvm_runtime.so(tvm::runtime::RPCSession::AsyncCallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::RPCCode, tvm::runtime::TVMArgs)>)+0x48) [0xffffacf45dd0]
  [bt] (6) libtvm_runtime.so(tvm::runtime::LocalSession::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)> const&)+0x84) [0xffffacf38fac]
  [bt] (5) libtvm_runtime.so(tvm::runtime::LocalSession::EncodeReturn(tvm::runtime::TVMRetValue, std::function<void (tvm::runtime::TVMArgs)> const&)+0x74) [0xffffacf38e1c]
  [bt] (4) libtvm_runtime.so(+0x125ac4) [0xffffacf45ac4]
  [bt] (3) libtvm_runtime.so(std::_Function_handler<void (tvm::runtime::RPCCode, tvm::runtime::TVMArgs), tvm::runtime::RPCEndpoint::EventHandler::HandleNormalCallFunc()::{lambda(tvm::runtime::RPCCode, tvm::runtime::TVMArgs)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::RPCCode&&, tvm::runtime::TVMArgs&&)+0x4c) [0xffffacf31aa4]
  [bt] (2) libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::ValidateArguments(TVMValue const*, int const*, int)+0x1ec) [0xffffacf311f4]
  [bt] (1) tvm_rpc(tvm::runtime::detail::LogFatal::Entry::Finalize()+0x6c) [0x40fef4]
  [bt] (0) libtvm_runtime.so(tvm::runtime::Backtrace[abi:cxx11]()+0x1c) [0xffffaceb6d84]
  File "../src/runtime/rpc/rpc_endpoint.cc", line 376
RPCError: Error caught from RPC call:
ValueError: Cannot pass argument 1, type Map is not supported by RPC

cc @Mousius @gromero @leandron @leeexyz @masahi

lhutton1 added a commit to lhutton1/tvm that referenced this issue Jun 17, 2022
As described in apache#11707, the RPC mechanism does not support
objects of type Map which breaks the use of tvmc run when using
RPC after apache#9889. This commit intends to workaround this issue by
providing a fallback to the old implementation when RPC is being
used. Further, a unit test has been provided to help prevent this
regression in the future.

Change-Id: I70c1863d00098270e27c08ba834a3587e9132d69
lhutton1 added a commit to lhutton1/tvm that referenced this issue Jun 17, 2022
As described in apache#11707, the RPC mechanism does not support
objects of type Map which breaks the use of tvmc run when using
RPC after apache#9889. This commit intends to workaround this issue by
providing a fallback to the old implementation when RPC is being
used. Further, a test has been provided to help prevent this
regression in the future.

Change-Id: I70c1863d00098270e27c08ba834a3587e9132d69
leandron pushed a commit that referenced this issue Jun 20, 2022
* [TVMC] Fix tvmc run when using rpc

As described in #11707, the RPC mechanism does not support
objects of type Map which breaks the use of tvmc run when using
RPC after #9889. This commit intends to workaround this issue by
providing a fallback to the old implementation when RPC is being
used. Further, a test has been provided to help prevent this
regression in the future.

Change-Id: I70c1863d00098270e27c08ba834a3587e9132d69

* fix lint

Change-Id: I958cf4e19988d047bdd2e02f6475b9f70afe80c8
@lhutton1
Copy link
Contributor Author

Submitted #11757 to work around the issue for now.

lhutton1 added a commit to lhutton1/tvm that referenced this issue Jun 21, 2022
* [TVMC] Fix tvmc run when using rpc

As described in apache#11707, the RPC mechanism does not support
objects of type Map which breaks the use of tvmc run when using
RPC after apache#9889. This commit intends to workaround this issue by
providing a fallback to the old implementation when RPC is being
used. Further, a test has been provided to help prevent this
regression in the future.

Change-Id: I70c1863d00098270e27c08ba834a3587e9132d69

* fix lint

Change-Id: I958cf4e19988d047bdd2e02f6475b9f70afe80c8
blackkker pushed a commit to blackkker/tvm that referenced this issue Jul 7, 2022
* [TVMC] Fix tvmc run when using rpc

As described in apache#11707, the RPC mechanism does not support
objects of type Map which breaks the use of tvmc run when using
RPC after apache#9889. This commit intends to workaround this issue by
providing a fallback to the old implementation when RPC is being
used. Further, a test has been provided to help prevent this
regression in the future.

Change-Id: I70c1863d00098270e27c08ba834a3587e9132d69

* fix lint

Change-Id: I958cf4e19988d047bdd2e02f6475b9f70afe80c8
@tqchen
Copy link
Member

tqchen commented Jul 26, 2022

This is the intended behavior of RPC for now to restrict the object that we can support (so we can support minimum cases like uTVM). Would be a good starting pt to document the related behavior

@areusch areusch added the needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it label Oct 19, 2022
@hpanda-naut hpanda-naut added core:rpc and removed needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it labels Nov 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants