[vm] add support for customized allocator

[MuchToMyDelight]

Hello guys!

After trying the VM in TVM, I found that the default memory allocator(PooledAllocator) doesn't work very well: in case of dynamic inputs, the size of used buffers are different from inference to inference and PooledAllocator cached buffers based on the size. Therefore, it ends with out of device memory.

This patch doesn't aim to solve this problem, since memory allocation is a very complicated issue. I believe users should have the choice to implement whatever suits their needs the most. In addition, the cost of allowing users to register and use their customized allocator is little to none: Registery::Get gets called only once.

I suppose @wweic is the reviewer for VM, what's your opinion?

Signed-off-by: JasonL [email protected]

Thanks for contributing to TVM! Please refer to guideline https://tvm.apache.org/docs/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from Reviewers by @ them in the pull request thread.

[jroesch]

Hey @MuchToMyDelight the patch looks overall good, I wonder if we should allow users to instead register by name or some other key to support multiple allocators to be loaded?

[MuchToMyDelight]

Hey @MuchToMyDelight the patch looks overall good, I wonder if we should allow users to instead register by name or some other key to support multiple allocators to be loaded?

Hello jroesch, I'm not sure what do you mean by 'some other key'. Do you mean add a C++ api like Set/GetAllocator, instead of register by name? I chose register by name because: 1. it makes code clean & neat. 2. There are other places using the same pattern in TVM, e.g. DeviceApiManger uses keys like "device_api.gpu"/"device_api.cpu". So I suppose this is the 'TVM way' of doing this kind of thing.

[MrJungle1]

After trying the VM in TVM, I found that the default memory allocator(PooledAllocator) doesn't work very well: in case of dynamic inputs, the size of used buffers are different from inference to inference and PooledAllocator cached buffers based on the size. Therefore, it ends with out of device memory.

@MuchToMyDelight Hello, I have the same problem on the mobile. Is there any way to solve it.