Deploy Swin-Transformer on NVidia Jetson AGX Xavier

[habjoel]

Hi there, I would like to run the swin-transformer fom MMDetection (mmdetection/configs/swin) on a Jetson AGX Xavier. However I am having trouble installing MMDetection on the Jetson. Searching through the web I am starting to think that it is not suited for the arm64 architecture. Is that correct?

If yes, I would like to use ONNX/TensorRT for the deployment of the swin-transformer. I saw that the swin-transformer is not officially supported by MMDeploy yet. When will it be? Also, is there any way how I can convert the model nonetheless?

Thanks a lot for your help!

[AllentDan]

Transformers are not in the model list up to now but will be supported in a month or two. Please refer to doc before installing mmdet.

[habjoel]

Okay, thanks. Could it be possible that I can successfully convert the model to onnx using tools/deploy.py nonetheless?

Also, there's a function pytorch2onnx.py under mmdetection/tools/deployment... do you think that one could work correctly?

[AllentDan]

Okay, thanks. Could it be possible that I can successfully convert the model to onnx using tools/deploy.py nonetheless?

Also, there's a function pytorch2onnx.py under mmdetection/tools/deployment... do you think that one could work correctly?

For models in the support list, yes. The deployment codes in mmdet will be deprecated in the future while still working now.

[habjoel]

Okay, thanks. Could it be possible that I can successfully convert the model to onnx using tools/deploy.py nonetheless?
Also, there's a function pytorch2onnx.py under mmdetection/tools/deployment... do you think that one could work correctly?

For models in the support list, yes. The deployment codes in mmdet will be deprecated in the future while still working now.

Yes, surely it works for models in the support list... but could it also work for the Swin Tranformer already although it's not "officially supported" yet? Like is there a reason why it should fail?

Furthermore, If the provided scripts really do not work for the swin yet, any advice on how I could make the conversion myself would be much appreciated.

[AllentDan]

There are docs for users to rewrite codes that are not suitable for exporting onnx. As for potential ops that are not supported in TensorRT, refer to examples to make the customized one.

[habjoel]

Hi @AllentDan
Is there any news regarding the support of the Swin instance segmentation model support in MMDeploy? I have been trying to make this work without any success.

[AllentDan]

Hi @AllentDan Is there any news regarding the support of the Swin instance segmentation model support in MMDeploy? I have been trying to make this work without any success.

Emm, we were busy with other stuff, and swin is the planned feature by June.

[habjoel]

@AllentDan Alright. By the way, exporting to ONNX worked quite well. Exporting to TensorRT doesn't seem to work yet. I'll try to export it further from ONNX to TensorRT using the TensorRT API for now. It probably won't work though.

The thing is that the inference time of the swin is at about 0.5s on the Jetson using onnxruntime-gpu and about 1s on mmdetection(pytorch) using Images of size 480x640. I am wondering if that's normal for the swin model or if it should be much faster. Could you maybe share some insight on that and to what extent TensorRT could speed this up?

[AllentDan]

Well, it's hard to say as it depends on model structure, the hardware, and so on. But in my experience, TensorRT may be about 0.5x faster than ONNXRuntime or more, and the bonus goes up if the batch size gets enlarged. Besides, fp16 or int8 are also faster than fp32 for TRT.

[lvhan028]

@habjoel Swin Transformer deployment has been released in v0.6.0. You can enjoy it now.

[habjoel]

@lvhan028 cool! thanks a lot for letting me know!