support tensor parallel #2631

Job	Run time
build-and-push-docker-images	19s
build-and-test-cpp-aarch64 (openblas)	4m 7s
build-and-test-cpp-x86_64 (ubuntu-20.04, mkl)	5m 41s
build-python-wheels (ubuntu-20.04, auto64)	19m 3s
build-and-test-cpp-x86_64 (ubuntu-20.04, dnnl)	5m 21s
build-python-wheels (macos-11, auto64)	19m 16s
check-python-style	10s
build-and-test-cpp-x86_64 (macos-11, mkl)	11m 10s
build-python-wheels (windows-2019, auto64)	19m 31s
build-and-test-cpp-x86_64 (macos-11, dnnl)	11m 31s
build-python-wheels (ubuntu-20.04, aarch64)	19m 22s
build-python-wheels (macos-11, arm64)	19m 17s
test-python-wheels	0s
build-and-deploy-docs	0s
publish-python-wheels-on-pypi	0s
	2h 14m 48s

Provide feedback