[Contrib] Workspace for cuBLAS backend

[MasterJH5574]

This PR adds a 32MB workspace for cuBLAS backend, so that functions like cublasLtMatmul can take the workspace as input.

The workspace is managed under CuBlasThreadEntry so that it will be allocated only once in each thread.

[MasterJH5574]

cc @vinx13 @masahi @junrushao @tqchen

[masahi]

There is a pass to allocate such workspace and append it to arguments of a BYOC function: https://github.com/apache/tvm/blob/unity/python/tvm/relax/transform/transform.py#L1305-L1317

If this can be used, I think that would be preferred.

[MasterJH5574]

There is a pass to allocate such workspace and append it to arguments of a BYOC function: https://github.com/apache/tvm/blob/unity/python/tvm/relax/transform/transform.py#L1305-L1317

If this can be used, I think that would be preferred.

@masahi Thank you for the great suggestion! Yes I think this one can be a next step, so that the workspace size can be adjustable. For now, given I am not particular familiar with the BYOC flow, I may not be able to quickly enable the workspace to work with the AllocateWorksapce pass. Adding a fixed size workspace in thread entry is the fastest way to enable workspace. I agree that we can leverage the pass later on.

[junrushao]

@masahi would you mind sharing more guidance to @MasterJH5574?

[masahi]

I'm assuming that 32MB is also good for pre-Hopper since it is bigger than the recommended size, 4MB. @vinx13

[masahi]

Ok, given the very specific nature of this workspace, I think a fast path like this is reasonable. The AllocateWorkspace-based approach can be used in more general settings but it is overkill when we only need one workspace of a fixed size.