RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn for running maml on TVM backend with CUDA

[shingjan]

My repro:

python benchmarks/torchbench.py --only maml --backend tvm -dcuda

The error message I am seeing:

Traceback (most recent call last):
  File "/home/yj/torchdynamo/benchmarks/common.py", line 974, in run_one_model
    new_result = model_iter_fn(model, example_inputs)
  File "benchmarks/torchbench.py", line 363, in forward_pass
    def forward_pass(self, mod, inputs, collect_outputs=True):
  File "/home/yj/anaconda3/envs/torchdynamo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1186, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/yj/benchmark/torchbenchmark/models/maml/meta.py", line 64, in forward
    def forward(self, x_spt, y_spt, x_qry, y_qry):
  File "/home/yj/benchmark/torchbenchmark/models/maml/meta.py", line 155, in finetunning
    def finetunning(self, x_spt, y_spt, x_qry, y_qry):
  File "/home/yj/benchmark/torchbenchmark/models/maml/meta.py", line 155, in finetunning
    def finetunning(self, x_spt, y_spt, x_qry, y_qry):
  File "/home/yj/anaconda3/envs/torchdynamo/lib/python3.8/site-packages/torch/autograd/__init__.py", line 294, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

This error doesn't seem to be a tvm but cannot be reproduced by other backends like TS or inductor. I am using @jansel 's branch of torchbench. Would that be possible that a rebase on torchbench is required?

Thanks!

[jansel]

maml "inference" actually requires training support. It does online finetuning of (a copy of) the model in order to make a prediction.

Does the tvm backend support training?

[shingjan]

I think training support is on the roadmap of the upcoming Relax frontend link. So probably I can circle back once that is available.