Reversible layers increase memory usage

[serkansulun]

I'm checking memory usage using nvidia-smi. When I turn on reversibility (setting reverse_thres to two times the input length) it's using 8.8 GB memory. When I turn it off (setting reverse_thres to half of the input length), it's using 3.1 GB memory, and it is (naturally) faster. But the memory part doesn't make sense. What can be the problem here?

[lzx325]

Same issue observed here. Is it because that Pytorch autograd is smart enough to identify on itself that it does not need to remember everything to compute the gradients of the reversible layers? Therefore, using the customized _ReversibleFunction will not provide any advantage?