please add some comments on these lines

[yorkerlin]

vadam/pytorch/vadam/optimizers.py

Line 67 in c09d3ac

for s in range(self.num_samples):

See also:

vadam/pytorch/vadam/optimizers.py

Line 92 in c09d3ac

loss = closure()

Make sure that the same mini-batch should be used here.
One way to achieve that is to use state['step'] as the seed of the random number generator for generating mini-batches. I did that in my Autograd implementation

[yorkerlin]

It will be nice to point out that the loss function in

vadam/pytorch/vadam/optimizers.py

Line 92 in c09d3ac

loss = closure()

computes the mean (instead of the sum) of the loss over data points in one mini-batch.