Own dataset doesn't work on latest commit

[RickvanHek]

For some reason I can't seem to train my own dataset on the latest commit. I am able to do it from an earlier commit e.g. this state. In this state if i run my training (with the exact same cfg files, dataset etc), i get these results after a couple epochs:

      Epoch      Batch          x          y          w          h       conf        cls      total          P          R   nTargets         TP         FP         FN       time
       0/99      99/99       1.49       1.47       7.46       12.8        111       7.38        141          0          0          3          0    2.9e+03          0      0.124
       1/99      99/99       1.26       1.19       1.99       3.07       16.1       7.35       30.9          0          0         10          0          1          8      0.131
       2/99      99/99       1.04       1.02      0.831       1.08       4.64       7.25       15.9          0          0          7          0          3          4      0.129
       3/99      99/99      0.756      0.796      0.666       0.79       3.67       7.25       13.9   0.000769    0.00187         10          0          2          7      0.129
       4/99      99/99       0.58      0.683      0.574      0.739       2.93       7.15       12.7    0.00314     0.0167          3          0          3          1      0.129
       5/99      99/99      0.455       0.54      0.462       0.64       2.62       7.14       11.9    0.00636     0.0221          6          0          6          1      0.132

If I use the latest commit, i get this:

      Epoch      Batch          x          y          w          h       conf        cls      total          P          R   nTargets         TP         FP         FN       time
       0/99      99/99       1.49       1.47       7.41       12.6        111       7.38        141          0          0          3          0          0          0      0.117
       1/99      99/99       4.91       4.93        nan        nan        nan        nan        nan          0          0         10          0          0          0      0.125
       2/99      99/99       5.52       5.24        nan        nan        nan        nan        nan          0          0          7          0          0          0      0.124
       3/99      99/99       5.23       5.21        nan        nan        nan        nan        nan          0          0         10          0          0          0      0.129
       4/99      99/99       5.35       5.17        nan        nan        nan        nan        nan          0          0          3          0          0          0      0.125
       5/99      99/99       5.65       5.41        nan        nan        nan        nan        nan          0          0          6          0          0          0      0.124
       6/99      99/99       5.13       5.13        nan        nan        nan        nan        nan          0          0          9          0          0          0      0.126

Also in the latest commit, line 197 of train.py causes the following error:

Traceback (most recent call last):
  File "/home/rick/Documents/yolov3-master/train.py", line 208, in <module>
    main(opt)
  File "/home/rick/Documents/yolov3-master/train.py", line 195, in main
    mAP, R, P = test.main(test.opt)
  File "/home/rick/Documents/yolov3-master/test.py", line 42, in main
    model.load_state_dict(checkpoint['model'])
  File "/media/rick/HDD/Env/yolov3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 719, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Darknet:
	size mismatch for module_list.81.conv_81.weight: copying a param of torch.Size([255, 1024, 1, 1]) from checkpoint, where the shape is torch.Size([303, 1024, 1, 1]) in current model.
	size mismatch for module_list.81.conv_81.bias: copying a param of torch.Size([255]) from checkpoint, where the shape is torch.Size([303]) in current model.
	size mismatch for module_list.93.conv_93.weight: copying a param of torch.Size([255, 512, 1, 1]) from checkpoint, where the shape is torch.Size([303, 512, 1, 1]) in current model.
	size mismatch for module_list.93.conv_93.bias: copying a param of torch.Size([255]) from checkpoint, where the shape is torch.Size([303]) in current model.
	size mismatch for module_list.105.conv_105.weight: copying a param of torch.Size([255, 256, 1, 1]) from checkpoint, where the shape is torch.Size([303, 256, 1, 1]) in current model.
	size mismatch for module_list.105.conv_105.bias: copying a param of torch.Size([255]) from checkpoint, where the shape is torch.Size([303]) in current model.

So I replaced it with (the old) code:

        with open('results.txt', 'a') as file:
            file.write(s + '\n')

I don't know if this is normal but i can't seem to find a solution.
Do you know why i get nan whilst using the exact same cfg files and data? My txt files for each image is spot on, the bounding boxes, width, height etc are all relative to the image width and height.

[glenn-jocher]

@RickvanHek Sorry, its possible one of the latest commits broke something, as I've been updating quite often. I've been running internal tests with a more advanced version with no issues. I just committed these changes now as b07ee41. Can you try to download this from scratch and rerun?

If this works I'd advise you to check back in a day or two, I'm compiling updates that should increase performance significantly. These updates are not reflected in the latest commit, they are being tracked in #2 (comment) and should be implemented in one single commit in the next few days.

FYI in your particular case you may have issues with your local dataset with SGD burn-in, which is supposed to last the first 1000 batches (you only have 99). So your first epoch may be doing nothing really, as the lr is programmed to ramp exponentially from zero to the initial lr = 0.001 over those 1000 batches.

yolov3/train.py

Lines 139 to 143 in b07ee41

	# SGD burn-in
	if (epoch == 0) & (i <= 1000):
	lr = lr0 * (i / 1000) ** 4
	for g in optimizer.param_groups:
	g['lr'] = lr

[RickvanHek]

@RickvanHek Sorry, its possible one of the latest commits broke something, as I've been updating quite often. I've been running internal tests with a more advanced version with no issues. I just committed these changes now as b07ee41. Can you try to download this from scratch and rerun?

If this works I'd advise you to check back in a day or two, I'm compiling updates that should increase performance significantly. These updates are not reflected in the latest commit, they are being tracked in #2 (comment) and should be implemented in one single commit in the next few days.

FYI in your particular case you may have issues with your local dataset with SGD burn-in, which is supposed to last the first 1000 batches (you only have 99). So your first epoch may be doing nothing really, as the lr is programmed to ramp exponentially from zero to the initial lr = 0.001 over those 1000 batches.
yolov3/train.py

Lines 139 to 143 in b07ee41

SGD burn-in

if (epoch == 0) & (i <= 1000):
lr = lr0 * (i / 1000) ** 4
for g in optimizer.param_groups:
g['lr'] = lr

Thank you for the fast reply, when I disable the SGD (on the last commit, like you said), I don't get the nan anymore, if i enable it i still get these nan's. These are my results after 4 epochs,

      Epoch      Batch          x          y          w          h       conf        cls      total          P          R   nTargets         TP         FP         FN       time
       0/99      99/99       1.31       1.28       3.92        8.5       9.03       7.27       31.3          0          0          3          0          0          0     0.0571
       1/99      99/99       1.21       1.11       1.04       1.34        3.3       7.27       15.3          0          0         10          0          0          0      0.115
       2/99      99/99       1.16       1.09       1.07       1.28       2.84       7.19       14.6          0          0          7          0          0          0      0.116
       3/99      99/99       1.04      0.963      0.891       1.09       2.61       7.21       13.8          0          0         10          0          0          0      0.115

My dataset has 70000 training and validation images, I was just using a 100 for testing purpose.
I am now running it on 7000 training images to see if it'll work.

[RickvanHek]

I did 6 epochs on 7000 images (batch size = 1), and i still got 0 precision, here are the results:

      Epoch      Batch          x          y          w          h       conf        cls      total          P          R   nTargets         TP         FP         FN       time
       0/99  6999/6999       0.33      0.315      0.789      0.896       6.69       6.02         15          0          0         10          0          0          0     0.0937
       1/99  6999/6999      0.144      0.133      0.474      0.463      0.836       3.41       5.46          0          0          7          0          0          0      0.116
       2/99  6999/6999     0.0981     0.0887       0.44      0.376      0.637       1.93       3.57          0          0          6          0          0          0      0.116
       3/99  6999/6999     0.0824     0.0764       0.39      0.321      0.576       1.13       2.58          0          0          6          0          0          0      0.121
       4/99  6999/6999     0.0725      0.065      0.347      0.261      0.522      0.662       1.93          0          0         10          0          0          0      0.117
       5/99  6999/6999     0.0666     0.0592      0.285      0.213      0.495      0.428       1.55          0          0          4          0          0          0      0.131

Any idea why i'm getting 0 precision? Left the code untouched only edited the writing to result.txt (to not do an mAp calculation, because this gives an error for me)

[glenn-jocher]

Ah that's perfect, everything is fine. The new -batch_report flag in train.py controls whether P, R, TP, FP, and FN print to screen during training. Printing to screen makes training about 20% slower so I set it to False by default, but you can turn this back on to see those print to screen.

The mAP calculation in test.py now adds P and R to results.txt in the last 3 columns. Is test.py still crashing?

[RickvanHek]

Ah that's perfect, everything is fine. The new -batch_report flag in train.py controls whether P, R, TP, FP, and FN print to screen during training. Printing to screen makes training about 20% slower so I set it to False by default, but you can turn this back on to see those print to screen.

The mAP calculation in test.py now adds P and R to results.txt in the last 3 columns. Is test.py still crashing?

Oh that's why, didn't know that.
And yes it still crashing, this is my error:

Traceback (most recent call last):
  File "/home/rick/Documents/yolov3/train.py", line 218, in <module>
    main(opt)
  File "/home/rick/Documents/yolov3/train.py", line 209, in main
    mAP, R, P = test.main(test.opt)
  File "/home/rick/Documents/yolov3/test.py", line 42, in main
    model.load_state_dict(checkpoint['model'])
  File "/media/rick/HDD/Env/latestyolov3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 719, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Darknet:
	size mismatch for module_list.81.conv_81.weight: copying a param of torch.Size([255, 1024, 1, 1]) from checkpoint, where the shape is torch.Size([303, 1024, 1, 1]) in current model.
	size mismatch for module_list.81.conv_81.bias: copying a param of torch.Size([255]) from checkpoint, where the shape is torch.Size([303]) in current model.
	size mismatch for module_list.93.conv_93.weight: copying a param of torch.Size([255, 512, 1, 1]) from checkpoint, where the shape is torch.Size([303, 512, 1, 1]) in current model.
	size mismatch for module_list.93.conv_93.bias: copying a param of torch.Size([255]) from checkpoint, where the shape is torch.Size([303]) in current model.
	size mismatch for module_list.105.conv_105.weight: copying a param of torch.Size([255, 256, 1, 1]) from checkpoint, where the shape is torch.Size([303, 256, 1, 1]) in current model.
	size mismatch for module_list.105.conv_105.bias: copying a param of torch.Size([255]) from checkpoint, where the shape is torch.Size([303]) in current model.

[glenn-jocher]

Ok. Layers 81, 93 and 105 are the yolo detection layers. 80 classes makes them size 255 but your model seems to have more classes as you can see.

If you are using a custom dataset you need to adjust test.py to work with it, just like you adjust train.py (i.e. set -class_path, -cfg, -data_config_path correctly), otherwise yes, it will crash.

[glenn-jocher]

@RickvanHek I just had an idea that might smooth things a bit after seeing #40 recently. If I pass the train.py namespace arguments (-class_path, -cfg, -data_config_path) to test.py then you wouldn't need to define them twice (during training).

If you ran test.py by itself you'd still need to properly set them again though. Do you think this change might have solved your training issues?

[RickvanHek]

@RickvanHek I just had an idea that might smooth things a bit after seeing #40 recently. If I pass the train.py namespace arguments (-class_path, -cfg, -data_config_path) to test.py then you wouldn't need to define them twice (during training).

If you ran test.py by itself you'd still need to properly set them again though. Do you think this change might have solved your training issues?

Hello, this fixed the problem! Thank you!