What is the mAP on the VOC dataset using YOLOv3?

[SkiffDong]

The mAP of my test is 76.8% in 416*416，what about you？

[wonchulSon]

@616848072
How can I check map of test image?
Let me know the command. Thx

[SkiffDong]

You can refer to the tutorial on this website. https://blog.csdn.net/amusi1994/article/details/81564504
If you are not familiar with Chinese, you can use translation software. @wonchulSon

[weiting0032]

@616848072
How many iterations did you train yolov3 on VOC dataset?

[SkiffDong]

@616848072
How many iterations did you train yolov3 on VOC dataset?

@WEITINGLIN32
50200 iterations，i used default parameters in the yolov3_voc.cfg

[weiting0032]

@616848072
OK, I also trained with yolov3_voc.cfg.
And, now it has trained for 39292 and its mAP is about 73%

[SkiffDong]

@616848072
OK, I also trained with yolov3_voc.cfg.
And, now it has trained for 39292 and its mAP is about 73%

@WEITINGLIN32
My friend used the same method and his map is 82.1%......,so I am very confused what‘s wrong with my result

[SkiffDong]

@616848072
OK, I also trained with yolov3_voc.cfg.
And, now it has trained for 39292 and its mAP is about 73%

@WEITINGLIN32
When you are finished training，tell me your result.Thx in advance！

[weiting0032]

@616848072
Maybe he test with resolution 608x608?

[weiting0032]

@616848072
OK, I also trained with yolov3_voc.cfg.
And, now it has trained for 39292 and its mAP is about 73%

@WEITINGLIN32
When you are finished training，tell me your result.Thx in advance！

OK, no problems!

[SkiffDong]

@616848072
Maybe he test with resolution 608x608?

@WEITINGLIN32
he gave me the weights he trained,and i test it in 416x416,the result is 82.7%...even more than his result.If you want the weights he trained,i can give you

[AlexeyAB]

@616848072

he gave me the weights he trained,and i test it in 416x416,the result is 82.7%...even more than his result.If you want the weights he trained,i can give you

Did he train it by using this repository https://github.com/AlexeyAB/darknet or original repository https://github.com/pjreddie/darknet ?

[weiting0032]

@616848072
After I trained yolov3-voc.cfg on VOC dataset, I got the mAP is 75.5% lower than your 76.%
I think your friend perhaps got the global optimum when training the model, so his mAP is too high?

[AlexeyAB]

@WEITINGLIN32
You can get mAP 75.92% by using yolov2-voc.weights, so mAP 75.5% for yolov3-voc.cfg is low.

• Training (2007_train.txt, 2007_val.txt, 2012_train.txt, 2012_val.txt)
• Validation (2007_test.txt)

[weiting0032]

@AlexeyAB
I want to ask why yolov3 mAP is lower than yolov2?

[SkiffDong]

@616848072
After I trained yolov3-voc.cfg on VOC dataset, I got the mAP is 75.5% lower than your 76.%
I think your friend perhaps got the global optimum when training the model, so his mAP is too high?

@WEITINGLIN32
lower...i'm confused why our result are so bad,may be sth wrong.I'll retrain yolov3

[SkiffDong]

@616848072

he gave me the weights he trained,and i test it in 416x416,the result is 82.7%...even more than his result.If you want the weights he trained,i can give you

Did he train it by using this repository https://github.com/AlexeyAB/darknet or original repository https://github.com/pjreddie/darknet ?

@AlexeyAB
Very thx for your remind.i used the original repository and my friend used yours.i'll retrain the model with yours.

[SkiffDong]

@616848072

he gave me the weights he trained,and i test it in 416x416,the result is 82.7%...even more than his result.If you want the weights he trained,i can give you

Did he train it by using this repository https://github.com/AlexeyAB/darknet or original repository https://github.com/pjreddie/darknet ?

@AlexeyAB
Very thx for your remind.i used the original repository and my friend used yours.i'll retrain the model with yours.

@AlexeyAB
Hello，AlexeyAB .I want to ask a question， does the hardware difference have a big impact on the training results?

[AlexeyAB]

@616848072

Hardware should not have any influence.

[SkiffDong]

@616848072

Hardware should not have any influence.

@AlexeyAB
thx for your kind help!

[tdurand]

Is the yolov3-voc.weights file available somewhere ?

[AlexeyAB]

@tdurand No.

[guozhengjin]

@616848072
After I trained yolov3-voc.cfg on VOC dataset, I got the mAP is 75.5% lower than your 76.%
I think your friend perhaps got the global optimum when training the model, so his mAP is too high?

Many people encountered the same problem. mAP of yolov3 on VOC is lower than v2 (76.8% in paper. I trained but got only 75.3%). As for the above 82.7% mAP, I think that result is trained by initializing model using the weight on COCO.

[guozhengjin]

@616848072
OK, I also trained with yolov3_voc.cfg.
And, now it has trained for 39292 and its mAP is about 73%

@WEITINGLIN32
My friend used the same method and his map is 82.1%......,so I am very confused what‘s wrong with my result

Nothing wrong with you. Maybe your friend initialized parameters from COCO weight.

[SkiffDong]

@616848072
OK, I also trained with yolov3_voc.cfg.
And, now it has trained for 39292 and its mAP is about 73%

@WEITINGLIN32
My friend used the same method and his map is 82.1%......,so I am very confused what‘s wrong with my result

Nothing wrong with you. Maybe your friend initialized parameters from COCO weight.

@guozhengjin
maybe，I will ask him for some details.

[SkiffDong]

@AlexeyAB @WEITINGLIN32 @guozhengjin
hello,everyone.This is the result of my training using Alexey's framework.Because I used two GPUs, I changed the Burn-in to 2000 according to the instructions of alexey.Also achieved an accuracy of 82.1.

Saving cached annotations to ./annots.pkl
motorbike : 0.902431864958
train : 0.866592038282
cow : 0.838487451715
horse : 0.914114451726
bus : 0.891987972973
chair : 0.662558691412
tvmonitor : 0.795599971457
bottle : 0.694394590807
sheep : 0.841912873102
sofa : 0.813624575285
boat : 0.728402406606
cat : 0.903439112587
aeroplane : 0.884658602388
bicycle : 0.890120160691
car : 0.902828541956
diningtable : 0.78790748813
pottedplant : 0.547992172733
bird : 0.811727747386
person : 0.85096312896
dog : 0.896658238543

---

mAP : 0.821320104085
user@one:~/darknet$

[SkiffDong]

@AlexeyAB
Thanks for your blog and kind help, they are very valuable.

[SkiffDong]

By the way, when I used the original framework to test the test set, it took 94.370315 seconds. Interestingly, when I used Alexey's framework, it took only 84.000 seconds. The same equipment, different time consuming, don't know why

[guozhengjin]

@AlexeyAB @WEITINGLIN32 @guozhengjin
hello,everyone.This is the result of my training using Alexey's framework.Because I used two GPUs, I changed the Burn-in to 2000 according to the instructions of alexey.Also achieved an accuracy of 82.1.

Saving cached annotations to ./annots.pkl
motorbike : 0.902431864958
train : 0.866592038282
cow : 0.838487451715
horse : 0.914114451726
bus : 0.891987972973
chair : 0.662558691412
tvmonitor : 0.795599971457
bottle : 0.694394590807
sheep : 0.841912873102
sofa : 0.813624575285
boat : 0.728402406606
cat : 0.903439112587
aeroplane : 0.884658602388
bicycle : 0.890120160691
car : 0.902828541956
diningtable : 0.78790748813
pottedplant : 0.547992172733
bird : 0.811727747386
person : 0.85096312896
dog : 0.896658238543

mAP : 0.821320104085
user@one:~/darknet$

@616848072
That is soooo interesting. Did you mean that you used Alexey's darknet framework and got 82.1 mAP, but 76+ mAP for original framework? If so, it is time for me to use Alexey's framework.

[SkiffDong]

@AlexeyAB @WEITINGLIN32 @guozhengjin
hello,everyone.This is the result of my training using Alexey's framework.Because I used two GPUs, I changed the Burn-in to 2000 according to the instructions of alexey.Also achieved an accuracy of 82.1.
Saving cached annotations to ./annots.pkl
motorbike : 0.902431864958
train : 0.866592038282
cow : 0.838487451715
horse : 0.914114451726
bus : 0.891987972973
chair : 0.662558691412
tvmonitor : 0.795599971457
bottle : 0.694394590807
sheep : 0.841912873102
sofa : 0.813624575285
boat : 0.728402406606
cat : 0.903439112587
aeroplane : 0.884658602388
bicycle : 0.890120160691
car : 0.902828541956
diningtable : 0.78790748813
pottedplant : 0.547992172733
bird : 0.811727747386
person : 0.85096312896
dog : 0.896658238543
mAP : 0.821320104085
user@one:~/darknet$

@616848072
That is soooo interesting. Did you mean that you used Alexey's darknet framework and got 82.1 mAP, but 76+ mAP for original framework? If so, it is time for me to use Alexey's framework.

@guozhengjin
when i used the original framework,i didn't change the burn_in to 2000,maybe this had some affect

[guozhengjin]

@616848072
Ok. Anyway, thank you a lot.

[AlexeyAB]

@guozhengjin @616848072

That is soooo interesting. Did you mean that you used Alexey's darknet framework and got 82.1 mAP, but 76+ mAP for original framework? If so, it is time for me to use Alexey's framework.

By the way, when I used the original framework to test the test set, it took 94.370315 seconds. Interestingly, when I used Alexey's framework, it took only 84.000 seconds. The same equipment, different time consuming, don't know why

There are many optimizations, so this repository faster 1.2x - 3x times depending on the case.

[Yeluoqianqiu]

@AlexeyAB @WEITINGLIN32 @guozhengjin
hello,everyone.This is the result of my training using Alexey's framework.Because I used two GPUs, I changed the Burn-in to 2000 according to the instructions of alexey.Also achieved an accuracy of 82.1.

Saving cached annotations to ./annots.pkl
motorbike : 0.902431864958
train : 0.866592038282
cow : 0.838487451715
horse : 0.914114451726
bus : 0.891987972973
chair : 0.662558691412
tvmonitor : 0.795599971457
bottle : 0.694394590807
sheep : 0.841912873102
sofa : 0.813624575285
boat : 0.728402406606
cat : 0.903439112587
aeroplane : 0.884658602388
bicycle : 0.890120160691
car : 0.902828541956
diningtable : 0.78790748813
pottedplant : 0.547992172733
bird : 0.811727747386
person : 0.85096312896
dog : 0.896658238543

mAP : 0.821320104085
user@one:~/darknet$

bro,did you get this mAP following the original cfg or not?can i have a copy of your yolov3-voc.cfg, i trained my own voc in 5 classes:motorbike, bus,car,bicycle,person, in your mAP,they can reach about 90, but mine is only 86. my max_batch is 10000 for 5 classes, did you really train them for 50000+ batches? my qq:838345014, waiting for your reply,thx.

[reactivetype]

@616848072 can you please share your cfg file and weights?

[gathierry]

@AlexeyAB @WEITINGLIN32 @guozhengjin
hello,everyone.This is the result of my training using Alexey's framework.Because I used two GPUs, I changed the Burn-in to 2000 according to the instructions of alexey.Also achieved an accuracy of 82.1.

Saving cached annotations to ./annots.pkl
motorbike : 0.902431864958
train : 0.866592038282
cow : 0.838487451715
horse : 0.914114451726
bus : 0.891987972973
chair : 0.662558691412
tvmonitor : 0.795599971457
bottle : 0.694394590807
sheep : 0.841912873102
sofa : 0.813624575285
boat : 0.728402406606
cat : 0.903439112587
aeroplane : 0.884658602388
bicycle : 0.890120160691
car : 0.902828541956
diningtable : 0.78790748813
pottedplant : 0.547992172733
bird : 0.811727747386
person : 0.85096312896
dog : 0.896658238543

mAP : 0.821320104085
user@one:~/darknet$

hi @616848072 @AlexeyAB any idea what makes this difference? 7% on mAP is not a small improvement. Could you kindly share the modifications you make?