DIoU and CIOU loss implementation #4360

simon-rob · 2019-11-22T17:30:00Z

https://github.com/Zzh-tju/DIoU-darknet
This already implemented in the about repository.

AlexeyAB · 2019-11-22T19:04:19Z

Paper: https://arxiv.org/abs/1911.08287v1

loss	AP
iou loss	46.57
giou loss	47.73
diou loss	48.10
ciou loss	49.21

LukeAI · 2019-11-23T10:30:12Z

Looks like a really nice big "free" performance boost! Can't wait to test it!

AlexeyAB · 2019-11-23T13:49:52Z

I added CIoU and DIoU to both [yolo] and [Gaussian_yolo] layers. But didn't test yet.

@LukeAI Did you test iou_thresh=0.213 or 0.3?

tuteming · 2019-11-23T15:47:32Z

if you tested. please give me the corresponding cfg file. thanks

AlexeyAB · 2019-11-23T18:02:38Z

@tuteming I fixed several bugs.

cfg-file with [Gaussian_yolo] + CIoU: yolov3-tiny_pan_ciou.cfg.txt

glenn-jocher · 2019-11-24T02:40:16Z

Very interesting!! I will try to implement DIoU and CIoU in ultralytics/yolov3 and test on COCO vs our default GIoU.

lq0104 · 2019-11-25T03:36:08Z

Hi @AlexeyAB I'm confused about the "scale_x_y" param. In this "yolov3-tiny_pan_ciou.cfg.txt", the "scale_x_y" params are as follows:

[Gaussian_yolo]
mask = 0,1,2,3,4
anchors = 8,8, 10,13, 16,30, 33,23, 32,32, 30,61, 62,45, 64,64, 59,119, 116,90, 156,198, 373,326
scale_x_y = 1.05

[Gaussian_yolo]
mask = 4,5,6,7,8
anchors = 8,8, 10,13, 16,30, 33,23, 32,32, 30,61, 62,45, 64,64, 59,119, 116,90, 156,198, 373,326
scale_x_y = 1.1

[Gaussian_yolo]
mask = 8,9,10,11
anchors = 8,8, 10,13, 16,30, 33,23, 32,32, 30,61, 62,45, 59,119, 80,80, 116,90, 156,198, 373,326
scale_x_y = 1.2

But In your topic about "Sensitivity Effects Near Grid Boundaries (Experimental Results) ~+1 AP@[.5, [.95]" #3293, you say the rule of setting param "scale_x_y" as follows:

1.05 * logistic - 0.025 - for yolo layer (large objects) scale_x_y = 1.05 in cfg-file
1.1 * logistic - 0.05 - for yolo layer (medium objects) scale_x_y = 1.1 in cfg-file
1.2 * logistic - 0.1 - for yolo layer (small objects) scale_x_y = 1.2 in cfg-file

so my question is: for large objects, like mask = 8,9,10,11, according to the rule, the value of scale_x_y should be 1.05, it's in conflict with "yolov3-tiny_pan_ciou.cfg.txt" setting, so which one is right then?

AlexeyAB · 2019-11-25T11:14:20Z

@lq0104 Yes, my mistake )

Should be:

1.05 * logistic - 0.025 - for yolo layer (large objects) scale_x_y = 1.05 in cfg-file
1.1 * logistic - 0.05 - for yolo layer (medium objects) scale_x_y = 1.1 in cfg-file
1.2 * logistic - 0.1 - for yolo layer (small objects) scale_x_y = 1.2 in cfg-file

Or 1.1 for all yolo-layers.

glenn-jocher · 2019-11-25T18:17:14Z

I tested the 3 box regression methods below on https://github.com/ultralytics/yolov3 using yolov3-spp.cfg with swish trained on full COCO2014 to 27 epochs each, but was not able to realize performance improvements with the new methods. I'll try again with LeakyReLU(0.1). The IoU function I implemented is here:

python3 train.py --weights '' --epochs 27 --batch-size 16 --accumulate 4 --prebias --cfg cfg/yolov3s.cfg

	[email protected]	mAP0.5:0.95	Epoch time on 2080Ti
GIoU	49.7	30.2	36min
DIoU	49.4	30.0	36min
CIoU	49.7	30.1	36min

AlexeyAB · 2019-11-25T18:50:17Z

@glenn-jocher

When using ciou - the model converges faster and accuracy increases faster during training? Or no change at all?

They also introduced new NMS function, that can be enabled in the last [yolo] layer by set

nms_kind = greedynms
beta_nms = 0.6

Mainly introduced beta1 - source code:

darknet/src/box.c

Lines 201 to 218 in b832c72

    
           float box_diounms(box a, box b, float beta1) 
        
           { 
        
               boxabs ba = box_c(a, b); 
        
               float w = ba.right - ba.left; 
        
               float h = ba.bot - ba.top; 
        
               float c = w * w + h * h; 
        
               float iou = box_iou(a, b); 
        
               if (c == 0) { 
        
                   return iou; 
        
               } 
        
               float d = (a.x - b.x) * (a.x - b.x) + (a.y - b.y) * (a.y - b.y); 
        
               float u = pow(d / c, beta1); 
        
               float diou_term = u; 
        
           #ifdef DEBUG_PRINTS 
        
               printf("  c: %f, u: %f, riou_term: %f\n", c, u, diou_term); 
        
           #endif 
        
               return iou - diou_term; 
        
           }

nms_kind == CORNERS_NMS is my experiment

darknet/src/box.c

Lines 855 to 917 in b832c72

    
           void diounms_sort(detection *dets, int total, int classes, float thresh, NMS_KIND nms_kind, float beta1) 
        
           { 
        
               int i, j, k; 
        
               k = total - 1; 
        
               for (i = 0; i <= k; ++i) { 
        
                   if (dets[i].objectness == 0) { 
        
                       detection swap = dets[i]; 
        
                       dets[i] = dets[k]; 
        
                       dets[k] = swap; 
        
                       --k; 
        
                       --i; 
        
                   } 
        
               } 
        
               total = k + 1; 
        
               for (k = 0; k < classes; ++k) { 
        
                   for (i = 0; i < total; ++i) { 
        
                       dets[i].sort_class = k; 
        
                   } 
        
                   qsort(dets, total, sizeof(detection), nms_comparator_v3); 
        
                   for (i = 0; i < total; ++i) 
        
                   { 
        
                       if (dets[i].prob[k] == 0) continue; 
        
                       box a = dets[i].bbox; 
        
                       for (j = i + 1; j < total; ++j) { 
        
                           box b = dets[j].bbox; 
        
                           if (box_iou(a, b) > thresh && nms_kind == CORNERS_NMS) 
        
                           { 
        
                               float sum_prob = pow(dets[i].prob[k], 2) + pow(dets[j].prob[k], 2); 
        
                               float alpha_prob = pow(dets[i].prob[k], 2) / sum_prob; 
        
                               float beta_prob = pow(dets[j].prob[k], 2) / sum_prob; 
        
                               //dets[i].bbox.x = (dets[i].bbox.x*alpha_prob + dets[j].bbox.x*beta_prob); 
        
                               //dets[i].bbox.y = (dets[i].bbox.y*alpha_prob + dets[j].bbox.y*beta_prob); 
        
                               //dets[i].bbox.w = (dets[i].bbox.w*alpha_prob + dets[j].bbox.w*beta_prob); 
        
                               //dets[i].bbox.h = (dets[i].bbox.h*alpha_prob + dets[j].bbox.h*beta_prob); 
        
                               /* 
        
                               if (dets[j].points == YOLO_CENTER && (dets[i].points & dets[j].points) == 0) { 
        
                                   dets[i].bbox.x = (dets[i].bbox.x*alpha_prob + dets[j].bbox.x*beta_prob); 
        
                                   dets[i].bbox.y = (dets[i].bbox.y*alpha_prob + dets[j].bbox.y*beta_prob); 
        
                               } 
        
                               else if ((dets[i].points & dets[j].points) == 0) { 
        
                                   dets[i].bbox.w = (dets[i].bbox.w*alpha_prob + dets[j].bbox.w*beta_prob); 
        
                                   dets[i].bbox.h = (dets[i].bbox.h*alpha_prob + dets[j].bbox.h*beta_prob); 
        
                               } 
        
                               dets[i].points |= dets[j].points; 
        
                               */ 
        
                               dets[j].prob[k] = 0; 
        
                           } 
        
                           else if (box_iou(a, b) > thresh && nms_kind == GREEDY_NMS) { 
        
                               dets[j].prob[k] = 0; 
        
                           } 
        
                           else { 
        
                               if (box_diounms(a, b, beta1) > thresh && nms_kind == DIOU_NMS) { 
        
                                   dets[j].prob[k] = 0; 
        
                               } 
        
                           } 
        
                       } 
        
                       //if ((nms_kind == CORNERS_NMS) && (dets[i].points != (YOLO_CENTER | YOLO_LEFT_TOP | YOLO_RIGHT_BOTTOM))) 
        
                       //    dets[i].prob[k] = 0; 
        
                   } 
        
               } 
        
           }

glenn-jocher · 2019-11-25T19:14:04Z

The 3 really didn't differ much throughout the entire training oddly enough. I did see CIoU help out more on a custom small dataset I had, so perhaps it helps more when there is less training data. The results73, 74 and 75 are G, D and CIoU.

Did you see any difference when using the different NMS techniques? I tried to implement soft-nms before without success.

AlexeyAB · 2019-11-25T19:42:46Z

@glenn-jocher

Did you check [email protected] and [email protected] ?
Did you check [email protected], [email protected] and [email protected] with confidence_threshold=0.1 instead of 0.001?

May be CIoU and DIoU improves only [email protected] - [email protected] like as GIoU: #3249 (comment)

I just tested [email protected] on the default yolov3-spp.cfg / weights 608x608:
./darknet detector map cfg/coco.data cfg/yolov3-spp.cfg yolov3-spp.weights -points 101

default NMS: 59.28% [email protected]
nms_kind = diounms beta_nms=0.8: 59.40% [email protected] so +0.12
nms_kind = diounms beta_nms=0.7: 59.43% [email protected] so +0.15
nms_kind = diounms beta_nms=0.6: 59.46% [email protected] so +0.18 <---
nms_kind = diounms beta_nms=0.5: 59.40% [email protected] so +0.12
nms_kind = diounms beta_nms=0.4: 59.17% [email protected] so -0.11

(beta_nms=0.6 by default)

May be if yolov3-spp.cfg will be trained with CIoU then nms_kind = DIOU_NMS, or if we will use lower beta then it can get better result.

LukeAI · 2019-11-26T11:56:41Z

CIOU hurt the mAP in my run
#4147 (comment)

nyj-ocean · 2019-12-02T06:24:21Z

@AlexeyAB
As shown in #4360 (comment)

Should be:

1.05 * logistic - 0.025 - for yolo layer (large objects) scale_x_y = 1.05 in cfg-file
1.1 * logistic - 0.05 - for yolo layer (medium objects) scale_x_y = 1.1 in cfg-file
1.2 * logistic - 0.1 - for yolo layer (small objects) scale_x_y = 1.2 in cfg-file

Or 1.1 for all yolo-layers.

But I found in Gaussian_yolov3_BDD.cfg , The value of scale_x_y is set to 1.0 for all 3 yolo-layers

Which one of the following should I set to my yolov3+Gaussian+CIoU.cfg?

scale_x_y = 1.05, 1.1, 1.2 for different yolo-layers ?
scale_x_y = 1.1 for all 3 yolo-layers ?
scale_x_y = 1.0 for all 3 yolo-layers ?

Or should I train 3 times and then choose the one with best mAP ?

dselivanov · 2019-12-02T06:37:04Z

Honestly guys I don't understand how giou, diou, ciou is better than just IOU for YOLOv3 framework. The main problem they all address compared to vanilla iou is that loss is not differential if intersection of an anchor and a target is 0. However in the yolo framework it is always non-zero - we pick the anchor with largest IOU with target. Otherwise box loss doesn't contribute to overall loss.

So I'm sceptical about these papers and reported results. At least in the experiments using my own pytorch yolo implementation with custom datasets I'm getting worse results with *IOU losses compared to original yolo loss.

AlexeyAB · 2019-12-02T12:14:07Z

@LukeAI Try to use these hyperparameters: #4430

@nyj-ocean

It isn't tested well, so use scale_x_y = 1.0 that is by default.

Or try scale_x_y = 1.05, 1.1, 1.2 for different yolo-layers ?


    1.05 * logistic - 0.025 - for yolo layer (large objects) scale_x_y = 1.05 in cfg-file
    1.1 * logistic - 0.05 - for yolo layer (medium objects) scale_x_y = 1.1 in cfg-file
    1.2 * logistic - 0.1 - for yolo layer (small objects) scale_x_y = 1.2 in cfg-file

AlexeyAB · 2019-12-02T12:19:58Z

@dselivanov

Honestly guys I don't understand how giou, diou, ciou is better than just IOU for YOLOv3 framework. The main problem they all address compared to vanilla iou is that loss is not differential if intersection of an anchor and a target is 0. However in the yolo framework it is always non-zero - we pick the anchor with largest IOU with target. Otherwise box loss doesn't contribute to overall loss.

I also have doubts about the effectiveness of C/D/GIoU. Perhaps they simply generate a higher delta and therefore improve the results of AP75 and reduce the results of AP50. Maybe we can achieve the same effect with default Yolo-loss (without GIoU) and with iou_normalizer=10

But:

default Yolo-loses doesn't use IoU. So may be IoU-loss (and C/D/GIoU) is better than default Yolo-loss
GIoU increases AP75 (but decreases AP50) https://github.com/WongKinYiu/CrossStagePartialNetworks#gpu-real-time-models

So I'm sceptical about these papers and reported results. At least in the experiments using my own pytorch yolo implementation with custom datasets I'm getting worse results with *IOU losses compared to original yolo loss.

This Pytorch-Yolov3 uses GIoU: https://github.com/ultralytics/yolov3

dselivanov · 2019-12-04T08:40:09Z

default Yolo-loses doesn't use IoU. So may be IoU-loss (and C/D/GIoU) is better than default Yolo-loss

yeah, of course I'm aware of that. Here is example of what i'm getting (without NMS):

DIoU loss

YOLO loss

AlexeyAB · 2019-12-04T13:26:58Z

@dselivanov

I don't know how good DIoU works with rotated bboxes.
How many anchors do you use?
What Loss do you use for angle?

dselivanov · 2019-12-04T15:24:39Z

9 anchors. Angle loss is just another separate loss component, so total loss = box loss + angle loss + class loss + confidence loss. I predict rotation between 0 and pi. The issue here not with rotation, but with width and height...

dselivanov · 2019-12-13T03:30:53Z

@glenn-jocher yes, I think it will be on par with other losses.

Are you looking at the individual loss component magnitudes

Yes

nyj-ocean · 2019-12-13T10:22:01Z

@AlexeyAB
IoU-aware

IoU-aware.pdf

AlexeyAB · 2019-12-13T17:29:09Z

@nyj-ocean The basic idea is the same as in [Gaussian_yolo] layer - to use confidence score of bounded boxes: #4147

It it better than [Gaussian_yolo] layer?

dselivanov · 2019-12-14T06:57:19Z

@AlexeyAB

Are you sure that your implementation of IoU and DIoU is correct?

Indeed loss was correct, but I had another bug - I scaled anchors 2 times... Now re-running experiments. Will keep posted.

dselivanov · 2019-12-14T12:53:56Z

@AlexeyAB @glenn-jocher. I've fixed bug in my code and now it seems:

optimizing IoU directly clearly gives much better results compared to YOLO proxy loss
DIoU loss is no better than IoU

class	ap_iou	ap_diou	ap_yolo
large-vehicle	0.470619	0.475856	0.4124
plane	0.703620	0.707864	0.6826
ship	0.629271	0.628140	0.5553
small-vehicle	0.268684	0.270890	0.2120
storage-tank	0.463676	0.467555	0.4130

AlexeyAB · 2019-12-14T13:10:03Z

@dselivanov Nice!
What about CIoU?
What activation do you use for angle-Loss?

dselivanov · 2019-12-14T13:29:32Z

Nice!

Btw this is ap@iou>=0.5

What about CIoU?

I haven't tried and I think will not. I believe my comment here is valid.

What activation do you use for angle-Loss?

Do you mean how loss is calculated? I predict angle between X axis and the largest side (width by convention).

AlexeyAB · 2019-12-14T13:40:34Z

@dselivanov

Do you mean how loss is calculated? I predict angle between X axis and the largest side (width by convention).

Thanks! Do you use Linear activation for Angle, or Logistic as for x,y,obj,class, or EXP as for w,h?

dselivanov · 2019-12-14T13:44:40Z

Thanks! Do you use Linear activation for Angle, or Logistic as for x,y,obj,class, or EXP as for w,h?

Logistic because angle is bounded by 0 and ~~pi/2~~ pi. So angle = pi * sigmoid(yolo_layer_output)

AlexeyAB · 2019-12-14T14:02:33Z

@dselivanov
Have you tried using trigonometric activation functions for an angle?
Like:

cos, sin - min=-1 max=1
tanh or th (hyperbolic tangent) - min=-1 max=1 - is implemented: activation=tanh
sech (hyperbolic secant) - min=0 max=1

More: https://en.wikipedia.org/wiki/Hyperbolic_function

nyj-ocean · 2019-12-15T01:52:27Z

@AlexeyAB

It it better than [Gaussian_yolo] layer?

I am not sure

abhigoku10 · 2020-01-10T06:30:59Z

@AlexeyAB @glenn-jocher @dselivanov thanks for the detailed discussion which you are currently have i had few queries
1.be it IOU ,DIOU/CIOU does it take into consideration the rotation of the object i,e the orientation of the object
2. @dselivanov is proving that std IOU is the best for his application , can we generalise it to all the other domain applications also
3.can this IOU/DIOU/CIOU features be applied to other similar architectures of yolo like YOLACT, Complex yolo and complexer yolo

Thanks in advance

AlexeyAB · 2020-01-10T10:47:02Z

@abhigoku10

CIoU has the best AP0.5...0.95 and AP75: https://github.com/WongKinYiu/CrossStagePartialNetworks#some-tricks-for-improving-ap
@dselivanov is proving that DIoU is slightly better than IoU
Yes.

abhigoku10 · 2020-01-10T11:49:29Z

@AlexeyAB thansk for your response but does these method take rotation/orientation of the object into considerations

AlexeyAB · 2020-01-10T11:59:14Z

No.
@dselivanov implemented rotation/orientation independently by himself in Pytorch.

abhigoku10 · 2020-01-11T02:15:36Z

@dselivanov can you pls share the rotation section of the CIOU/DIOU ?

dselivanov · 2020-01-11T05:42:58Z

There is no "rotation part" of *IOU. Rotation loss is a separate component - see our discussion above.

…

On Sat, 11 Jan 2020, 06:15 abhigoku10, ***@***.***> wrote: @dselivanov <https://github.com/dselivanov> can you pls share the rotation section of the CIOU/DIOU ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4360?email_source=notifications&email_token=ABHC5XN4VTG5NIAZ2GMT6WDQ5ETUTA5CNFSM4JQTY7Q2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIVWRDI#issuecomment-573270157>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHC5XJNG477YYWUPI23RULQ5ETUTANCNFSM4JQTY7QQ> .

abhigoku10 · 2020-01-11T05:45:11Z

@dselivanov thanks for confirming this , just wanted to knw of by including the rotation/ orientation param in the loss calculation is effective or not ?

dselivanov · 2020-01-11T09:41:52Z

@abhigoku10 yes

spaul13 · 2020-04-07T04:54:12Z

@simon-rob @AlexeyAB @LukeAI @glenn-jocher can anyone please tell me how can I change the loss functions for training a classification model while using this repo.

WongKinYiu · 2020-05-11T13:44:54Z

@AlexeyAB

https://github.com/Zzh-tju/CIoU
https://arxiv.org/pdf/2005.03572.pdf
CIoU team update their Cluster-NMS. +0.6 AP.

AlexeyAB added the want enhancement Want to improve accuracy, speed or functionality label Nov 22, 2019

AlexeyAB mentioned this issue Nov 23, 2019

Implement Yolo-LSTM (~+4-9 AP) for detection on Video with high mAP and without blinking issues #3114

Open

nyj-ocean mentioned this issue Nov 24, 2019

How to calculate mAP for (Gaussian+GIOU+YOLOv3) in this repository? #4357

Open

WongKinYiu mentioned this issue Nov 25, 2019

MixNet (Mix_Conv) - 0.360 (0.5) BFlops - 77.0% (71.5%) Top1 #4203

Closed

AlexeyAB added enhancement and removed want enhancement Want to improve accuracy, speed or functionality labels Dec 2, 2019

AlexeyAB mentioned this issue Dec 2, 2019

DIoU YOLOv3 #4429

Open

glenn-jocher mentioned this issue Feb 5, 2020

add diou_nms ultralytics/yolov3#795

Closed

AlexeyAB mentioned this issue Feb 10, 2020

Training for rotated bounding boxes #4740

Open

AlexeyAB changed the title ~~DIoU loss implementation~~ DIoU and CIOU loss implementation Mar 1, 2020

AlexeyAB mentioned this issue Mar 1, 2020

what's that, GREEDY_NMS #4943

Open

brian208579 mentioned this issue Apr 12, 2020

The ciou is better than giou ? #5218

Closed

dselivanov mentioned this issue Oct 20, 2020

Rotated Bounding Box output #2148

Open

cenit closed this as completed Jan 23, 2021

DIoU and CIOU loss implementation #4360

DIoU and CIOU loss implementation #4360

Comments

simon-rob commented Nov 22, 2019

AlexeyAB commented Nov 22, 2019

LukeAI commented Nov 23, 2019

AlexeyAB commented Nov 23, 2019

tuteming commented Nov 23, 2019

AlexeyAB commented Nov 23, 2019

glenn-jocher commented Nov 24, 2019

lq0104 commented Nov 25, 2019

AlexeyAB commented Nov 25, 2019

glenn-jocher commented Nov 25, 2019

AlexeyAB commented Nov 25, 2019

glenn-jocher commented Nov 25, 2019 • edited Loading

AlexeyAB commented Nov 25, 2019

LukeAI commented Nov 26, 2019

nyj-ocean commented Dec 2, 2019

dselivanov commented Dec 2, 2019 • edited Loading

AlexeyAB commented Dec 2, 2019

AlexeyAB commented Dec 2, 2019 • edited Loading

dselivanov commented Dec 4, 2019

DIoU loss

YOLO loss

AlexeyAB commented Dec 4, 2019

dselivanov commented Dec 4, 2019

dselivanov commented Dec 13, 2019

nyj-ocean commented Dec 13, 2019

AlexeyAB commented Dec 13, 2019

dselivanov commented Dec 14, 2019

dselivanov commented Dec 14, 2019

AlexeyAB commented Dec 14, 2019

dselivanov commented Dec 14, 2019 • edited Loading

AlexeyAB commented Dec 14, 2019

dselivanov commented Dec 14, 2019 • edited Loading

AlexeyAB commented Dec 14, 2019

nyj-ocean commented Dec 15, 2019

abhigoku10 commented Jan 10, 2020

AlexeyAB commented Jan 10, 2020

abhigoku10 commented Jan 10, 2020

AlexeyAB commented Jan 10, 2020

abhigoku10 commented Jan 11, 2020

dselivanov commented Jan 11, 2020 via email

abhigoku10 commented Jan 11, 2020

dselivanov commented Jan 11, 2020

spaul13 commented Apr 7, 2020

WongKinYiu commented May 11, 2020

glenn-jocher commented Nov 25, 2019 •

edited

Loading

dselivanov commented Dec 2, 2019 •

edited

Loading

AlexeyAB commented Dec 2, 2019 •

edited

Loading

dselivanov commented Dec 14, 2019 •

edited

Loading

dselivanov commented Dec 14, 2019 •

edited

Loading