Possible bug in AP calculation

[tkanmae]

Thanks for making your work available.

This is a report for a possible bug in computation of AP.

Let say that I have ranks of [0, 3, 4, 5]. If I pass it to compute_ap() with nres = 4, and inspected precision_0 and precision_1 in the function, their values in the for-loop are

rank 0: (precision_0, precision_1) = (1.000, 1.000)
rank 3: (precision_0, precision_1) = (0.333, 0.500)
rank 4: (precision_0, precision_1) = (0.500, 0.600)
rank 5: (precision_0, precision_1) = (0.600, 0.667)

However, the values at rank 3 (zero-based) is odd to me, because the precision of ranks at each k is 1.0, 0.5 (2/4), 0.6 (3/5) and 0.667 (4/6). Therefore, the value of precision_0 and precision_1 must be 1.0 and 0.5 respectively at rank 3.

I could be wrong, but it seems to me that there is a bug in conpute_ap().

[filipradenovic]

Sorry for a long reply time, I somehow missed this issue.

precision_0 is the precision at the previous rank:

• for rank 0 previous rank is imaginary (or it can be seen as having query as the first retrieved image) so precision_0 = 1.
• for rank 3 previous rank is rank 2 and the precision there is precision_0 = 1/3 (only one positive out of 3 retrieved images) hence the 0.333.

Hope this resolves it.

[filipradenovic]

I have made a quick draw of this example. I hope you appreciate this old style Precision-Recall (PR) curve drawing:

The colored trapezoid areas are computed at each of the ranks from your example (rank 0,3,4,5), and their sum is the final area under the PR curve or Average Precision (AP) for this query.

Precision-Recall (PR) curve drawing in PDF