HiFiC: request for rate-distortion data to reproduce Fig. 4, A10, A11

[Nikolai10]

Hello @fab-jul,

first of all - thank you very much for your great work!

May I kindly ask you to release the raw rate-distortion data to reproduce Fig. 4, A10, A11 (HiFiC, Baseline (no GAN), M&S Hyperprior, BPG)? This would be very helpful to enable comparisons.

Concerning the model evaluation, I have another question: How did you identify the optimal checkpoint? With GANs the final checkpoint is not necessarily the optimal one, I suppose. What is your experience here? It would be quite interesting if you could share some details about the checkpoint + evaluation policy used in your work.

Kind regards,

Nikolai

[fab-jul]

Hi Nikolai,

I can dig up all the raw numbers, how urgent is it? Are you submitting to NeurIPS?

re. eval: we actually just pick the last checkpoint. We took some care to make the training semi-stable, see Table 1 in the paper, where we explored reducing across-run variation. The main problem we had was that FID is not always a reliable predictor of what a good model is, so it's hard to find the optimal one I would say

Fabian

[fab-jul]

I'll get back to you after the NeurIPS deadline.

[hific]

Hi Nikolai, I managed to run the code that produced the plot. I will follow-up with a CSV by tomorrow.

[fab-jul]

Faster than expected, here is the raw data. I will check this into the repo soon

# clic2020
FID [0.12902283 0.24981422 0.37383949] [4.19736528 2.38322568 1.66647744]
KID [0.12902283 0.24981422 0.37383949] [0.00108268 0.00053555 0.00035608]
NIQE [0.12902283 0.24981422 0.37383949] [9.38350187 9.28048124 8.6921952 ]
LPIPS [0.12902283 0.24981422 0.37383949] [0.05394955 0.03485062 0.02499845]
MS-SSIM [0.12902283 0.24981422 0.37383949] [0.9553839  0.97377288 0.98172444]
PSNR [0.12902283 0.24981422 0.37383949] [30.86970338 32.9375868  34.59938206]

# div2k
FID [0.17268653 0.32447103 0.489503  ] [9.96830177 5.84229946 3.94786787]
KID [0.17268653 0.32447103 0.489503  ] [0.00120337 0.00053627 0.00024038]
NIQE [0.17268653 0.32447103 0.489503  ] [9.47402614 9.36838145 8.82207186]
LPIPS [0.17268653 0.32447103 0.489503  ] [0.06124068 0.0385301  0.02592616]
MS-SSIM [0.17268653 0.32447103 0.489503  ] [0.95155809 0.97143019 0.98089158]
PSNR [0.17268653 0.32447103 0.489503  ] [28.49394484 30.59894682 32.46940243]

# kodak
NIQE [0.1825587  0.35108609 0.53773159] [11.2748119  11.86824715 10.93795208]
LPIPS [0.1825587  0.35108609 0.53773159] [0.06864742 0.04278131 0.02784851]
MS-SSIM [0.1825587  0.35108609 0.53773159] [0.93721463 0.96430352 0.97751017]
PSNR [0.1825587  0.35108609 0.53773159] [27.56470037 29.65373484 31.7428445 ]

[Nikolai10]

Great, thank you very much!

[fab-jul]

Added a CSV in ccf33ea