Skip to content

Latest commit

 

History

History
83 lines (56 loc) · 4.29 KB

benchmark.md

File metadata and controls

83 lines (56 loc) · 4.29 KB

Benchmarks

Photo

Note: waifu2x's photo models was trained on the blending dataset of kou's photo collection and ukbench.

Note: PSNR in this benchmark uses a MATLAB's rgb2ycbcr compatible function (dynamic range [16 235], not [0 255]) for converting grayscale image. I think it's not correct PSNR. But many paper used this metric.

command: th tools/benchmark.lua -dir <dataset_dir> -model1_dir <model_dir> -method scale -filter Catrom -color y -range_bug 1 -tta <0|1> -force_cudnn 1

Datasets

BSD100: https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/ (100 test images in BSDS300) Urban100: https://github.com/jbhuang0604/SelfExSR

2x - PSNR

Dataset/Model Bicubic vgg_7/photo upconv_7/photo upconv_7l/photo resnet_14l/photo
BSD100 29.558 31.427 31.640 31.749 31.847
Urban100 26.852 30.057 30.477 30.759 31.016

2x - benchmark time (sec)

Dataset/Model vgg_7/photo upconv_7/photo upconv_7l/photo resnet_14l/photo
BSD100 4.057 2.509 4.947 6.86
Urban100 16.349 7.083 14.178 27.87

2x with TTA - PSNR

Note: TTA is an ensemble technique that is supported by waifu2x. TTA method is 8x slower than non TTA method but it improves PSNR (~+0.1 on photo, ~+0.4 on art).

Dataset/Model Bicubic vgg_7/photo upconv_7/photo upconv_7l/photo resnet_14l/photo
BSD100 29.558 31.474 31.705 31.812 31.915
Urban100 26.852 30.140 30.599 30.868 31.162

2x with TTA - benchmark time (sec)

Dataset/Model vgg_7/photo upconv_7/photo upconv_7l/photo resnet_14l/photo
BSD100 36.611 20.219 42.486 60.38
Urban100 132.416 65.125 129.916 255.20

Art

command: See appendix/benchmark.sh

Dataset

art_test: This dataset contains 84 various fan-arts. Sorry, This dataset is private.

2x - PSNR

Filter/Model Bicubic vgg_7/art upconv_7/art cunet/art
Lanczos 31.022 37.495 38.330 39.886
Sinc 30.947 37.722 38.538 40.312
Catrom(Bicubic) 30.663 37.278 37.189 40.184
Box 30.891 37.709 38.410 39.672

2x - benchmark time (sec)

Dataset/Model vgg_7/art upconv_7/art cunet/art
art_test 24.153 10.794 24.222

2x with TTA - PSNR

Filter/Model Bicubic vgg_7/art upconv_7/art cunet/art
Lanczos 31.022 37.777 38.677 40.289
Sinc 30.947 38.005 38.883 40.707
Catrom(Bicubic) 30.663 37.498 37.417 40.592
Box 30.891 38.032 38.768 40.032

2x with TTA - benchmark time (sec)

Dataset/Model vgg_7/art upconv_7/art cunet/art
art_test 207.217 99.151 211.520