Conformance testing

[benjamingeer]

Can we do something like this?

https://github.com/uclouvain/openjpeg/wiki/ConformanceTesting

[benjamingeer]

OpenJPEG lossless encoding is broken: uclouvain/openjpeg#892

[lrosenth]

Holy shit - others have problems too.

…

Sent from my iPad On 11 Feb 2017, at 23:34, Benjamin Geer <[email protected]<mailto:[email protected]>> wrote: OpenJPEG lossless encoding is broken: uclouvain/openjpeg#892<uclouvain/openjpeg#892> — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub<#117 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFN9zAPjcVKQwOhpGqAl5tY7JnRKzcAAks5rbjd-gaJpZM4L4JZS>.

[benjamingeer]

JPEG 2000 conformance testing at the US National Institute of Standards and Technology:

https://www.nist.gov/programs-projects/conformance-testing-jpeg-2000-codecs

ISO/IEC conformance testing procedures for JPEG 2000:

http://www.iso.org/iso/catalogue_detail.htm?csnumber=39079

International Telecommunications Union recommendation for JPEG 2000 conformance testing:

https://www.itu.int/rec/T-REC-T.803-200211-I/en

[benjamingeer]

ITU reference software: http://www.itu.int/rec/T-REC-T.804-201504-I

[boxerab]

Hello. OpenJPEG is fully compliant with ISO conformance tests for JPEG 2000. Unfortunately, it also has bugs :)

[benjamingeer]

Hello @boxerab and thanks for replying. If I understand correctly, OpenJPEG uses Kakadu for conformance testing. Since Sipi uses Kakadu internally, we'd like to do conformance testing without relying on Kakadu. I haven't yet received my copy of the ISO conformance testing document; does it involve using a reference implementation? And do you have any advice on how we should proceed?

[boxerab]

Hi Benjamin,

OpenJPEG test suite tests both conformance and non-regression. I am pretty sure that conformance does not have any dependency on Kakadu. Non-regression also can be done without Kakadu, but if Kakadu is available, then more tests are run using that option.

I am not familiar with the ISO docs, but I think the best approach might be to clone OpenJPEG, enable testing in CMake, and take a look at all of the conformance tests that get run.

[benjamingeer]

@boxerab The page https://github.com/uclouvain/openjpeg/wiki/ConformanceTesting says:

The encoder should be tested in the following way:

1. Encode a test image with a given set of parameters
2. Try to decode it with Kakadu.
3. Compare the original image to the decoded one and verify that the MSE is below the tolerated MSE defined for this test (typically, MSE = 0 for lossless tests, etc).

Is that correct? I also see Kakadu being used in this file:

https://github.com/uclouvain/openjpeg/blob/master/tests/conformance/CMakeLists.txt

Although that file is in a directory called conformance, the comments in the file refer to non-regression, so I'm not sure how to separate the two.

What do you use for measuring MSE?

[boxerab]

Yes, that page does mention Kakadu, but the Kakadu binaries are not used. I think it just refers to images that originated from Kakadu somehow, and are in the public domain, or the conformance suite.

If you don't see the text kdu_expand or kdu_compress, then Kakadu binaries are not being used.

As for MSE, OpenJPEG has a compare_images utility that looks at that.

[benjamingeer]

@boxerab OK thanks, that's very helpful.

[benjamingeer]

I've had a look at the ISO/IEC conformance testing procedures for JPEG 2000, ISO/IEC 15444-4:2004. Interestingly, the standard only gives quality requirements for decoders, not for encoders:

The only requirement for encoder compliance to produce compliant codestreams; any other requirements using quality criteria (such as PSNR or MSE) are not part of this Recommendation | International Standard.

Specifically:

An encoder is found to be compliant if the reference decoder can fully decode the image.

But this isn't good enough for us, because we need to show that we can both encode and decode without significant distortion. In particular, a crucial use case is converting TIFF images to JP2 and back again without significant distortion.

The standard suggests testing an encoder like this:

1. Encode the image using your encoder.
2. Decode it using one of the reference decoders (JasPer, OpenJPEG, or JJ2000).
3. Check that the decoded image is acceptable according to some criteria.

As for how you do (3), it says:

It is up to the implementor to establish requirements for accepting a reconstructed image. For example, a lossless encoder would be expected to produce a reconstructed image identical to the test image. Requirements for a lossy encoder might include obtaining peak error or MSE values in a specified range or similar evaluation.

For our use case with TIFF, it doesn't seem practical to use the reference decoders directly. JasPer doesn't support TIFF, OpenJPEG has a conformance issue at the moment, and JJ2000 seems to be a dead project.

The standard provides a set of reference images for testing decoders, and it says you can use them for testing encoders, too:

The reference decoded images provided for decoder compliance tests are acceptable but not required.

Here are the descriptions of the files, each of which is provided in TIFF and JPEG 2000 formats.

• file1.jp2: Three 8-bit components in the sRGB colourspace. The components are in the standard order and are transformed using the RCT. This file also includes XML metadata. If a reader fails on this file, the likely cause is a problem with parsing of the boxes.
• file2.jp2: Three 8-bit components in the sRGB-YCC colourspace. All components are at the full resolution, but are stored in reverse order in the codestream. The JP2 file contains a Channel Definition box that correctly associates each physical component with the correct colour in the sRGB-YCC definition. If a reader fails on this file, the likely causes are an incorrect interpretation of the Channel Definition box or an error in the conversion from sRGB-YCC to sRGB.
• file3.jp2: Three 8-bit components in the sRGB-YCC colourspace, with the Cb and Cr components being subsampled 2x in both the horizontal and vertical directions. The components are stored in the standard order. If a reader fails on this file, the likely causes are an error in either resampling or in conversion from sRGB-YCC to sRGB.
• file4.jp2: One 8-bit component in the sRGB-grey colourspace. If a reader fails on this file, the likely cause is an error in parsing of the boxes.
• file5.jp2: Three 8-bit components in the ROMM-RGB colourspace, encapsulated in a JP2 compatible JPX file. The components have been transformed using the RCT. The colourspace is specified using both a Restricted ICC profile and using the JPX-defined enumerated code for the ROMM-RGB colourspace. If a decoder fails on this file, the likely cause is an incorrect implementation of the Restricted ICC Three-Component transformation.
• file6.jp2: One 12-bit component in the sRGB-grey colourspace. If a reader fails on this file, the likely cause is an error in the extraction of 8 bits from the 12-bit codestream.
• file7.jp2: Three 16-bit components in the e-sRGB colourspace, encapsulated in a JP2 compatible JPX file. The components have been transformed using the RCT. The colourspace is specified using both a Restricted ICC profile and using the JPX-defined enumerated code for the e-sRGB colourspace. If a reader fails on this file, the likely causes are an error in conversion from the 16-bit data to 8-bit, or an error in implementation of Restricted ICC support.
• file8.jp2: One 8-bit component in a gamma 1.8 space. The colourspace is specified using a Restricted ICC profile. If a reader fails on this file, the likely cause is the implementation of the Monochrome Input profile as part of Restricted ICC.
• file9.jp2: One 8-bit component, which is used as input to a 256-entry palette that maps the single component to three 8-bit components. The depaletized components are in the sRGB colourspace. If a reader fails on this test, the likely cause is the application of the palette to the single component.

So I think we can do tests like this:

Check encoding

1. Have Sipi convert jp2_1.tif (a reference TIFF image) to JPEG 2000, to make sipi_file1.jp2.
2. Check Sipi's encoding by using GraphicsMagick (which uses JasPer) to compare file1.jp2 (the corresponding reference JPEG 2000 image) with sipi_file1.jp2, checking Peak Absolute Error (PAE) and Mean Absolute Error (MAE).

Check decoding

1. Have Sipi convert file1.jp2 to TIFF, to make sipi_jp2_1.tif.
2. Check Sipi's decoding by comparing jp2_1.tif with sipi_jp2_1.tif, checking PAE and MAE.

Check round trip

1. Have Sipi convert the already encoded sipi_file1.jp2 back to TIFF, to make sipi_sipi_jp2_1.tif.
2. Check Sipi's decoding by comparing jp2_1.tif with sipi_sipi_jp2_1.tif, checking PAE and MAE.

[benjamingeer]

Found some bugs when trying the above: #144 #145

It looks like GraphicsMagick isn't going to work for comparing JPEG 2000 images:

• gm compare can't read the reference image file3.jp2
• gm compare reports lots of distortion in most of the JP2 images converted by Sipi, but I'm a bit suspicious because visually they look OK.

I'll try OpenJPEG's compare_images utility.

[benjamingeer]

Just realised that OpenJPEG's compare_images utility (source code here) can't read JP2, only PGX, which is limited to greyscale. Hmm.

[benjamingeer]

ImageMagick with OpenJPEG seems OK, trying that.