DNG GainMap support #10289

paolodepetrillo · 2021-10-29T03:02:15Z

Implements #8728. Danger - this changes the parameters of the rawprepare module and will break your edits! This seems to work but definitely needs careful review from someone who knows more than I do about the internals of darktable.

The GainMap is documented in the Opcode List Processing section of the Adobe DNG Spec. This feature is to support DNGs that are produced as the native raw format of devices like smartphones and GoPros, not for DNGs produced by Adobe DNG Converter.

Usage
Since the GainMaps are applied to the entire image after black point subtraction but before demosaicing, the raw black / white point (rawprepare) module is the appropriate place to apply the GainMap. A new option "flat field correction" has been added to rawprepare, which is only visible for images which contain a GainMap which is of a form that we can apply. The option is enabled by default for supported images, since it is mandatory to apply and the colors can be completely wrong near the edges of the image if it is needed but not applied.

For any images containing a GainMap that were already in the library before upgrading to a version with this feature, the flat field correction option will not appear. To fix this, use the refresh exif button in selected images - metadata in the lighttable. The reason is that the image in the database would not have the flag set that indicates the image contains additional exif tags.

Implementation
CPU and OpenCL are both supported. I don't know much about openmp / vectorization / etc, but the CPU version seems fast enough. The loop that applies the GainMap takes about 3x longer than the loop that does the black point subtraction and scaling, which is only 30 or 40ms on my system.

The definition of the GainMap in the DNG spec is very open ended - I did not try to support every possible valid GainMap, just the typical ones from phones etc: four GainMaps, one for each RGGB Bayer filter, of the same size, which cover the entire image. If the GainMap is valid but unsupported the option does not appear in rawprepare.

There was an existing feature to support the DefaultUserCrop exif tag, which sets a flag in the sql database indicating that the image has something in the exif tags which isn't itself stored in the database, but needs to be loaded from the image file into dt_image_t. I just extended this to also handle the OpcodeList2 exif tag at the same time. In the future this could also support getting the OpcodeList3 tag containing the WarpRectilinear opcode, for lens distortion correction.

For putting dynamically allocated memory inside dt_image_t, I followed the example of the profile field which is set by the colorin module and freed from dt_image_cache_deallocate.

Validation
I got a few samples to test from this thread. A couple, like the Samsungs, contained no-op gain maps (size 1x1) which are ignored.

To validate that the GainMaps were being applied correctly I used the Adobe DNG SDK implementation as a reference.
I had been using the dngpreprocess script to make phone photos usable in darktable prior to this feature. It takes the original dng and uses the Adobe DNG SDK to perform black point subtraction, rescale white point to 1.0, apply the GainMaps, and writes out a floating point DNG for further processing.
For test images I compared three cases exported from darktable with default settings: the preprocessed dng, the original dng with flat field correction on and OpenCL off, and with OpenCL on. All three implementations produce almost the same results, although not exactly down to the last bit.

The only other open source raw developer which currently supports the GainMap is ART - see gainmap.cc.

github-actions · 2021-12-29T00:10:27Z

This pull request did not get any activity in the past 60 days and will be closed in 365 days if no update occurs. Please verify it has no conflicts with the master branch and rebase if needed. Mention it now if you need help or give permission to other people to finish your work.

TurboGit · 2022-02-12T10:10:59Z

@paolodepetrillo : This works for me and the legacy_params are properly handled. So from a technical point of view it looks right.

@kmilos : Can you comment on this as you know far better than me on this part of the processing? TIA.

TurboGit

Looks good.

kmilos · 2022-02-12T12:35:43Z

Will take a look as well when I get a chance, on the road now...

TurboGit · 2022-03-17T10:57:26Z

@kmilos : Just in case you forgot about it.

kmilos

Nice addition in general, but I still have to check the formulas yet (wrt the relative coordinate scaling and the 0.5 pixel offset in particular).

~~Again, remind me why we're not trying to do this in rawspeed?~~ Edit: never mind, followed the discussion threads now...

kmilos · 2022-03-17T13:29:19Z

src/common/dng_opcode.c

+static double get_double(uint8_t *ptr)
+{
+  double v;
+  ((uint8_t *)&v)[0] = ptr[7];
+  ((uint8_t *)&v)[1] = ptr[6];
+  ((uint8_t *)&v)[2] = ptr[5];
+  ((uint8_t *)&v)[3] = ptr[4];
+  ((uint8_t *)&v)[4] = ptr[3];
+  ((uint8_t *)&v)[5] = ptr[2];
+  ((uint8_t *)&v)[6] = ptr[1];
+  ((uint8_t *)&v)[7] = ptr[0];
+  return v;
+}
+
+static float get_float(uint8_t *ptr)
+{
+  float v;
+  ((uint8_t *)&v)[0] = ptr[3];
+  ((uint8_t *)&v)[1] = ptr[2];
+  ((uint8_t *)&v)[2] = ptr[1];
+  ((uint8_t *)&v)[3] = ptr[0];
+  return v;
+}
+
+static uint32_t get_long(uint8_t *ptr)
+{
+  uint32_t v;
+  ((uint8_t *)&v)[0] = ptr[3];
+  ((uint8_t *)&v)[1] = ptr[2];
+  ((uint8_t *)&v)[2] = ptr[1];
+  ((uint8_t *)&v)[3] = ptr[0];
+  return v;
+}


These assume the host is always little endian, which might not be true...

For long and float, might as well use existing ntohl() instead of custom code, and implement something similar for double (something like long long version ntohll() for example).

Ha, GLib actually provides all of this already, see g_ntohl (alias of GUINT32_FROM_BE()) and GUINT64_FROM_BE()

I didn't try to support big endian, since the entire application only builds on little endian systems:

darktable/ConfigureChecks.cmake

Line 81 in 83a96df

test_big_endian(BIGENDIAN)

Nonetheless, I'd prefer use of existing functions here that have a known implementation via builtins, one less thing to worry about.

Just pushed a fix to use the glib macros.

Nice, thanks!

kmilos · 2022-03-17T13:29:45Z

src/common/dng_opcode.c

+
+    if(offset + 16 + param_size > buf_size)
+    {
+      fprintf(stderr, "[dng_opcode] Invalid opcode size in OpcodeList2\n");


Prefer using the logging mechanism

I assume DT_DEBUG_IMAGEIO is the correct logging channel for these?

kmilos · 2022-03-17T13:29:59Z

src/common/dng_opcode.c

+    }
+    else
+    {
+      fprintf(stderr, "[dng_opcode] OpcodeList2 has unsupported %s opcode %d\n",


kmilos · 2022-03-17T13:31:02Z

src/common/dng_opcode.h

+
+#pragma once
+
+#include <glib.h>


Only include what you really need

kmilos · 2022-03-17T13:40:23Z

src/common/dng_opcode.c

+#include <glib.h>
+#include <stdint.h>
+#include <stdio.h>
+
+#include "dng_opcode.h"
+#include "image.h"


Remove duplicate headers already included from dng_opcode.h

kmilos · 2022-03-17T13:46:11Z

src/common/exif.cc

+    g_free(data);
+    has_opcodes = TRUE;
+  }
+  return has_opcodes;


Might want to a add a debug message (at some reasonable verbosity level) here if none found: while these two locations should cover almost all known cases, there is no way to be 100% a priori sure where the raw IFD really is.

(Same applies for the user crop above btw)

kmilos · 2022-03-17T13:56:35Z

src/common/exif.cc

@@ -877,13 +900,19 @@ static bool _exif_decode_exif_data(dt_image_t *img, Exiv2::ExifData &exifData)

    if (dt_check_usercrop(exifData, img))
      {
-        img->flags |= DT_IMAGE_HAS_USERCROP;
+        img->flags |= DT_IMAGE_HAS_ADDITIONAL_EXIF_TAGS;


These really are DNG specific, and do not exist in the Exif spec, please rename the enum (EXIF->DNG).

paolodepetrillo · 2022-03-18T00:24:46Z

I had experimented with applying the GainMap in rawspeed a while ago: https://github.com/paolodepetrillo/rawspeed/tree/dng-gain-map
If you think that would be a better way to do it I can bring that branch up to date, make sure it's all working correctly, and submit a PR for rawspeed. That would require that the black level and GainMap would be applied unconditionally for all DNGs having a supported GainMap, there wouldn't be an easy way for the user to switch the correction on and off per image from within darktable although I don't know if there would ever be any good reason to.

kmilos · 2022-03-18T14:46:25Z

Re rawspeed, it does make sense we don't impose this ATM I guess.

OTOH, still not 100% clear on the coordinate and center offset normalization, i.e. whether 1.0=width or 1.0=width-1 (zero based pixel numbering of course) and the 0.5 pixel correction as consequence... I couldn't find the exact definition in the spec.

Finally, what about the ROI comment from @jenshannoschwalm ?

jenshannoschwalm · 2022-03-18T18:15:37Z

I can't test the code atm but from 1.6 specs linear interpolation should be done for centre of pixels (thus +.5) and width should be full (not -1).

At least that's how I understand it

paolodepetrillo · 2022-03-21T02:09:17Z

I can't test the code atm but from 1.6 specs linear interpolation should be done for centre of pixels (thus +.5) and width should be full (not -1).

At least that's how I understand it

The new DNG spec v1.6 gives the exact details about the linear interpolation for the new ProfileGainTableMap tag, but doesn't say anything more about what should be done with the old GainMap tag.

I had tested my implementation, the CPU and OpenCL versions, by comparing it to the Adobe DNG SDK to make sure the results were the same. I used a DNG from my phone, opened the original file in darktable and exported it with default settings using this PR to apply the GainMap. I then made a modified version of the DNG which had black level, white level, and GainMap applied by Adobe DNG SDK and the GainMap tag removed, and exported that from darktable. The resulting images aren't exactly the same down to the last digit but they are very close. I had used this method to find and fix a small error in the math while I was coding it.

Also, I just realized that I should probably either hide the flat field option entirely from rawprepare or remove the gainmap option from the dropdown for images which don't support it.

todd-prior · 2022-04-05T05:26:43Z

Great to see this moving along...thanks for all your work everyone....

paolodepetrillo · 2022-05-10T01:35:01Z

Is there still enough time before feature freeze to possibly get this into 4.0? I can bring it up to date so it works with the latest master, and hide the setting automatically for images that don't support it. Is there anything else that would need to be done?

kmilos

LGTM

TurboGit · 2022-05-10T16:26:38Z

@paolodepetrillo : Yes, it is still on time. I'll try to test it again soon.

TurboGit

Works for me, code looks ok, legacy params properly setup. Let's merge. Thanks!

todd-prior · 2022-05-11T14:57:21Z

Very grateful for this one...thanks everyone for making it happen Get Outlook for Android From: Pascal Obry ***@***.***> Sent: Wednesday, May 11, 2022 2:35:00 AM To: darktable-org/darktable ***@***.***> Cc: Prior,Todd ***@***.***>; Comment ***@***.***> Subject: Re: [darktable-org/darktable] DNG GainMap support (PR #10289) Merged #10289 into master. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

* Read dng OpcodeList2 exif tag * Load GainMaps into dt_image_t * Module parameter for flat field * Check if gain maps are supported * Apply gain map * Support OpenCL * Use glib macros for endian conversion * Only show flat field option if there is a GainMap

elstoc · 2022-05-28T17:55:16Z

If there are documentation requirements for this PR, please can someone write a short summary for me.

todd-prior · 2022-05-28T22:52:56Z

Hey Chris I am not really qualified to comment but they are nicely defined here ( https://m.dpreview.com/news/1055224750/unless-otherwise-specified-dng-gains-lens-corrections ) in the context of DNG files. The feature to recognize files supporting them is added to the raw black/white module. It is applied by default on import to new images. It can be toggled off in the module. Old edits won't apply it automatically to preserve them but it could be enabled. Images that don't support it don't show any dropdown in the module. The most obvious impact is an overall lightening of the image and removal of the typical strong vignette esp found on drone and smartphone raws.. I could send you an image to demonstrate on your end?? Get Outlook for Android From: Chris Elston ***@***.***> Sent: Saturday, May 28, 2022 1:55:27 PM To: darktable-org/darktable ***@***.***> Cc: Prior,Todd ***@***.***>; Comment ***@***.***> Subject: Re: [darktable-org/darktable] DNG GainMap support (PR #10289) If there are documentation requirements for this PR, please can someone write a short summary for me. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

todd-prior · 2022-05-28T22:53:36Z

https://m.dpreview.com/news/1055224750/unless-otherwise-specified-dng-gains-lens-corrections Get Outlook for Android From: Chris Elston ***@***.***> Sent: Saturday, May 28, 2022 1:55:27 PM To: darktable-org/darktable ***@***.***> Cc: Prior,Todd ***@***.***>; Comment ***@***.***> Subject: Re: [darktable-org/darktable] DNG GainMap support (PR #10289) If there are documentation requirements for this PR, please can someone write a short summary for me. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

elstoc · 2022-05-29T12:31:37Z

Thanks. I found a test image from that thread and pushed darktable-org/dtdocs@9c5efc2 to document it. Please let me know if it needs anything more.

todd-prior · 2022-05-29T16:18:55Z

Short and to the point. We could hyperlink the word Gainmap to a reference if that would be useful to guide people as to what it is but otherwise I think you clearly identify the option Get Outlook for Android From: Chris Elston ***@***.***> Sent: Sunday, May 29, 2022 8:31:48 AM To: darktable-org/darktable ***@***.***> Cc: Prior,Todd ***@***.***>; Comment ***@***.***> Subject: Re: [darktable-org/darktable] DNG GainMap support (PR #10289) Thanks. I found a test image from that thread and pushed ***@***.*** to document it. Please let me know if it needs anything more. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

electro-logic · 2023-10-28T19:58:26Z

Hello,

I am the author of Dng Opcodes Editor and I'm trying to implement the GainMaps opcode in a compliant way.

Usage Since the GainMaps are applied to the entire image after black point subtraction but before demosaicing, the raw black / white point (rawprepare) module is the appropriate place to apply the GainMap. A new option "flat field correction" has been added to rawprepare, which is only visible for images which contain a GainMap which is of a form that we can apply. The option is enabled by default for supported images, since it is mandatory to apply and the colors can be completely wrong near the edges of the image if it is needed but not applied.

From my understanding, when apply the GainMaps is defined by the OpcodeList index (See DNG Specs v1.7 pag 57). If the GainMaps is in the OpcodeList3 for example, then should be applied AFTER the demosaicing. DJI DNG files I tried always have the GainMap in OpcodeList3. I found nothing about the black point subtraction in the DNG Specs, where have you read about it?

The definition of the GainMap in the DNG spec is very open ended - I did not try to support every possible valid GainMap, just the typical ones from phones etc: four GainMaps, one for each RGGB Bayer filter, of the same size, which cover the entire image. If the GainMap is valid but unsupported the option does not appear in rawprepare.

DJI for example uses 3 channels (RGB) for processing after demosaicing.

To validate that the GainMaps were being applied correctly I used the Adobe DNG SDK implementation as a reference. I had been using the dngpreprocess script to make phone photos usable in darktable prior to this feature. It takes the original dng and uses the Adobe DNG SDK to perform black point subtraction, rescale white point to 1.0, apply the GainMaps, and writes out a floating point DNG for further processing. For test images I compared three cases exported from darktable with default settings: the preprocessed dng, the original dng with flat field correction on and OpenCL off, and with OpenCL on. All three implementations produce almost the same results, although not exactly down to the last bit.

Could you share your test pictures?

Thanks

electro-logic · 2023-10-28T20:01:47Z

OTOH, still not 100% clear on the coordinate and center offset normalization, i.e. whether 1.0=width or 1.0=width-1 (zero based pixel numbering of course) and the 0.5 pixel correction as consequence... I couldn't find the exact definition in the spec.

From my understanding if your image is 1000px then 1000=1.0 (if you count pixels from 1).
If the image is stored counting from 0 then 999=1.0

paolodepetrillo · 2023-11-01T02:01:16Z

@electro-logic DNG spec 1.7.0.0 defines OpcodeList2 as "Specifies the list of opcodes that should be applied to the raw image, just after it has been mapped to linear reference values." Black subtraction is described in the "Mapping Raw Values to Linear Reference Values" section on pg96.

Almost any .dng from a Pixel phone will have a GainMap in OpcodeList2 and is usable for testing - for example https://raw.pixls.us/getfile.php/4176/nice/Google%20-%20Pixel%204a%20-%2016bit%20(4:3).dng

electro-logic · 2023-11-01T11:57:01Z

Hello @paolodepetrillo thank you for the info. Is DarkTable handling images with GainMap in the OpcodeList3 too?

paolodepetrillo · 2023-11-01T12:31:50Z

@electro-logic No, it does not currently support GainMap in OpcodeList3, so those DJI DNGs don't get corrected. Maybe in the future it could be applied within the lens module, or it could be converted to a pre-demosaic OpcodeList2 GainMap that has approximately the same effect and the existing implementation could be used.

paolodepetrillo requested a review from LebedevRI as a code owner October 29, 2021 03:02

supertobi mentioned this pull request Nov 2, 2021

GoPro HERO5 GPR raw file support RawTherapee/RawTherapee#3569

Open

github-actions bot added the no-pr-activity label Dec 29, 2021

TurboGit self-requested a review February 12, 2022 10:01

TurboGit added this to the 4.0 milestone Feb 12, 2022

TurboGit approved these changes Feb 12, 2022

View reviewed changes

kmilos reviewed Mar 17, 2022

View reviewed changes

paolodepetrillo force-pushed the flatfield-dng-gain-map-2 branch from bdc3fe1 to 48145c1 Compare March 18, 2022 01:12

TurboGit removed this from the 4.0 milestone May 5, 2022

paolodepetrillo added 5 commits May 9, 2022 22:01

Read dng OpcodeList2 exif tag

d536479

Load GainMaps into dt_image_t

2eab3c1

Module parameter for flat field

a924dd3

Check if gain maps are supported

f11a32c

Apply gain map

980a2eb

Use glib macros for endian conversion

a9f5e8c

paolodepetrillo force-pushed the flatfield-dng-gain-map-2 branch from d579b81 to a9f5e8c Compare May 10, 2022 02:10

kmilos approved these changes May 10, 2022

View reviewed changes

TurboGit added this to the 4.0 milestone May 10, 2022

Only show flat field option if there is a GainMap

2bdd5ec

TurboGit approved these changes May 11, 2022

View reviewed changes

TurboGit merged commit 9552de2 into darktable-org:master May 11, 2022

TurboGit removed the release notes: pending label May 11, 2022

johnny-bit mentioned this pull request May 13, 2022

[FR] Lens shading map (DNG GainMap) support #8728

Closed

elstoc added documentation-complete needed documentation is merged in dtdocs and removed documentation-pending a documentation work is required labels May 29, 2022

paolodepetrillo mentioned this pull request Jul 5, 2022

GainMap and highlight reconstruction problem #12128

Closed

sgotti mentioned this pull request Oct 30, 2022

iop/lens: add embedded lens metadata correction #12714

Merged

sgotti mentioned this pull request Nov 7, 2022

Improved user experience when images require exif metadata refreshing. #12804

Closed

paolodepetrillo mentioned this pull request Feb 1, 2024

WIP / RFC - DNG opcode list processing hanatos/vkdt#110

Merged

ralfbrown mentioned this pull request Aug 30, 2024

Support HDR gain maps #17399

Open


		#pragma once

		#include <glib.h>

DNG GainMap support #10289

DNG GainMap support #10289

Conversation

paolodepetrillo commented Oct 29, 2021

github-actions bot commented Dec 29, 2021

TurboGit commented Feb 12, 2022

TurboGit left a comment

Choose a reason for hiding this comment

kmilos commented Feb 12, 2022

TurboGit commented Mar 17, 2022

kmilos left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paolodepetrillo commented Mar 18, 2022

kmilos commented Mar 18, 2022 • edited Loading

jenshannoschwalm commented Mar 18, 2022

paolodepetrillo commented Mar 21, 2022

todd-prior commented Apr 5, 2022

paolodepetrillo commented May 10, 2022

kmilos left a comment

Choose a reason for hiding this comment

TurboGit commented May 10, 2022

TurboGit left a comment

Choose a reason for hiding this comment

todd-prior commented May 11, 2022 via email

elstoc commented May 28, 2022

todd-prior commented May 28, 2022 via email • edited Loading

todd-prior commented May 28, 2022 via email

elstoc commented May 29, 2022

todd-prior commented May 29, 2022 via email

electro-logic commented Oct 28, 2023

electro-logic commented Oct 28, 2023

paolodepetrillo commented Nov 1, 2023

electro-logic commented Nov 1, 2023

paolodepetrillo commented Nov 1, 2023

kmilos left a comment •

edited

Loading

kmilos commented Mar 18, 2022 •

edited

Loading

todd-prior commented May 28, 2022 via email •

edited

Loading