dcraw - gamma tests - comparing linear RAW conversion with sRGB

[spittle]

Hi

I've been browsing these forums for a while. Wondering if someone can shed some light on what dcraw is doing to my CR2 files.

I have been experimenting with different dcraw conversions and am comparing the results in Nuke. I'm using Nuke as it works in linear space but displays all inputs in sRGB space, you just tell each input node what colour space the incoming image is.

Anyway I'm trying to verify that I'm able to acquire images with the following conversions:

1) a linear image
2) a linearized sRGB image
3) a gamma sRGB image

And I want to open in Nuke to compare, linearize/gamma/convert to ensure I have used the corerect settings and understand what I'm working with.

Conversions 1) and 2) look (in non technical terms) the same in terms of exposure/contrast (viewed in sRGB gamma space). There's a clear difference in vibrancy but I'm pretty happy that is because of using sRGB primaries and the other not. Although I'm wondering if it's more complex than I first imagined?

The thing I'm not sure about is with regards to my sRGB conversion(s), I would expect to be able to linearize these and they match my linear images but they don't, they are much brighter - and I say 'they' because I've tried a couple of different options.

Here are my dcraw settings: NOTE: All saved out as sRGB JPEGs in Nuke

1) Linear
dcraw.exe -v -w -H 0 -o 0 -q 3 -4 -T


2) Linearized sRGB
dcraw.exe -v -w -H 0 -o 1 -q 3 -4 -T

3) sRGB
dcraw.exe -v -w -H 0 -o 1 -q 3 -g 2.4 12.9 -6 -T


4) sRGB II
dcraw.exe -v -w -H 0 -o 1 -q 3 -6 -T


Can anyone suggest whether I'm going wrong somewhere? The files are pretty big, should I convert to Jpeg and upload?

Thanks

[spittle]

I also discovered that my '4) sRGB II' conversion is exactly the same as opening the CR2 in RawDigger and saving it as 'RGB render', with 'As Shot' white balance.

And If I save the file again from RawDigger, and disable 'automatic exposure correction for RGB render' then the results are similar to my linear conversions.

Also if I check 'Display RGB Render in RAW colours' then that is also similar to the linear renders. I'll post some more images tomorrow if needed.

[revi]

Are you sure your conversions do exactly what you think they are doing?
I'll just list the conversions you used in the order they appear above, then say where I think something might have gone differently than you expect (or perhaps not)

For those not familiar with dcraw, I'll explain all the options used:
-v : be verbose
-w : use camera whitebalance
-H 0 : clip highlights to solid white
-q 3 : use AHD for demosaicing
-T : write converted image as TIFF with metadata

dcraw.exe -v -w -H 0 -o 0 -q 3 -4 -T
this equivalent to dcraw.exe -v -w -H 0 -o 0 -q 3 -6 -W -g 1 1 -T
image will be in raw colour space (-o 0, i.e. camera-specific), with 16 bits/channel (-6), fixed white level (-W) and no gamma transformation (-g 1 1)

dcraw.exe -v -w -H 0 -o 1 -q 3 -4 -T
almost the same, except that the image will be in the sRGB colour space (-o 1). Using the sRGB color space does NOT by itself apply a gamma transformation, and the -4 option implies -g 1 1 (so we stay in linear space)

dcraw.exe -v -w -H 0 -o 1 -q 3 -g 2.4 12.9 -6 -T
Here's where things get interesting: now we specify a gamma transformation corresponding to the sRGB (-g 2.4 12.9). And the whitepoint will be determined based on the histogram.

dcraw.exe -v -w -H 0 -o 1 -q 3 -6 -T
This looks almost the same as the second command, 'just' with -4 replaced by -6. But '-4' means '-6 -W -g 1 1', in other words, we no longer use -W, so white level will be based on the histogram, and no more -g option, so the default gamma transformation will be used. And the default gamma transformation is -g 2.222 4.5 ('BT.709').

So actually, images 3 and 4 differ in two ways from the preceding one (white level and gamma). And specifically, the last image does not have -g 1 1 as you might have expected, but a gamma gamma curve with exponent of 2.22 and toe slope 4.5, closer to sRGB than to linear...
What I think you are looking for in the last image would be

dcraw.exe -v -w -H 0 -o 1 -q 3 -6 -g 1 1 -T -W
sRGB color space, 16 bit/channel output with no gamma transform and automatic white level

[spittle]

Hey revi

I'm pleased you've seen some flaws in my method. I'll take a closer look later in the week and compare the processed files in Nuke. I have a few more questions but will see if I can figure them out first. Thanks for the reply.

[xpatUSA]

And If I save the file again from RawDigger, and disable 'automatic exposure correction for RGB render' then the results are similar to my linear conversions.

A word of caution here, @spittle (do you have a name?).

I use RawDigger a lot for my Sigma X3F images. For the RGB export, the 'with/without gamma' checkbox is disabled. If it is also so for CR2s then the auto EC option output might look similar to your linear conversions but only because it is darker.

Must say that Remco's analysis re: DCraw was absolutely excellent!

[spittle]

I had some time to look into this again today and things are starting to make sense, and I can convert between linear/gamma and colour spaces perfectly (for now anyway!). So thanks again.

However, I do have some more questions:

1) Any ideas why RawDigger would load images (with RGB render enabled) that look like my 'sRGBII conversion'? i.e. the same as:

*summary*
dcraw.exe -v -w -H 0 -o 1 -q 3 -6 -T

no longer use -W,
so white level will be based on the histogram,
and no more -g option,
so the default gamma transformation will be used.
And the default gamma transformation is -g 2.222 4.5 ('BT.709').

Would it not make better sense to use an sRGB gamma curve? Maybe there is a reason I'm unaware of why dcraw defaults to BT.709 gamma?

Whilst there are other options in RawDigger - I couldn't see the logic for it even being able to produce the above conversion as a default option.

2) What exactly does it mean ---> -W Use a fixed white level, ignoring the image histogram ?

I can see that if I convert a RAW, with/without this option then it is possible to grade one to match the other perfectly. So it's some kind of scaling, and as far as I can tell it's perfectly linear.

Note: I'm aiming to produce perfectly linear HDRIs and a conclusion I'm drawing from this that it doesn't really matter whether I use this option or not as I intend to calibrate the intensity of the HDRI later.

Saying that - I also use HDRShop to process Raws. This has an option to scale per absolute value - so it will do the RAW conversion and convert to 32bit floating point, and it uses the exposure values in the metadata to convert to absolute intensity.

Does the -W option have any 'absolute'.... significance, for lack of a better word?

[xpatUSA]

I had some time to look into this again today and things are starting to make sense, and I can convert between linear/gamma and colour spaces perfectly (for now anyway!). So thanks again.

However, I do have some more questions:

2) What exactly does it mean ---> -W Use a fixed white level, ignoring the image histogram ?

Does the -W option have any 'absolute'.... significance, for lack of a better word?

When you type the command name without arguments in the command window, it lists all the options and, for -W, it says:

-W Don't automatically brighten the image

However, tutorials often describe it per your quote above.

So, we can perhaps deduce what happens. Without the -W option dcraw maps the range of levels to the intended output which we can say for now is 8-bit i.e. 0-255 decimal. Thus raw data is converted to RGB by dcraw magic and then the brightest channel (i.e. the largest V in HSV space) is scaled to 255 and all other channel levels are scaled accordingly. This effect can be seen in FastStone Viewer when review images are always nice and bright and also in RawDigger depending on preference settings.

By using -W, no such scaling takes place.

So, let's say that after conversion to some color space, or raw composite or even XYZ, the DCrawvalue for an image pixel says 200, 150, 100 (ignoring considerations of WB, gamma and black level settings). DCraw will set those output levels as 255/200*200 = 255, 255/200*150 = 191 and 255/200*100 = 128 i.e. 255, 191, 128.

With the -W option, DCraw will leave those output levels as 200, 150, 100.

I didn't understood what was meant by " 'absolute' .... significance. "

The presence of -W in the options string simply sets a flag which tells the code to branch around the auto-brightening function. I've seen it called a "switch" too, if that means anything to you.

Good luck with the project.

[dje]

When you type the command name without arguments in the command window, it lists all the options and, for -W, it says:

-W Don't automatically brighten the image

However, tutorials often describe it per your quote above.

So, we can perhaps deduce what happens. Without the -W option dcraw maps the range of levels to the intended output which we can say for now is 8-bit i.e. 0-255 decimal. Thus raw data is converted to RGB by dcraw magic and then the brightest channel (i.e. the largest V in HSV space) is scaled to 255 and all other channel levels are scaled accordingly. This effect can be seen in FastStone Viewer when review images are always nice and bright and also in RawDigger depending on preference settings.

By using -W, no such scaling takes place.

Good bit of detective work Ted. That sounds plausible to me. The only question then left for me is if the -W option is used, what value of white point is used ? I assume it's the Saturation level but I'm not sure. Whilst there won't be any auto brightness scaling, there must still be scaling of the raw data values to either an 8 bit or 16 bit range (where the top of each range is the white point).

Dave

[xpatUSA]

Good bit of detective work Ted. That sounds plausible to me. The only question then left for me is if the -W option is used, what value of white point is used ? I assume it's the Saturation level but I'm not sure. Whilst there won't be any auto brightness scaling, there must still be scaling of the raw data values to either an 8 bit or 16 bit range (where the top of each range is the white point).
Dave

As I understand it, your namesake Dave Coffin was not noted for writing fully-commented easily followed code. Some can follow it; I've never tried. I've got some links on my HD that are sometimes helpful and sometimes not. If you can find Guillermo Luijck's site, he has a tutorial there written in Spanglish.

I'll guess that the code works in floating point and that there is a scaling which normalizes the output for any camera to linear 1,1,1 floating point during conversion. If that is so, then there must indeed be a raw value that relates to that value of 1 and for many but not all cameras that value could be 4095 (12bit) or whatever it is for 14bit. Whether that means sensor saturated (well full) is another matter, I suspect. But, if I were a programmer (was once), that's what I would go for. Pure speculation on my part.

[dje]

Ted comments or no comments, you wont get me trying to read his code !!

I think that with some cameras, the saturation point is very close to the max value of a 12 bit or 14 bit number. However for others, particularly Canon, the saturation point is somewhat below the maximum value of a 12 or 14 bit number.

I have a link to Guillermo's site but he doesn't go into the fixed white point issue in any detail as far as I know.

Dave

PS : I just did a quick test using a severely under-exposed image and can confirm your theory of the auto brightness adjustment.

[spittle]

I didn't understood what was meant by " 'absolute' .... significance. "

Hi
Sorry that was pretty vague... I use a package called HDRShop - this will do a RAW conversion to 32bit float - and it has an option which scales values to 0-1 range.

With this option disabled - the software converts the image to an absolute EV (as far as I can tell). So for example, a sequence of bracketed images can be loaded, and their intensity may be out of the dynamic range of my monitor until I (non destructively) adjust the exposure in the viewport.

I wondered whether the -W option did something similar but you're right with your thinking.

After some thought - I think that the absolute scaling of RAWs is kind of insignificant to my workflow as there are too many variables that will skew the results.