Exposing to the Right - Revisited

[Inkanyezi]

Blinkies too are based on the jpeg, but as they appear only when all three channels clip, they are rather close to what's in the RAW. There is however a bit of overhead, and some conversion software can extrapolate a tad more when just one of the channels clips.

[Colin Southern]

Hi Colin,

When discussing the dynamic range of output media (e.g. around 4 for print, 6 for screen) it is important to consider the tonal gradation that is possible to show within that dynamic range.

Screens can output at 256 light levels (8 bit) or 1024 (10 bit) if you pay a lot of money. This is the upper limit for levels unless in between light levels are produced by flicking between brightness for each pixel. However the upper brightness and lower brightness set the dynamic range. Due to the inability to have a true black and the limits of display output this will be more like 6 stops.

Prints can output dozens of blobs of ink per pixel of different grey inks. This will allow a much larger number of tones to be displayed that the 256 levels of a monitor. So although the dynamic range is limited by Dmax (maximum black) and paper white, the number of tones that can be represented is very high.

In either case I feel it is valuable to understand that the 4 stops of print or 6 stops of screen is not a reflection of how good the output will look. If you have a 10 stop dynamic range scene captured in camera then you are not limited to only showing 4 stops of it in print. The ability to do tonal compression from a higher dynamic range capture allows many more stops of dynamic range to be represented successfully in a print or on screen.

I agree that poor tonal compression will lose the contrast in the scene where it is expected. For example you would want to print a white dress for the bride and black suit for the groom. If compressed too far then both these tones will tend towards grey and the resulting muddy midtone look. But with careful processing you can print much more than 4 stops of range because you have the advantage that the viewer will expect a certain appearance and the mind will compensate, e.g. it will accept a slightly less bright white bridal dress and less black groom suit if they fit within the entire image.

Alex

Hi Alex,

I think we're pretty much in agreement. Although a print may physically be capable of representing more than 256 levels, it becomes somewhat of a moot point because our eyes can't detect more than about 200 levels over that range.

I appreciate what you're saying about DR compression - but - it still becomes a "theory" -v- "real-world" thing. In "real-world" photography it's the highlights and midtones that we mostly appreciate detail from (and these are easily captured in a normal exposure) - and then we may (or may not) want to apply a degree of DR compression to reveal shadow detail - but - there are limits to this and the majority of the time what determines that limit is more the fact that the image becomes unappealing due to lack of local contrast than due to revealed noise. In essence what ETTR allows you to do is get "more bang for your bucks" before having to use more traditional HDR techniques - but regardless, it's really only applicable to those wanting a "HDR Result" - and that's a different kettle of fish to most photography. I don't have any particular beef with ETTR in this regard, but those "pushing the cause" always seem to make out that it's a technique that should be applied all the time whereas in reality it's only beneficial in a very specific (and relatively uncommon) situation.

Ten years ago or so when the ETTR article first appeared it was perhaps a different story, but really a different story. If it were so beneficial I suspect camera makers would have introduced an "ETTR metering mode" a long time ago.

[Colin Southern]

@Colin: I'm not sure I understand your calculation about the 6 stop safety margin though:
you say: '12-4-2 puts me 6 stops about the noise floor' while NOT using ETTR.
But that would imply NO margin at the highlight side, which is what ETTR is all about... Confused...

Start with 12 stops - subtract 2 for the normal safety margin in a RAW exposure when photographing a reflective scene (ie gray point in the middle - 2 stops below the highlights which are in turn another 2 stops below the sensor saturation point) - so the highlights start at "2 stops down" and the shadows finish another 4 stops down - so the shadows are a total of 6 stops "down from the top" which is still 6 stops "up from the bottom".

In this situation ETTR removes the 2 stop safety margin and takes you 8 stops above the noise floor. So the ETTR "advantage" becomes the difference in noise between 6 stops above the noise floor and 8 stops above the noise floor which is approximately NOTHING. So no gain in terms of noise, but significant risk of some highlights being in the non-linear region (less than 1/3 stop @ base ISO safety margin, typically).

So just not worth the risk in my opinion.

Make sense?

[Colin Southern]

Blinkies too are based on the jpeg, but as they appear only when all three channels clip, they are rather close to what's in the RAW. There is however a bit of overhead, and some conversion software can extrapolate a tad more when just one of the channels clips.

That's where RGB histograms can give a better indication of the overall exposure situation.

The short answer to "what are the blinkies based on" is really "who cares" ... it's not an exact science and they're still only a tool to be used or misused. Things like specular reflections are ALWAYS going to muck up "correct" exposures. Although they're based on all channels clipping, this is mitigated to a significant degree by normal exposure metering safety margin (including the RAW advantage).

I think what many folks need to do is get a better understanding of just what constitutes a "correct" exposure (<- can of worm opening alert!!!)

[Inkanyezi]

Ten years ago or so when the ETTR article first appeared it was perhaps a different story, but really a different story. If it were so beneficial I suspect camera makers would have introduced an "ETTR metering mode" a long time ago.

And maybe they did that?

The Nikon D-Lighting was a function to enhance shadows, by raising contrast in the darkest parts of the image, lifting the tone curve in a way that decreased contrast in the brighter parts of the image. Then somewhat later they introduced Active D-Lighting, that also held highlights back, effectively doing the ETTR routine. To help this process, there is a small camera mounted inside the viewfinder, that analyses the image in various fields, few in the cheaper instep models, more numerous in the more expensive ones. This system is said to hold highlights so that they don't wash out. Canon introduced their own system somewhat later, called Highlight Tone Priority, said to do the same, but on the EOS 40D it seems to be moot only, not accomplishing anything, only setting the numbers 200 to 2oo in the display. Rumour has it that something later really has been implemented, but AFAIK Canon does not have the hardware necessary to make a highlight measurement. Also other manufacturers have their own design of some similar system, but so far only cameras with either the Nikon system with a camera in the viewfinder or those that use the image sensor for measurement have the hardware to do a highlight reading. I tried the "intelligent" exposure setting on my Panasonic G1 only to find it counter-productive. It didn't do what the marketing blurb said.

So as it is, there is at least one brand that has implemented such a function, and it does work. However, as just pointed out, results are not always pleasing. It is something to use very cautiously, and it does not suit all scenes to be photographed out there. The ETTR raindance however is more appealing when light is scarce and one needs short exposure times for hand-held shots, when the dynamic range of the sensor gets more narrow. When the actual dynamic range of the sensor goes below eight steps, it makes sense to hold exposure to the right but not blow highlights too much. This is still the case with many cameras when you get into the region of ISO 1600 or more. It is akin to when we "pressed" Tri-X film in the sixties. We could use a higher exposure index, at the price of less tonality, higher contrast.

So, Colin is very much right. For normal shooting, ETTR is mostly not needed, and it might even become counter-productive, but sometimes, it is the way to go to get reasonably well exposed images. When your dynamic range shrinks because you need high ISO, it becomes more important to expose just right, not too little, but also not too much.

[Colin Southern]

And maybe they did that?

Can't speak for Nikon, but I'm not sure why any model couldn't do a reasonable job of highlight (area) protection just using their normal zone metering.

Canon have a couple of technologies - the older HTP (as far as I know) just under-exposed the entire shot then applied a curve to compress the highlights by a stop (hence the reason the base ISO "increased" to 200) (because it's really still using 100) and shadow noise increased (because the under-exposure is being compensated for my amplifying the shadow and mid-tones), and the newer Auto Lighting Optimiser (which I assume is doing a similar thing to Nikon's equivalent) (never use it).

[Colin Southern]

When your dynamic range shrinks because you need high ISO, it becomes more important to expose just right, not too little, but also not too much.

ETTR becomes mandatory at high ISO settings because one simply cannot afford to give away 2 stops of DR; it pushes one too close to the noise floor.

[Inkanyezi]

Can't speak for Nikon, but I'm not sure why any model couldn't do a reasonable job of highlight (area) protection just using their normal zone metering.

If we delve into this, we'll go far beyond the scope of the thread and it won't help anyone a lot, but I can assure you that your comment here made me smile. Some day they might implement a "zone metering" or "evaluative metering" system for DSLR:s, and apparently, Nikon already did.

[Bobobird]

Canon have a couple of technologies - the older HTP (as far as I know) just under-exposed the entire shot then applied a curve to compress the highlights by a stop (hence the reason the base ISO "increased" to 200) (because it's really still using 100) and shadow noise increased (because the under-exposure is being compensated for my amplifying the shadow and mid-tones), and the newer Auto Lighting Optimiser (which I assume is doing a similar thing to Nikon's equivalent) (never use it).

On many sites that I have checked for info about setting up the 5D3, the majority of the people actually recommend putting these settings to on.

[Colin Southern]

On many sites that I have checked for info about setting up the 5D3, the majority of the people actually recommend putting these settings to on.

For JPEG shooters it'll make a difference, but not for RAW shooters.

[Bobobird]

I read somewhere (here ?) that it does make some modification to RAW data but that is only seen in DPP but show no effect in ACR.

Will need to recheck this in the snow which tends to blow the highlights quite easily.

[Colin Southern]

I read somewhere (here ?) that it does make some modification to RAW data but that is only seen in DPP but show no effect in ACR.

Will need to recheck this in the snow which tends to blow the highlights quite easily.

HTP does by virtue of the fact that as far as we can tell it's using a different ISO to the "one advertised" (although comparitive RAW tests have shown that in practice there's really no difference), but all the others simply pass metatags which DPP honours, but all the others don't. Much like picture styles.

Snow will always give you a bazillion specular highlights ... there's little point in trying to protect them because you'll have to under-expose the shot severely (they're like little mirrors shining the sun directly onto the sensor).

[Bobobird]

No the tiny mirrors are normal and don't worry me. It is the large ones that look like big suns in daylight that present the most problems. Exposing for white of course leads to some of these blowouts.

[MilT0s]

Not really...

Here's an interesting PDF describing several demosaicing methods...
http://www.unc.edu/~rjean/demosaicing/demosaicing.pdf

Raw Therapee's AMAZE algorithm and the latest Adobe Camera Raw algorithms are even more complex.

I haven't found a description of the entire transformation process. You have to download the source code for DCRAW and review it. You can get it here...
http://www.cybercom.net/~dcoffin/dcraw/

You have to understand the C programming language...even then, a good debugger with a robust "watch" function is very helpful in following the process.

Very helpful, thank you.

[Colin Southern]

No the tiny mirrors are normal and don't worry me. It is the large ones that look like big suns in daylight that present the most problems. Exposing for white of course leads to some of these blowouts.

Well if it's not specular nor incident light then if it's not exposed correctly then it's just a plain old basic exposure problem. Why not just let the camera expose normally and take the approx 1.75 stop under-exposure - and fix in PP? If it's white enough to fool the metering then it's probably far enough away from the noise floor to not be a problem anyway -- and you'll get around a 4 stop safety margin

[Bobobird]

Thanks Colin, yes that is the good thing about snow - no noise.

[Graystar]

Can you post any links about that if handy?

You know, I totally forgot about DNG. Adobe does document the entire process, and you can read through in the DNG spec, which you can find here...

http://wwwimages.adobe.com/www.adobe...ec_1.4.0.0.pdf

Although it's 101 pages, it's not a how-to. They'll say, "this step should be performed in 'this way'"...but if you don't know what the "this way" means, then that's another trip to Google! Implementing DNG is no trivial task, and Adobe provides an SDK found here...

http://www.adobe.com/support/downloads/dng/dng_sdk.html

(which, btw, requires the XMP SDK found here...
http://www.adobe.com/devnet/xmp/

...as well as the open source EXPAT XML Parser, found here...
http://sourceforge.net/projects/expat/ )

[xpatUSA]

I haven't found a description of the entire transformation process. You have to download the source code for DCRAW and review it. You can get it here...
http://www.cybercom.net/~dcoffin/dcraw/

You have to understand the C programming language...even then, a good debugger with a robust "watch" function is very helpful in following the process.

I have used DCRAW quite a lot and, without needing to know "C" or use a debugger, you can select from various options on how to handle blown highlights:

-H 0 Clip all highlights to solid white (default).
-H 1 Leave highlights unclipped in various shades of pink.
-H 2 Blend clipped and unclipped values together for a gradual fade to white.
-H 3 (& onward) Reconstruct highlights. Low numbers favor whites; high numbers favor colors. Try -H 5 as a compromise. If that's not good enough, do -H 9, cut out the non-white highlights, and paste them into an image generated with -H 3.

[Graystar]

I have used DCRAW quite a lot and, without needing to know "C" or use a debugger...

Right...you definitely don't need to know about programming to use DCRaw. I believe the original inquiry was in reference to my view that concerns over "RAW levels" were unfounded, and this was due to the complexity of the demosaicing process. And if you want to actually watch RAW data get demosaiced...well...there's only one way to do that. So that was the intent of my post.

[revi]

I needed some time to digest all this, but I'm not so sure that we can ignore the 'RAW levels' that easily. I agree that there's a lot going on in the demosaicing step (although not all algorithms are very complicated), but the problem of clipping doesn't usually occur on isolated pixels. In most cases there's a region of several adjacent pixels that gets overloaded, and that can not be corrected by interpolation within a colour channel.

And wrt. DNG, that's a raw format container, that does not contain demosaiced data... And as far as I can see, the document cited doesn't give any instructions concerning demosaicing (there's a lot about colour spaces and transformations, but that's a different kettle of fish).