Nikon raw conversion tone curves and exposure

[ajohnw]

I'm a bit confused about this area and wonder if some one can clear it up for me.

A web page that should be reliable states that Nikon raw files contain a tone curve. The base curve section here.

http://ufraw.sourceforge.net/Guide.html

Maybe they do or maybe the comment is out of date.

I have seen comments about exposure problems. The jpg tone curves in the Dpreview look over exposed to me but that might be a miss understanding on my part. The curve is shown on this page and mid grey corresponds to about 112 bits rather than 128.

http://www.dpreview.com/reviews/nikond7000/14

The following page is linked to mentioning a number of Nikon cameras including the D7000 as well as the ones it mentions - here the suggestion is that the cameras under expose.

http://fotogenetic.dearingfilm.com/c...ne_curves.html

The page may interest other people if they want to calibrate their camera in another way other than colour.

The way uk weather is at the moment there are few opportunities to make much use my D7000 and I would appreciate some idea on what to expect exposure wise for when I can use it in anger and take a lot of photo's. The 300mm shot I posted recently looks slightly over exposed to me which fits in with the dpreview.

The curve in raw file is even more confusing. The rawtherapee manual explains how to get hold of Nikon's curves from ViewNX2 or the full package. Reading about D light I get the impression that to make use of this from raw that NX2 is needed? If the curve is in the raw file why get it from NX2.

Help

John
-

[ajohnw]

A bit more web research has "illuminated" the exposure aspect. 3 video's but well worth the time to watch even for Canon users.

http://testcams.com/blog/2011/04/25/...00-overexpose/

I'm gob smacked - Nikon abandoning what I call beach photography - some rich bloke buys his SLR and shoots his holiday snaps with it and gets near perfect results when the film is developed at the chemists. Sorry should have said person not bloke.

On the other hand a few more examples would help as it could just be an attempt to say what problem when there is a bit of one. On the other hand exposing like that does make sense but will still need some experience as to what compensation to dial in. The AF point aspect in the video's is interesting too.

John
-

[ajohnw]

Yet more web info that suggests that tone curves can be added to the camera - even the D7000 but it all seems to relate to an earlier camera, the D2X.

http://imagesbyeduardo.com/main/tag/tone-curves/

No one seems to know anything about Nikon raw and embedded tone curves. Playing with Ufraw I found that using it's camera setting produced totally different results to custom but the same as linear. In custom mode small changes to the black and white points vertically NOT horizontally appears to bring in more of the bit depth in the raw file. Using colour matrix debayering ( same as Adobe's faithful I think - no shaped profile at all) I did these 2 and like the tones. The bright shot needed the changes to the embedded curve section and changes to the luminance curve.


-
-

-
-
John
-

[ajohnw]

Bit more camera colour profiles. Several places on the web point out that these can be obtained by extracting them from a ViewNX2 temporary directory while as nef files is being viewed in it. It creates them individually for each shot. Seems they are none standard but other software packages will still make use of them. I just did this with the car shot using the icm file ViewNX2 generated for it.

Colours are now more real but I would reckon the shot is about 1 stop over exposed. The one in the previous post wouldn't adjust sensibly to this brightness level - however that one wasn't clipped. Typical example of what can happen when different colour scaling is used on raw files. Other things can happen for the same reason eg the tonal gradation can get messed up.



John
-

[John Morton]

I have a few extra tone curves loaded into my D700. There are the D2 curves you've already mentioned (which I don't use for shooting but which I have used on at least one occasion in Capture NX2, rendering by far the best white balance in a restaurant where the available light was a mix of dim incandescent and candle), and one very low contrast tone curve I set up to try and replicate the tone range of an old low contrast 35mm film.

I use the low contrast tone curve extensively when shooting in winter, when I am dealing with lots of subtle tonal gradients in white values; but I also like it when shooting inside under diffuse yet adequate available light because that tends to be a low contrast situation, too.

Generally, I shoot using the Nikon tone curves for "Neutral" because I tend to do a lot of HDR landscapes, and this setting tends to help keep the end saturation of the images from going too far over the top.

[ajohnw]

I just wish that the weather in the UK would settle plus no rain to take more shots. in real terms I'm more or less stuck with 2 shots at the moment to play around with. I've managed another raw conversion using a low contrast conversion curve but bright blue is clipping severely as seen in ufraw's top raw file histogram in this. The camera was focused on the 1st car number plate, matrix metering - too far to the right shooting but at least the grass is more or less right now.



John
-

[xpatUSA]

I'm a bit confused about this area and wonder if some one can clear it up for me.

A web page that should be reliable states that Nikon raw files contain a tone curve. The base curve section here.

http://ufraw.sourceforge.net/Guide.html

Maybe they do or maybe the comment is out of date.
-

I'm a bit confused myself about raw data having a 'tone curve' unless it's a look-up table in meta-data. Is the UFraw mention perhaps instead an oblique reference to Nikon's 'compressed' raw files as discussed here:

http://kronometric.org/phot/xfer/NEF_Compression.doc

?

[ajohnw]

I strongly suspect that the tone curve is for jpg production. Going on how Ufraw behave with the camera in "normal" it's a linear one. I haven't used any other camera tone curves yet.. When Nikon software converts it uses it and in Ufraw it's one of the options. This link explains it all rather well and why it's preferable to have custom curves in the camera. They are base curves - in other words applied before the usual curve is manipulated.

http://imagesbyeduardo.com/main/tag/tone-curves/

It's a bit like NX2 after a fashion. Nikon doing it's own thing - NX2 generates a specific ICM file for each shot on the fly. They vary for every shot.

My problem at the moment is ICM files (ICC). I used the same one as ViewNX2 uses to produce the car shot and there is an awful amount of blue clipping - about 3.6 stops into jpg space and even that is clipped in the camera. I get the same colouration as the camera jpg but it's not real. The colour matrix conversion is better in that respect as far as the car is concerned. Colour matrix is there for people who don't have an ICM/ICC file for their camera. With that the red channel has the highest values.

There is an interesting review showing colour accuracy here. Another states it's very good with only a Delta E of 6.4 !!!!

http://www.photoreview.com.au/review...ed/nikon-d7000

Trust Oz to show it nice and clearly like this. I suspect I will be buying a colour card and generating my own ICC file.

John
-

[pnodrog]

I'm a bit confused myself about raw data having a 'tone curve' unless it's a look-up table in meta-data. Is the UFraw mention perhaps instead an oblique reference to Nikon's 'compressed' raw files as discussed here:

http://kronometric.org/phot/xfer/NEF_Compression.doc

?

Just the other day I was looking at a NEF file with a Hex Editor and came across what looks like a table. I was actually trying to work out how the tags in Exif worked but I spotted a table. Part of it below.

[xpatUSA]

Just the other day I was looking at a NEF file with a Hex Editor and came across what looks like a table. I was actually trying to work out how the tags in Exif worked but I spotted [the table above].

Ah, yes, the numbers appear to go in groups of 4 (R,G1,B,G2?) and they increase with increasing location.

Good catch,

[ajohnw]

Not so sure Ted. Do Nikon used compressed TIFF for raw? Thought i had read that some where. If compressed it will have tables in it.

John
-

[xpatUSA]

Not so sure Ted. Do Nikon used compressed TIFF for raw? Thought i had read that some where. If compressed it will have tables in it.

John
-

Sorry, John, I'm not sure which post you're not so sure about.

I've only mentioned NEF, not TIFF. The link I posted here says:

"The encoding tables are stored in the NEF file along with the encoded/compressed data."

The link itself is old now and covers Nikon D1 'up' to D200.

cheers,

[ajohnw]

Interesting doc Ted. I have been curious about what happens to our 12/14 bit colours and there is little info about. I've been thinking about asking an expert but doubt if he would take the time. I like to know how things tick. Even on landscape shots. I mentioned the visual photographic bit depth problem in another post just. 14 might be great but could we see them even if screens etc supported them? Helps with PP though.

The confusion you mention arise because of the nature of compression in general. See the simple explanation of Huffman on top right here

http://en.wikipedia.org/wiki/Huffman_coding

Putting it another way say the data was "zipped". There can be need for a symbol table to "unzip" it.. Simply put say something only contains 8 variations of some 32 bit pattern and there are several 100's of them. They can be identified and replaced with a single number from 1 to 8. When the original data is reconstructed a table is needed that may well be included in the compressed file. So when some one examines some compressed file there may be a table in it that doesn't relate to the data at all just reconstructing it.

I was referring to your reply to the hex editor post and that I had read that nikon use compressed tiff. I suppose all are a form of tiff really underneath.

John
-

[xpatUSA]

Thanks for the Huffman link, John, and thanks also for the cerebral shutdown caused by reading even just the first bit!

However, the article that I linked to says that the 'tone curve' is applied first, and then data compression is applied, so it looks like they are separate issues in a way. I'll hazard a guess that the "encoding table" is not a Huffman-related table? I have a text file of the encoded values and it comprises values from 0-4095 with missing values and larger gaps as it approaches 4095, as one might expect.

Here's a RawDigger histogram of a D50 NEF, clearly showing missing codes for what was a fairly homogenous subject:



cheers,

[ajohnw]

Yes. I have that problem with pages like the Huffman. Most of my work has been in assembler on micro controllers ;-) I only need a slight clue on compression just in case so wouldn't even want to guess what they might use.

Most cameras seem to get at least 9bits into jpg's these days. Olympus seem to be over 10. I am not sure how the nikon tone curve in the raw file fits in. It seems to be applied after the raw data has been tone mapped into screen colour space.in some fashion or might even control how that is done. That's the part I have been trying to bottom out. If I change the base curve in Ufraw it seems to be more or less instantaneous so I suspect it's used after DCraw has done it's stuff. So far I have only taken shots using the default curve as I am more concerned about camera colour profiles. Those nicked from ViewNX2 are a little strange. If I use one on the car blue is clipped in raw and needs -3.5 stops or so to bring it down into sRGB space.

I have managed to get a decent car bonnet now - with a bit of highlight clipping shown that is in the raw file - using a conventionally produced ICC file for the D7000. There is still excess blue but paints are famous for looking different in photo's. Bit like taking shots of people in mixed fabric blacks. The grass and brick colour and the other bits and pieces are about right this time too.



I'm really pleased with this car bonnet version as I was beginning to wonder what I had bought. I can sort of run ViewNX2 on Linux and it doesn't make a very good job of it. Crap springs to mind.

I'll try and take a shot with a different tone curve and see what effect that has on a loaded raw file as the default one seems to be the same as Ufraw's linear.

John
-

[xpatUSA]

Yes. I have that problem with pages like the Huffman. Most of my work has been in assembler on micro controllers ;-) I only need a slight clue on compression just in case so wouldn't even want to guess what they might use.

Years ago I enjoyed programming the Beeb (Rockwell 6502 uP) in assembler, not as my job though. Which makes me think that the following statements need a little explanation, if you would be so kind:

Most cameras seem to get at least 9 bits into jpg's these days. Olympus seem to be over 10.

Later,

[ajohnw]

Years ago I enjoyed programming the Beeb (Rockwell 6502 uP) in assembler, not as my job though. Which makes me think that the following statements need a little explanation, if you would be so kind:
Later,

Easy Ted. This is the tone curves that are available in an EM-5



That might be called the "normal" colour control ones however gradation can be controlled too. These are the curves for that



There is also a manual one that can be set in the viewfinder. These clearly control raw -> jpg tone mapping.

It's interesting to look at say Nikon's approach on say the D7000 or even the D800. They play with the dark end of the "normal" curves and the gradation bottom ends don't differ by much. Olympus do it the other way round. Not of interest to people who don't PP jpg's or set these up. Jpg's are lossy though so if some one saves 90% 3 times they finish up with 73%. Suppose they might get down to something like that if they made very dramatic changes each time. Personally I suspect that if the extra stops available in raw over those that now can be stuffed into jpg's were of any use the camera would put them in anyway.

EM-5 curves in a Nikon thread well I have those to remind me at the moment and the Nikon ones can be found on dpreview in specific camera reviews under dynamic range.

John
-

[xpatUSA]

Easy Ted. This is the tone curves that are available in an EM-5

[DPR dynamic response curves]

There is also a manual one that can be set in the viewfinder. These clearly control raw -> jpg tone mapping.
John
-

Ah, but what you said was:

Most cameras seem to get at least 9 bits into jpg's these days. Olympus seem to be over 10.

I was thrown by the word "bits" - taking it to mean the bit-depth of a image, i.e. the color resolution which, in jpg's, is fixed at 8 bits and cannot be more than 8. So it seems that, by "bits", you meant EV?

[ajohnw]

Ah, but what you said was:



I was thrown by the word "bits" - taking it to mean the bit-depth of a image, i.e. the color resolution which, in jpg's, is fixed at 8 bits and cannot be more than 8. So it seems that, by "bits", you meant EV?

I usually use the word stops but probably have an aversion due to problems in another post.

John
-