Compatibility - Adobe Digital Negative Converter

[Pica]

Hello Everyone,
I have recently had I use for the Adobe Digital Negative Converter. Thus I've got a reason to think about the following. Would you like to support me with your input please?

I know about the ACR versions (of course).

My questions are brief but vital:
- What are the practical differences in contents between a DNG file created with compatibility "Camera Raw 7.1 and later" and "Camera Raw 5.4 and later"?
- What are the pros and cons, when comparing ditto DNG-files?

_______

Miscellaneous: As a former senior systems engineer I did two tests, which made me even more interested in the above topic! Before doing the tests I may have thought: 1) The sizes of DNG-files would have been different. 2) It would not be possible to open the compatibility Camera Raw 7.1 in ACR 6.5.

Result: File size exactly the same; and it was possible to open compatibility "Camera Raw 7.1 and later" in ACR 6.5.


Thank you for your cooperation from Scandinavia!
-

[Shadowman]

Pica,

Can't answer your questions but can add a bit too your query. I also use the DNG converter and recently found out that Nikon's ViewNX2 doesn't recognize the DNG files. You might ask (or perhaps not) why use ViewNX2 when you've already converted the files, I liked the simple editing functions in the Nikon software for images that didn't need a lot of processing. So even though DNG files can be edited fully as if RAW, there must be some differences in the format that aren't fully tranferable. I think what can be said about DNG files are that they are as different as a NEF (Nikon) file is to an ORF (Olympus) RAW file.

[Manfred M]

Unfortunately, I am not quite sure of the various versions and how they were released with the various Photoshop versions.

The Camera Raw engine was given a major overhaul in 2012. I believe that this was when CS6 was released. As an end user, I saw significant quality improvements in the output from the raw converter, especially with respect to skin tones on Nikon cameras. Prior to that I would use either Nikon View NX2 or DxO Optics Pro for critical conversion work.

The main difference between the other releases has more to do with additional camera models that were added to the compatibility list. Camera Raw is not backwards compatible, so a camera that was released this year would only work with the ACR shipped with Photoshop CC 2015. In some cases, I suspect that the only changes were the data sets for the camera models, with the basic raw conversion engine not changing at all and a new version number would be assigned corresponding to the version of Photoshop.

I am surprised that you would have thought the files sizes would be different. The files contain the headers for the metadata and then specific pixel data. The value of the specific pixel data could change, but a 16-bit integer value would be the same size, even if the data content of a specific data value changed. As neither of these two would change between images, I can't see why there would be any file size change.

As for opening the files, Adobe has maintained forward compatibility for older cameras. If a file can be opened in ACR 6.5, it would certainly open in 7.1. The reverse is not necessarily true.

[dje]

Hi Pica

There are two considerations here

Updates to DNG Converter

Updates to DNG Spec

Most of the updates to the converter are simply to include new camera models. However along the way, the converter would also have been updated at times as a result of updates to the DNG Spec. The converter produces files that conform to the spec. The current spec, 1.4 I think, has an appendix which lists compatability issues with previous versions of the spec.

I'm not able to provide a link to the spec as I'm travelling at present and have limited internet. But you could google it.

Dave

PS will be landing in Copenhagen soon!

[Manfred M]

Updates to DNG Converter

So far as I understand, Adobe has had three different "modern" versions of its raw converter "engine". The first one was in 2003, it was updated again in 2010 and the current one came out in 2012.

[dje]

So far as I understand, Adobe has had three different "modern" versions of its raw converter "engine". The first one was in 2003, it was updated again in 2010 and the current one came out in 2012.

Manfred I'm referring to the Adobe DNG converter, either standalone or in LR or Bridge, not the Adobe RAW converter as used in ACR and LR. The DNG converter merely converts the RAW File to DNG format.

Dave

[Pica]

Hello Dave,
Nice to meet you.

You are following my train-of-thought: As I stated in the OP, this thread's scope is the Adobe Digital Negative Converter; nothing else.

Thank you for your present (and future) contributions to my exploration of this topic. And hope you will have a safe travel to my part of the globe from Australia! Hope Copenhagen will meet you with all its best!

Manfred I'm referring to the Adobe DNG converter, either standalone or in LR or Bridge, not the Adobe RAW converter as used in ACR and LR. The DNG converter merely converts the RAW File to DNG format.
Dave

[Manfred M]

Manfred I'm referring to the Adobe DNG converter, either standalone or in LR or Bridge, not the Adobe RAW converter as used in ACR and LR. The DNG converter merely converts the RAW File to DNG format.

Dave

I understand Dave, but the underlying RAW conversion engine is common to all three pieces of software.

[Dave Humphries]

I understand Dave, but the underlying RAW conversion engine is common to all three pieces of software.

Does the DNG conversion engine actually need to do any "conversion"? (you can tell I'm guessing here)

Is it not just, to over simplify, 'topping and tailing' the proprietary file format with some Adobe standard header and footer meta-data, etc. It may also 'bit shift', multiply or otherwise adapt the numeric range of the exposure values from the sub-pixels.

[Manfred M]

Does the DNG conversion engine actually need to do any "conversion"? (you can tell I'm guessing here)

Is it not just, to over simplify, 'topping and tailing' the proprietary file format with some Adobe standard header and footer meta-data, etc. It may also 'bit shift', multiply or otherwise adapt the numeric range of the exposure values from the sub-pixels.

My understanding is that it does. The camera (other than the Sigmas with their Foveon sensors) captures data using a Bayer filter. That means that the actual colour for a specific part an image has to be assigned specific RGB value. Each individual photodiode is surrounded by 9 other photodiodes based on a RGGB pattern so the adjacent colour values have to be calculated in. How far out these "close together" pixels are blended together are the secret of the specific raw converter. Every sensor is going to have "dead pixels" and "hot pixels" and these have to be found and incorporated into the final image data as well.

[DanK]

I don't understand what DNG really is, but Adobe's technical documentation has these two statements. They seemed contradictory to me, which may show how little I understand this:

Adobe has defined a new non-proprietary format for camera raw files.

DNG is an extension of the TIFF 6.0 format, and is compatible with the TIFF-EP standard. It
is possible (but not required) for a DNG file to simultaneously comply with both the Digital
Negative specification and the TIFF-EP standard.

A raw file, I would have thought, needs no conversion, whereas a TIFF does.

The documentation is here

[xpatUSA]

I don't understand what DNG really is, but Adobe's technical documentation has these two statements. They seemed contradictory to me, which may show how little I understand this:

<>

A raw file, I would have thought, needs no conversion, whereas a TIFF does.

The documentation is here

I see that the documentation includes two forward matrices for conversion from camera space to XYZ space and I am wondering if the DNG image data is actually in XYZ format? I seem to recall from prior interest in DNG that storing the original raw image data in the DNG file is an option and is not necessarily required.

Point here is that if the original raw data is not embedded, there's no way back to it. The only part-way back is to go via the inverse transform to some sort of three-color TIFF file, not the original RGGB values.

And having the camera space transformed to XYZ would maximize the number of Readers capable of rendering the image.

Regrettably, it seems to me that "conversion to DNG" is not a simple matter of sticking new meta-data into a file + the raw image data. Not simple at all.

Personally, I treat raws and TIFFs as ephemeral. Usually I trash them when done, i.e. when the output is saved to wherever it's going. No need to jump me, I am well aware of the pitfalls of that approach and have regretted it occasionally.

[Manfred M]

Ted - you are right and the issue is that it is a one-way street.

That is my main issue with it as while it is sort of non-proprietary (i.e. Adobe will licence it to anyone under certain terms and conditions that likely seem steeped in Adobe's favour), it does mean that it limits future conversions with, say DxO Optics Pro's excellent raw converter. It used to also exclude Phase One Capture One, but I think it is now capable of reading the format, but I'm not sure to what extent certain parts of the conversion process will already have been done.

[dje]

My understanding is that it does. The camera (other than the Sigmas with their Foveon sensors) captures data using a Bayer filter. That means that the actual colour for a specific part an image has to be assigned specific RGB value. Each individual photodiode is surrounded by 9 other photodiodes based on a RGGB pattern so the adjacent colour values have to be calculated in. How far out these "close together" pixels are blended together are the secret of the specific raw converter. Every sensor is going to have "dead pixels" and "hot pixels" and these have to be found and incorporated into the final image data as well.

My understanding is that the DNG converter does not do any processing of the data such as demosaicing.

Dave

[dje]

I see that the documentation includes two forward matrices for conversion from camera space to XYZ space and I am wondering if the DNG image data is actually in XYZ format? I seem to recall from prior interest in DNG that storing the original raw image data in the DNG file is an option and is not necessarily required.

Point here is that if the original raw data is not embedded, there's no way back to it. The only part-way back is to go via the inverse transform to some sort of three-color TIFF file, not the original RGGB values.

And having the camera space transformed to XYZ would maximize the number of Readers capable of rendering the image.

Regrettably, it seems to me that "conversion to DNG" is not a simple matter of sticking new meta-data into a file + the raw image data. Not simple at all.

Personally, I treat raws and TIFFs as ephemeral. Usually I trash them when done, i.e. when the output is saved to wherever it's going. No need to jump me, I am well aware of the pitfalls of that approach and have regretted it occasionally.

My understanding is that the colour space profiles (specific to the camera model)are embedded in the DNG file for use by ACR or LR. The raw data is not converted in any way.

Dave

[Manfred M]

My understanding is that the DNG converter does not do any processing of the data such as demosaicing.

Dave

It has to do this to a certain extent (execpt for perhaps Ted's Sigma / Foveon). In all other cameras we use a Bayer Array which generally uses a RGGB filter (although some manufacturers have gone beyond the standard Bayer design) and has to construct individual RGB values from this data. I can see no technical way of turning raw data into image data without doing this.

[dje]

It has to do this to a certain extent (execpt for perhaps Ted's Sigma / Foveon). In all other cameras we use a Bayer Array which generally uses a RGGB filter (although some manufacturers have gone beyond the standard Bayer design) and has to construct individual RGB values from this data. I can see no technical way of turning raw data into image data without doing this.

Yes Manfred but it's done by the raw conversion software in ACR or LR, not by the DNG conversion software which is separate. The latter can be run from Bridge or LR but it can also be run as standalone software.

Dave

[Manfred M]

Dave - I have used this piece of software. It is basically ACR without the parametric editor thrown in. Throw your raw files at it and it will spit out dng files. I suspect it is what Bridge uses when you use the "Import Images from Canmera" using the dng option.

[xpatUSA]

More on DNG . .

First, some may find this interesting:

http://www.openraw.org/node/1482/

Basically the author lists reasons why he doesn't like DNG.

More here:

http://www.barrypearson.co.uk/articles/dng/history.htm

I'm personally looking for what image data is stored and in what format. The stupid Adobe Spec never actually says

I'm still looking but it does appear that earlier posters are correct in that the raw data is stored as-is but the meta-data itself is what is standardized, in accordance with their spec. Trying to keep it simple, they specify what the meta-data is and how the Reader shall use it.

So, if their meta-data and it's usage differs from the manufacturer's in any way . . . draw your own conclusions

By virtue of the DNG meta-data being standardized (the Adobe way) the differences are in the conversion and there can be significant differences. One such is good old WB. The Adobe way is have two WBs e.g. 6500K and 2850K in the DNG meta-data and expect the Reader to interpolate betwixt the two according to the embedded (I think) DCP data. So, goodbye to your camera's WB multipliers, whatever they were.

Then there is the camera manufacturer's transformation (often 3x3 matrices) from camera space to XYZ or RGB, i.e. from raw to color. I believe that what you get is what Adobe thinks is best, not the camera manufacturer; and I believe further that, if the camera manufacturer's matrices or whatever are many and complex, the Adobe "converter" just lumps them all into one matrix and calls it "ForwardMatrix" I don't really understand their system of matrices so that's a bit of a guess, to be honest. The main point is that the matrices are Adobe's best guesses, not the manufacturer's specifics. Which means that your DNG Reader will render image colors differently to what the proprietary converter will show you. How different? Hmmm . . .

Now I realize exactly why Adobe dropped Foveon! With the introduction of Sigma's True II processor, the amount and complexity of meta-data that is to be used by a converter, took a giant leap and much of it (while being necessary for conversion) is unmappable to the Adobe Standard Way.

Out of interest for some, here's the Sigma Merrill meta-data extracted from a raw .X3F file:

http://kronometric.org/phot/sensor/SDIM0759.X3F.meta

It can be read in Windows 'WordPad' *** (sorry, don't know the Mac, linux equivalents) and it will make your eyes glaze over, garawnteed

*** FireFox renders it OK on my system (Win 7).

[Pica]

Hello Everyone,
When reading your answers it seems to me that this has been a stimulating as well as challenging topic.

Perhaps this thread is now coming to its close, and I would therefore like to thank every contributer for supporting me on this and coming explorations of the DNG-file versus other camera raw-files.

Thank you for your written posts, as well as links, which I appreciate.

See you. Allt the best from Scandinavia!
-