Raw-Tiff-Demosaic-Gamma

[Peter Pershan]

I would appreciate an explanation of whether the conversion process from a RAW file to a TIFF
includes

1) Demosaicing
2) Gamma Correction.

Thanks

Peter

[Colin Southern]

Hi Peter,

In a word, yes (to both).

[drjuice]

Respectfully, I'd disagree with Colin's answer. If you use a professional package, think Photoshop or one of the vendors' packages for doing the conversion, his answer is yes. But, for some of the freely available shareware, the programs' developers have clearly decided to go for the exact representation of the source image, mosaic and all. Unfortunately, I no longer have the list of programs I looked at before I chose the one I like. But, as I recall, it was about 10 items long, so it's not uncommon.

In the program I chose to use, dcraw (see wikipedia pages) the only color "correction" is whether you want to use the camera's setting for white balance or tediously build your own. I generally set the camera for AWB and use that setting as it gets me much closer than anything else I've tried.

v

[revi]

Dcraw allows export of the raw data (converted to 8 or 16 bit, though) as a gray scale image with gamma=1 when you use the -d, -D or -E option (see
the man page for all the details).

I don't know of any other programs like this, but I wouldn't be too surprised if at least some use dcraw under the hood (the dcraw page has an impressive list of programs based on it )

[ajohnw]

As I have mentioned before it is hard to find any raw based software that doesn't use some open source software called dcraw. There are a list of packages that use it on it's web site easily found on the web as revi mentions. These include Adobe. Ufraw uses it too. Dcraw does have an option for generating an image without demosaicing. I wasn't aware that ufraw could use it but as Ufraw is really a graphical interface for dcraw I have never used it with command line options. There are a couple of dcraw tutorials about on the web but if I want a pure raw image I now use ImageJ as it saves typing in the dos box or console in my case. ImageJ is a graphics processing engine for technical users with a vast array of specialised plugins. It's fairly easy to use it for this. Much of the rest is intended for image processing specialists. I some times use a pure raw image to test a lens out of curiosity. This is what they look like. The crosses are pixel boundaries. On this one contrast variations can be seen in the GB letters, blur and vertical camera shake. A shadow across the letters as well. The blur is probably CA as most lenses have at least a pixel to 1 1/2 pixels of it.



On the original question for the best gamma correction it's probably better to use the correct camera icc file rather than the defaults that dcraw provides. The defaults generate different results that most noticeably differ in brightness. I some times make marginal manual adjustments to gamma later as it can be useful at times.

-

[Colin Southern]

As I have mentioned before it is hard to find any raw based software that doesn't use some open source software called dcraw. There are a list of packages that use it on it's web site easily found on the web as revi mentions. These include Adobe.

I thought it "unusual" that Adobe would need to use it, so I checked the DCRAW website. I can see a list of "Other RAW Photo Decoders", but can't see anything there to say that all of the ones listed are using DCRAW, unless I've missed something.

[ajohnw]

I thought it "unusual" that Adobe would need to use it, so I checked the DCRAW website. I can see a list of "Other RAW Photo Decoders", but can't see anything there to say that all of the ones listed are using DCRAW, unless I've missed something.

Well I do know that a number of the developers in that list do use it so assume that all do. It's not usual for companies to use specialist contractors or there code in the software world.

-

[Colin Southern]

Well I do know that a number of the developers in that list do use it so assume that all do.

Really not sure how one comes to that assumption I'm afraid, especially considering Adobe's phenomenal in-house human resources.

[drjuice]

Well I do know that a number of the developers in that list do use it so assume that all do. It's not usual for companies to use specialist contractors or there code in the software world.

-

Hi, John in Birmingham -

First, I would not make any assumptions about people specifying the requirements for software or software designers or programmers using ANY "feature" of any sort in any other product, whether it's $old, shared, or free. I have performed all three of the preceding tasks and, depending on what the person wanting the software asks for, the program may or may not have particular feature(s). If it's something I think is necessary or appropriate or would make the program "better" (more in a moment), I'll ask the client. But, for custom software, which is basically what my team does, nothing goes in unless the client has agreed to pay for it. So, if a client wanted a custom program for reading RAW and creating a .PSD or .TIFF or ..., as bad an opinion as I would have of the client who didn't want demosaicing as part of his order, it wouldn't be included.

This is just a fundamental part of our business process, whether anybody else likes how we do it or not.

virginia

[revi]

Really not sure how one comes to that assumption I'm afraid, especially considering Adobe's phenomenal in-house human resources.

Because it's cheaper to use already written (and tested!) code than to start from scratch? Or (as stated in the list of programs 'using' dcraw) they might have used it as
documentation to avoid re-inventing the wheel from scratch. Or the reference to Adobe is to an old version? Or, ...

[ajohnw]

Hi Virginia as you work with software you are probably aware of the old and ancient proverb of if it exists use it or maybe modify it if you can. There is a lot of windows application software about that has used software written initially for unix let alone that which makes use of freely available source. There is also software about that has been literally ripped off with hardly any modification at all from open source and sold. Linux crops up in all sorts of things as it's free and works. Windows going back to very early versions prior to few people using PC's let alone windows was correctly reckoned to have sprung from an idea IBM had at the time. As the dcraw site points out people are free to use his code and I am 99% certain all of the listed software will have used it as a start at least. In that respect one of Adobe's other products Acrobat has some interesting associations Ghostscript. That in and odd sort of was has associations with Postscript and so it goes on. This area is why I stated what I did. One of the latest and most extensive examples of the same sort of thing is the Mac. Neglecting the desktop very strong rumours suggest it's running linux underneath. It would be amazing to find that the gloss on top didn't largely originate from a similar source,

Also to add to the 99% a quote from his resume

1997 - Present
Author of dcraw, an essential tool for digital photography.
Although raw digital photos provide better quality than JPEG, many photographers do not use them for fear that computers will someday be unable to read these undocumented file formats.
Dcraw is a free ANSI C program to decipher any raw photo from any digital camera on any computer running any operating system. It is my gift to programmers, photographers, and future generations of historians.
Thanks to dcraw source code, Adobe Photoshop and dozens of other image tools now have built-in support for raw photos, and the popularity of raw photography has grown enormously.

You see few companies employ people full time with skills that are only needed from time to time. Adobe and others probably employ him as a contractor. Even microsoft use contratcors and consultants at certain levels in there software. Quote from some training material I read some years ago " These people tend to live in big houses and are very hard to see". Made me green with envy. I am a real time specialist - different field and much lower wages.

As to dcraw this is the bulk of the source code. Undertaken from scratch with no knowledges would be a fair undertaking and would probably involve vast expense for your clients and many mistakes even though there isn't that much of it really. It runs incredibly quickly too.

http://www.cybercom.net/~dcoffin/dcraw/dcraw.c

-

[Colin Southern]

You see few companies employ people full time with skills that are only needed from time to time. Adobe and others probably employ him as a contractor. Even microsoft use contratcors and consultants at certain levels in there software. Quote from some training material I read some years ago " These people tend to live in big houses and are very hard to see". Made me green with envy. I am a real time specialist - different field and much lower wages.

As to dcraw this is the bulk of the source code. Undertaken from scratch with no knowledges would be a fair undertaking and would probably involve vast expense for your clients and many mistakes even though there isn't that much of it really. It runs incredibly quickly too.

http://www.cybercom.net/~dcoffin/dcraw/dcraw.c

-

John,

You may be interested to learn that I popped Dave Coffin an eMail to ask if in fact Adobe use dcraw in any of their products - here is a copy/paste of his reply ...

Hi Colin,

Other Raw Photo Decoders

Dcraw has made it far easier for developers to support a wide range of
digital cameras in their applications. They can call dcraw from a
graphical interface, paste pieces of dcraw.c into their code, or just
use dcraw.c as the documentation that camera makers refuse to provide:

In Adobe's case, they "use dcraw.c as the documentation
that camera makers refuse to provide". Most others use the
dcraw source code with their own customizations.

I'm not sure I completely understand what he means by that, but since he makes the distinction between "calling dcraw from a graphical interface", "pasting pieces into their own code" and "using it as documentation" - and then specifically states that Adobe are doing only the latter, then I would take it from that that Adobe aren't using his code in their products.

[revi]

Some code in dcraw uses information that the camera producer didn't document, so that had to be reverse engineered. There are for instance data blocks needed for decompression (huffman trees if I understood those bits). Then there's information on how to recognise different file formats (extension only isn't enough, apparently). Then there's some things about decryption of encrypted data as well.

In short, enough information that is NOT provided by the camera makers to be interesting for other programmers (and taking that from a known working program is always easier than figuring it out for yourself).
And as dcraw allows you to use a large part of the source code without any limitations (see the license in the dcraw.c file), who knows what the different users actually use (as opposed to what they say they use...)

Remco

[ajohnw]

Some code in dcraw uses information that the camera producer didn't document, so that had to be reverse engineered. There are for instance data blocks needed for decompression (huffman trees if I understood those bits). Then there's information on how to recognise different file formats (extension only isn't enough, apparently). Then there's some things about decryption of encrypted data as well.

In short, enough information that is NOT provided by the camera makers to be interesting for other programmers (and taking that from a known working program is always easier than figuring it out for yourself).
And as dcraw allows you to use a large part of the source code without any limitations (see the license in the dcraw.c file), who knows what the different users actually use (as opposed to what they say they use...)

Remco

Basically it's a million to 1 that they also took a close peek at all of the source. A lot of that goes on with open source software of one sort or another. Lots of software originated from Unix. Linux came along and then software from there started playing a part. More recently lots of originally Linux software is still open source but is also available for Windows and Mac. When these applications are used and sold commercially various things happen to them. Restructuring - that basically means shifting stuff around and maybe doing the same thing in a different fashion. It's a trivial activity for a competent software engineer as the hard work has been done. Call the labels something else - software basically runs from one named section of a program to another. All this does is make compilations look look a little different at a debug level. And last but not least do nothing at all to it. Just sell it. There are obviously mixes of these as well.

I would strongly suspect that Adobe fall weakly into the 1st category at least initially. As to skills base there are various levels of competence in the software world. I shudder to think what would happen if a C++ application programmer went to work on dcraw from scratch or as it is. People don't seem to realise that "general" software mostly makes use of functions built into the operating system and little else. Actually I take my hat off to Mr Coffin. He has provided a relatively clean easy to read piece of software that also happens to run fairly quickly.

-

[xpatUSA]

Actually I take my hat off to Mr Coffin.

Me too. Especially his support for the RAW files from my early Sigma DSLR (SD10). With his stand-alone command-line version I finally solved a puzzle with the -D option (Document Mode, no interpolation) apart from all the other arcane options available. Well done, DC!