Converting raw to DNG

[herbert]

Um, not sure where you got that from Alex -- it's a option with the DNG converter, but is isn't even the default behaviour of ACR -- not sure about LR, but I'd be surprised if it was.

Hi Colin,

I'm not sure where I got it from either. It has been in my head for a long time and so I think it had solidified. I'm probably regurgitating rubbish I read somewhere else. I apologise since I do not want to misinform anyone. I'll edit my original post to note this error.

It makes no sense that the image would be demosaiced before storage as DNG. The file is simply a 14-bit value per pixel (or whatever the bit depth is). It should not have to be demosaiced to be stored elsewhere as raw. It would be for an RGB TIFF (or any other RGB file). If it was demosaiced, it would not be raw. I think the gains in smaller file storage are due to the use of a better compression algorithm than that used in the manufacturer raws. They have to be compressed on-the-fly within camera and so it is advantageous to use a faster less efficient algorithm.

The demosaicing is done before storage in the ACR cache. However this will be specific to the ACR version and so will be done again if ACR gets updated to a new process version anyway.

Incidentally I read that the ACR cache for LR 4 stores the files as Adobe RGB JPEGs and so the cache files are much smaller. I was puzzled by this since it means any heavy lifting within ACR development will always have to go back to the original raw and do the demosaicing again. I am not sure how the new ACR will use the cache files. Perhaps they are now only used for thumbnail tiling previews where the lossy JPEG compression is not noticeable. Or more probably they are using the lossless compression part of the JPEG specification without using any of the lossy compression routines that reduce colour bit depth using fixed sized lookup tables for 8x8 grids (hence the blocky appearance of highly compressed JPEGs). As usual there are lots of things you could do to the image and they never tell me enough about the process for my curiosity.

Alex

[Colin Southern]

I used to convert all my files when downloaded to LR into DNG. I have since altered my method.

I think that the optimum method possibly (probably?) depends on individual requirements. For me, I'm usually shooting sets in the studio or a landscape scene outside - either way, I end up with several dozen to several hundred files, but (a) only a small percentage of those are ever processed beyond a basic level (ie are "keepers") (even though I keep all of the physical files anyway), and (b) since they all pertain to a particular event, it makes sense to keep them in their own "container" (folder) ... so for me, importing them all into a master database like LR makes no sense at all; I simply have Bridge place them in an appropriately named folder - having been converted to DNG as part of the transfer process - and then that container is backed up from there.

What makes me super-nervous about the LR "catalog" (database) concept is that if edits are stored in there then one literally has an ever-increasing amout of work contained in that one databse -- and that's a scary thing. Most seem to be aware of the importance of backing up such an important database, but I suspect few ever throughly check the integrity of those backups -- or the integrity of the database itself. Consider the following scenario (it's one that's bitten me fair and square in the bum) (although it wasn't a LR database, but same principles apply); a photographer creates a new calalog for the new year -- lets call it "2012". In January he shoots a bit - imports the shots into lightroom - edits them - and next month moves on to his February work - and then March, April, May, June, July etc. He's careful to backup the database every week - but by the time he's processing September work, he's LONG SINCE FORGOTTEN ABOUT JANUARY WORK. Now if somewhere along the way that database gets corrupted - and edits are lost for, say, January's work, he's quite likely not going to notice - and if he doesn't notice he won't be aware that anything's wrong - and if he's not aware that anything is wrong then he'll quite happily backup the database and probably over-write some or all (eventually) of his earlier backups. Come November he discovers the problem, but it's too late -- the edits are gone, and he's over-written all backups with the corrupt database.

For me, working in DNG means the edits are NEVER disassociated with the actual RAW data - so I know that if I backup the container that they're in then I've got the RAW files - the edits - and I don't have to worry increasingly that "one day" I might lose the lot. And lets be honest, most folks are lousy at regular backups - let alone testing the backups - let alone handling restores.

Just my 10c worth ...

[Colin Southern]

I'm not sure where I got it from either. It has been in my head for a long time and so I think it had solidified. I'm probably regurgitating rubbish I read somewhere else. I apologise since I do not want to misinform anyone. I'll edit my original post to note this error.

Hi Alex,

No worries - apologies too if I appeared to "jump down your throat" on that one -- unfortunately there's a lot of mis-information floating about with regards to DNG and it's becoming (become?) a bit of a "personal mission" to try and keep the info accurate.

Personally, I don't think we're going to see major steps forward in demosaicing anyway (and even if we did, it still begs the question of "what would we do with the enhanced detail" when we already throw 95% of it away most of the time.

The big issue with demosaicing before storing in DNG form - in my mind anyway - would be the fact that it would treble the file size.

[Glenn NK]

In LR, when one exits the program, the following screen appears:

Back-up Catalog:

Test integrity before backing up? this can be checked or not.

Optimize Catalog after backing up? this can be checked or not.

Then there is a Back-up" button, and LR takes a few minutes to do its "thing" - creating a back-up of the catalog where the edit information is stored. I do this procedure every time - and I recommend that everyone does it.

Once a week/month/whenever I feel like it, I copy all my files to an external HDD. All the edits are backed up.

I'm still very much in the learning process of PP, so I frequently go back to old files (worked two hours last night on this), and re-process file with my newer knowledge and experience.

Glenn

[herbert]

Hi Alex,

No worries - apologies too if I appeared to "jump down .

No problem. I did not feel offended in the slightest. More ashamed of my error. That has now been sorted. I was more apologising for spreading misinformation to the whole forum. This is a learning forum so we should make sure we get it right.

I think demosaicing might be improved on an image by image basis for parts of the scene. However improved is subjective and so this would invariably require an interactive demosaicing where you can choose the number of pixels that contribute to your RGB values and the weightings. Although such a tool may be interesting from an image processing perspective it is unlikely to actually make a difference in the real world or we would have seen it already. In fact I suspect that the number of surrounding pixels for the RGB values is what is varied when adjusting the colour noise sliders in ACR. The detail slider may adjust the spread too depending on the spatial frequency of the pattern, i.e. the edge size.

Demosaicing of the new Fuji RGB sensor in the X1 Pro may require some new thoughts though. It will be interesting to see if any other manufacturers try something other than a Bayer filter in their cameras.

Alex

[Colin Southern]

In LR, when one exits the program, the following screen appears:

Back-up Catalog:

Test integrity before backing up? this can be checked or not.

Optimize Catalog after backing up? this can be checked or not.

Then there is a Back-up" button, and LR takes a few minutes to do its "thing" - creating a back-up of the catalog where the edit information is stored. I do this procedure every time - and I recommend that everyone does it.

Hi Glenn,

It absolutely does - it does it does it does - no question about it. Dunno - call me old, call me a fool, but for me, (a) it's just too many eggs in one basket, and (b) I've just learned over the years not to trust everything that a computer tells me. Good case in point at the moment; I'm working on a clients PC - the computer is telling me that everything is fine -- and yet more advanced tools are detailing 24 issues ...

[gaijin]

Thanks everyone for this little burst of energy. To kind of summarise, it looks like some people do (convert to DNG) and some people don't... Some people don't because they feel that that may be right strategy to follow to take advantage later on as software technology moves on (erroneously, it would seem) - I myself am not too convinced by this argument anyway because I can't really see myself reprocessing photos that I took some years before. Some point out that it may not be possible to get older versions of Photoshop to process newer raw formats - not a problem I have yet because I've got CS5. The processing cost (time is money) of doing the conversion and the need to backup bulky DNG files and not simply the sidecar files evidently puts some people off. Changing something that works for something you're not sure about is, I imagine, reason enough for people to stay put.

My summary of sorts probably doesn't do justice to the various people who have contributed here - my apologies. If I've scandalously misrepresented anyone or omitted something important, please feel free to correct!

Thanks one and all.

[Colin Southern]

The processing cost (time is money) of doing the conversion and the need to backup bulky DNG files and not simply the sidecar files evidently puts some people off.

Hi Allen,

I think you summarised it well. I might add though that I really don't account for a "processing cost" - my PC can convert a file to DNG faster than it can transfer it from the memory card (so "no penalty") and it actually takes far longer to generate the 100% previews & thumbnails -- which I STILL don't account for since I just "leave the PC to it"; there's always clients to talk to - studio to be reset etc.

In terms of backup time - it rates right up there with "watching grass grow" and "watching paint dry" and as such I simply just start a backup and then go home -- all done when I come back in the morning

[gaijin]

An afterthought, or rather another question: how do you view DNG files on 64bit Windows 7 (or on Apple computers for that matter)? I've been using ViewNX2 to look at my NEF files but that (obviously?) doesn't handle DNG files. Are people who convert to DNG not bothered that Windows etc can neither display thumbnails nor the files themselves? Do I have to get the Fast Picture Viewer and its codec (or something else)?

Many thanks!

[Colin Southern]

An afterthought, or rather another question: how do you view DNG files on 64bit Windows 7 (or on Apple computers for that matter)? I've been using ViewNX2 to look at my NEF files but that (obviously?) doesn't handle DNG files. Are people who convert to DNG not bothered that Windows etc can neither display thumbnails nor the files themselves? Do I have to get the Fast Picture Viewer and its codec (or something else)?

Many thanks!

Maybe it's a bold assumption, but I've assumed that if folks were converting to DNG then they'd probably use an Adobe product for post-processing.

Personally, no, it doesn't bother me in the slightest that Windows can't do anything with my DNG files - Bridge does the job 100 times better (sorting / filtering / ranking / magnification etc)

[Black Pearl]

An afterthought, or rather another question: how do you view DNG files on 64bit Windows 7 (or on Apple computers for that matter)? I've been using ViewNX2 to look at my NEF files but that (obviously?) doesn't handle DNG files. Are people who convert to DNG not bothered that Windows etc can neither display thumbnails nor the files themselves? Do I have to get the Fast Picture Viewer and its codec (or something else)?

Many thanks!

Can't be bothered to dig a Windows machine out but a quick test show that my iMac can preview and open DNG files perfectly.

[jprzybyla]

I convert all RAW files to DNG when importing into Adobe Lightroom. After post-processing in Lightroom sometimes I use the HP Photo Viewer or Windows Photo Viewer for quick views. I have Windows 7 64 bit on my laptop which previews and opens DNG files without problems.

[proseak]

An afterthought, or rather another question: how do you view DNG files on 64bit Windows 7 (or on Apple computers for that matter)? I've been using ViewNX2 to look at my NEF files but that (obviously?) doesn't handle DNG files. Are people who convert to DNG not bothered that Windows etc can neither display thumbnails nor the files themselves? Do I have to get the Fast Picture Viewer and its codec (or something else)?

Many thanks!

I don't use Windows to view images at all. I use either Bridge, Irfanview, or Faststone to browse folders, all of which I've found to be faster and more capable than XP, or 7.

They all show.dng files well.

HTH

Peter

[proseak]

Almost, but not quite, returning to the original subject, I do use DNG for another purpose. I still do a lot of film scanning, and get 16-bit tiff as output. I use ACR to convert these to DNG, which brings file size from c. 200Mb to c.130Mb. This results in a big saving in space.

cheers

peter

[drjuice]

This very issue is one of the reasons I believe in FOSS image handling software. It is VERY easy to keep an intact copy on my STA memory stick (32GB) that goes with me EVERYWHERE. Besides my digital books in case I need a reference when I'm on the road, copies of the CDs/DVDs for my critical software, it copies of all the tools I use to diagnose hardware/software problems (which is why it's called STA -- Save Their ----- from when I used to do a lot of that type of work for other professors who fouled up their computers).

Proprietary software has the unfortunate property that the owning company/individual wants us to buy upgrades to the software. So "old features" are often dropped when they decide those capabilities aren't used any more. (The claim is usually to prevent code bloat, but the software keeps getting bigger and bigger and bigger.) But, keeping old proprietary software around to defeat this attitude doesn't work many times because they won't allow having V1, V2, V3 ... Vn-1 + the latest version running on the same system. ;~(

I can't remember the last time I couldn't convert a file from a format I had to a format that I wanted to have and, with one exception, that has happened in FOSS software. The one exception was when I wanted an image converted from IBM's proprietary .ADM format to a format I could work on on a PC. Fortunately, I came on an inexpensive image handling and drawing package.

I hope this helps.

v