Archiving your original files - which format?

[darkslide]

Firstly I'm not a pro, just an amateur petrified of losing his data...

I backup my original files (before and after any post processing) to three seperate hard disks - working in IT has taught me not to rely on a single hard disk...

The questions I have relate to the file format - I shoot in RAW and I'm concerned about future-proofing my data. Is there any value in converting the files from .NEF to .DNG for example?

It's reasonable to expect my .JPG files to be readable in the future, but if I want to absolutely sure I can re-process RAW data, can I be sure that the .NEF conversion for the backed-up (and thus older) data will still be working?

Food for thought...

[Simon Garrett]

Good question!

I agree about 3 copies: also been working in IT long enough to know that the only people that have not lost data are either very young or very lucky. I take the view that data doesn't exist unless there are at least 2 copies (I don't wipe the memory card until the images are copied to a computer and backed up).

As for archive format: I asked Nikon a few years ago whether they guaranteed to support NEF format for obsolete cameras indefinitely, and the answer was a definite no! They recommended TIF for long-term archive. Clearly TIF and jpeg are standards that will probably be readable for many many years. However, my guess is that NEF will be readable for a pretty long time in one way or another. I use Lightroom, and my working assumption is that if Lightroom becomes obsolete or stops supporting old NEF formats (and I'm still alive!) then I'll have to export all my images from NEF plus Lightroom edits to TIF or JPEG before finally dumping Lightroom. In fact, I periodically export all my images to jpeg for archive, just so I've got something. As for DNG, well it's sort of vendor-independent, but probably in fact depends on Adobe's continued support. If Adobe abandon it, then it's not clear to me it will be any more useful than NEF.

Then there's the question of archive medium. CD and DVD (even blu-ray) are really too small to be useful. Anything much less than about 500G currently (and perhaps much more if camera resolutions continue to rise) becomes tedious. Which means hard drive is the only practical possibility for archiving. And how long before current drives are obsolete? It was IDE until a few years ago, now it's SATA, what next? My current policy is to keep all my images on one drive (plus two backups). Currently I've got 450G of images. When my 1T drives become full, I'll get 2T, in whatever is the current technology. That's not really a long term archive strategy: it's simply keeping up with whatever's current. And that's before considering physical durability of the media.

It's ironic that we can easily read all archives from 2,000 years ago (and most from 3,000 years ago). But computer files from 30 years ago may be virtually unreadable.

[darkslide]

I'm glad to see I'm not alone in this - slightly less lonely!

DNG is supposed to be open-source - but as you correctly state, this, to some extent, also depends on Adobes continuing support. It seems odd to me that there hasn't been some long term agreement from the manufacturers on this - we can read and write to hard drives, CD's etc. which implies common 'standards' so why not image files?

In terms of price per Gb hard disks are not particularly expensive, but it seems a huge hassle backing-up in different formats to several disks. I think I'd be prepared to have separate NEF and DNG disks, but where does it end?

[jcuknz]

Tiff and jpg seem to be the only formats that have stayed around. jpgs can be saved as lossless, or near lossless, when you have control over the process. I don't have a crystal ball so cannot suggest what the future will bring.

[Geoff F]

I take it, Ian, that you are talking about long term storage of Raw files not processed images which can easily be stored as Tiff or Adobe psd. The psd option does save a bit of space.

I normally save my Raw files in the 'native format' which is Canon CR2 in my case. But, yes there is a risk of those formats being obsolete in the future. And I find the extra tagged 'sidecar' files to be something of an additional risk.

So I have experimented with Dng as an alternative but with some drawbacks. Firstly, the conversion process does take a bit of extra time; at least 10 mins for 100 photos for me.

Also, and probably a computer fault on my part, I have been getting the occasional corrupted file when converting to Dng with ACR which doesn't happen with the simple Canon EOS downloader.

But on the plus side, Dng gives substantially smaller file size.

Whether Dng will be a universal format for all time is something of an unknown quantity. At the moment however, I find that most software will work with CR2 but many will not accept Dng. Although, of course, there is no problem with Adobe software.

[darkslide]

Yes Geoff I was referring to long term storage.I'm not too fussed about the time it takes as I don't shoot thousands of images per day and I feel sure I'd be able to automate (to some extent) a transfer/conversion process.

I'm more concerned with being able to re-work older archived data in a few years time. I agree that for the most part, I'll be happy to leave older stuff well alone but you never know...

Thanks for your input

[Bryan Conner]

I really do not think that there should be concern about being able to open common proprietary raw formats in the future. Sure, there are raw formats that are no longer supported in most raw converters, but these formats are ones that are very uncommon. Besides, why would one believe that the people in the future will not be knowledgeable enough to do what people currently do each time a new raw format is released?

Having said that, I do support the notion of having the option of saving your raw images into a generic raw file format. We already have the option of either writing to a raw file, or to a jpeg, so why not add a third option? It does not even have to be Adobe's dng format, it can be another one, just give us the choice to do what we wish with our files that our camera produces.

[Colin Southern]

I convert all of my images to DNG format, and then discard the original CR2 files.

I see it this way ...

(a) Even if Adobe dropped support for DNG, since it's an open format, somebody else would be sure to pick it up, and

(b) We still have existing programs that could read existing files.

Losing data is more of a threat in my opinion; having multiple copies guards against a single HDD failure, but not against things like fire / theft / flood / hurricane / tornado / virus / lightning etc if they're all stored in the same location or connected to the PC concurrently.

My current practice is to keep the current projects (being worked on) on a portable solid-state drive that travels to/from home/work each day - I copy the latest revisions to home & work PCs each time I'm home or at work, and I upload various key images & files to Google Drive as an "ace up my sleeve".

[MrB]

Why do we have all the different Raw formats? Is one better than another - e.g. Is Nikon Raw superior to Canon Raw? Or is it simply the profit motive - presumably each manufacturer is paid by companies such as Adobe for software to read their files?

Some camera manufacturers, e.g. Pentax, include the in-camera option to record .dng files. I have set this option in my camera, assuming that .dng files are more likely to be future-proof - is my assumption mistaken?

Philip

[Simon Garrett]

Why do we have all the different Raw formats? Is one better than another - e.g. Is Nikon Raw superior to Canon Raw? Or is it simply the profit motive - presumably each manufacturer is paid by companies such as Adobe for software to read their files?Philip

Not totally down to profit, more a case of "not-invented-here". And wanting a format they control, and not someone else (Adobe). Also a (misguided, IMHO) believe that they can do better, and need the flexibility to put whatever they want in the raw files without having to disclose the contents. It's misguided as DNG can (as I recall) include proprietary fields.

RE Brian Conner's comments about software to read any raw format in future. I sort of agree, but not quite! There have been plenty of data formats from years gone by which have been widely supported, but now the only software for them runs only in MS-DOS or Windows 3 or some such. Yes, dcraw opens all raw formats and is available in open source C code, but in 20 years time, will you personally have the expertise to modify it to run on Windows 20 in whatever .NET 10 language they're using then? I'm a reasonably experienced programmer, but I know I'd find it pretty hard work.

I think we're pretty safe with raw and DNG for the next decade or so, but if you want something you know for sure can be read without hassle beyond that, I'd convert to TIF or JPEG as an archive copy. Personally, I keep archive jpegs of the editied versions of my photos as well as the raws (plus Lightroom edits in the catalogue and xmp files, and TIFs of stuff edited in Photoshop).

[ajohnw]

I believe the size of raw files vary with camera but that is likely to be the most compact method available as it stores black and white images along with the colour information and no interpolation has been carried out.

I just tried several formats on a single raw file. Olympus raw file 10.9mB, tif 58.7mB, 100% jpg 5.8mB. The raw files in this case vary from about 10 to 15mB. I believe most manufacturers compress them. DNG comes out at 11.6mB. Other formats. A ppm file comes out at 72.3mB and png 57.8mb both 16bit. These are for 12mb images.

Personally I feel that the best option for archiving is the original raw file or the dng along with what is sometimes called the pipe. Many packages allow the processing settings that have been used to be saved - the pipe. If the package becomes obsolete these wont be of much use but the original raw or dng file is still available. I would assume that adobe would offer converters if dng changes and in any case the original raw file is still very likely to be viable. It's interesting to see where the raw converter comes from. List of users and most of the cameras supported on this page. http://www.cybercom.net/~dcoffin/dcraw/ Also instructions about what to do if a camera isn't supported. Problem here is the software producers - adobe etc may have to do something as well.

-

[darkslide]

Many thanks for the time you've all taken to reply - gives food for thought doesn't it.

[Brocken]

Hi Ian, a few thoughts.

It may help your future-proofing to store not only the image files, but also a copy of the software that was used to read/convert them, such as, for example only, Nikon NX2. If you wanted to go further, you could also store a mirror copy of a virtual machine that will run this software.

For extra security, you can also store your most important material on archival grade CD/DVDs- guaranteed for a claimed 100 or 200 years - and store these at one or more separate locations in case your house burns down, is bulldozed by zombies, or other force majeure event.

[darkslide]

This is a perfectly reasonable, logical train of thought - but you must remember that we are also dealing with the "We're a fruit and we're going to rule the world" attitude which dominates at Apple. I use Mac - and you don't have to go too far back into the past to hear stories of software which is distinctly NOT future proofed...and they're not alone in this.

There is no perfect solution - I even store a copy "in the cloud" reassured by the knowledge that my service provider backups up his data hourly/daily/weekly etc. but just so long as I pay the bills and he remains in business.

Sadly even archival CD/DVDs are not sufficiently robust (to my mind at least) - how do we test archival permanence? Wait a hundred years then panic when we can't read the damn things? I'm joking, and I'm certainly not paranoid about this - when I'm no longer here I feel sure I won't be the one losing sleep over it!

I'll continue to mull this over - my even do a few conversions to DNG - again, many thanks for the input.

[Ady]

I've worked with a number of archivists who maintain archives of written and printed records and more recently of photographic materials. Some of the archives hold records that go back a 1000 years or more so the whole photography thing is still a bit modern and new fangled

Of course most of these archives are now recieving digital media for inclusion and the last time I checked the baseline recommendations from an archivists perspective were:

For the Master Archive: Uncompressed* baseline TIFF** v6 with Intel byte order, stored along with a copy of the file specification in a plain text file. A checksum should be generated for each file and stored separately so you can later confirm if the source had been modified/corrupted. All data should be periodically copied to fresh media (the period will depend on the recommendations for the media in use) and copied to appropriate new media formats as they are introduced. Ideally parity files should also be generated and stored to assist in the recovery of corrupt data, particularly if the archive is not duplicated.

* many applications use LZW compression for TIFF, the use of LZW was discouraged as it was a licenced, proprietary format covered by several patents, however these patents have now expired so the use of LZW is now considered acceptable, as long as you keep a copy of the LZW specification in a plain text file.

** It has recently been suggested that DNG may be an appropriate format, but archivists being a conservative bunch will probably want to see it in common use for at least another 20 years before they consider it mature enough

There are additional precautions such as producing a high resolution prints using stable pigments and archival paper and even keeping text prints of the hex content of the digital files along with full file and process specifications to recreate the digital files. But this is all way over the top for most (if not all) of us and does require access to a secure, environmentally controlled archival store in a geologically and politically stable region.

Cheers,
A

[ajohnw]

I would say that given the nature of Adobe's market dng is probably the safest option and where needed saving the processing steps that were used to produce the shot. Personally I would also keep a copy of the final full sized jpg. I don't really see odd camera raw formats disappearing though.

What is most likely at risk is 24bit colour formats such as jpg but given the number of them that are around what ever replaces them will probably be backwards compatible. If anyone is interested these 2 web pages give some interesting information about jpg's and something that may replace it. Jpg wars is one aspect. Also that it varies. I've read that windows 7 can support 48bit colour so maybe we will all have to to throw a monitors and video cards away when 8 or what ever arrives. Hard to be sure. There have been movie formats that produce better quality in smaller files that aren't used much. Largely down to companies wishing to patent one for their sole use. Fortunately the standards people usually sort it out in the end.

http://en.wikipedia.org/wiki/Jpeg

http://en.wikipedia.org/wiki/JPEG_XR

Well I find this sort of thing interesting

-
PS More reason for shooting jpg+raw. If the shot's hopeless for one reason or the other they can be sorted very quickly.
-

[ajohnw]

I've worked with a number of archivists who maintain archives of written and printed records and more recently of photographic materials. Some of the archives hold records that go back a 1000 years or more so the whole photography thing is still a bit modern and new fangled

Of course most of these archives are now recieving digital media for inclusion and the last time I checked the baseline recommendations from an archivists perspective were:

For the Master Archive: Uncompressed* baseline TIFF** v6 with Intel byte order, stored along with a copy of the file specification in a plain text file. A checksum should be generated for each file and stored separately so you can later confirm if the source had been modified/corrupted. All data should be periodically copied to fresh media (the period will depend on the recommendations for the media in use) and copied to appropriate new media formats as they are introduced. Ideally parity files should also be generated and stored to assist in the recovery of corrupt data, particularly if the archive is not duplicated.

* many applications use LZW compression for TIFF, the use of LZW was discouraged as it was a licenced, proprietary format covered by several patents, however these patents have now expired so the use of LZW is now considered acceptable, as long as you keep a copy of the LZW specification in a plain text file.

** It has recently been suggested that DNG may be an appropriate format, but archivists being a conservative bunch will probably want to see it in common use for at least another 20 years before they consider it mature enough

There are additional precautions such as producing a high resolution prints using stable pigments and archival paper and even keeping text prints of the hex content of the digital files along with full file and process specifications to recreate the digital files. But this is all way over the top for most (if not all) of us and does require access to a secure, environmentally controlled archival store in a geologically and politically stable region.

Cheers,
A

Interesting. The BBC in the UK lost lots of media on the basis that "modern" media would last a long time. The problem is that until it has failed there is no real way of knowing when it will fail really other than some form of simulation / guestimate. The various storage media appear so quickly and change so often that it's often impossible to know. 2 interesting ones are all over the place. E^2 and flash. Both have a read/write life but no one really knows how often it can be read. E^2 has been around for a long time but increasingly things use flash instead. E^2 was thought to have 10,000 read write cycles initially but later went up to 1,000,000 on some parts. Flash ?

-

[lenelg]

Unlike 1000 year old parchments, digital image files can easily be recopied to new storage media, and given how price/peformance has developed these media will have much more storage capacity than what I am using at the moment. So, from a physical integrity point of view, what makes most sense to me is to have multiple copies on quality hard disks, so I can easily transfer my image collection in one go (remember trying to keep track of backups on a large bunch of Compact Disks ?).

File formats do not disappear overnight, so if JPEGs, TIFFs or PNGs go out of fashion there will be a market for conversion software, and with ever increasing processing power, batch conversions should not be too much of a problem either (I would be most concerned about proprietary RAW formats from smaller manufacturers).

This, of course assumes a living, working image archive. Now, if we want to bury a time capsule, to be found after the next ice age, we have to face other issues..

[Simon Garrett]

Unlike 1000 year old parchments, digital image files can easily be recopied to new storage media

Well, parchment lasts 1000 years without having to be recopied every few years!

File formats do not disappear overnight

Compared to previous non-digital media, relatively speaking yes they do! But this isn't necessarily the problem. It's fine if you are aware of the need to recopy and reformat old media at about the time the existing format is going out of use. For our own personal use, we can see when existing formats and media are becoming obsolete, and copy to new formats.

But for archivists, there is a real problem. What do you do to recover old formats where the owners didn't reformat as the old forms became obsolete? Reading computer 30 year old computer files can be a bit of a problem.

It's not just whether recovery is possible, it's whether it's economic. For example, my wife (a museum curator) has access to some very old microfiche archives. There are readers for that format, but transcribing the information will be prohibitively expensive, and the risk is that the information will be lost for this reason.

[darkslide]

So sadly it all comes down to the money!

Seriously, I can image that this could be a huge headache for a museum - particularly as many 'older' archiving systems relied on photographic prints being made as records. My father did some work for the National Trust where he basically removed all the silver in the prints he had made and replaced it with (I have no idea what) something 'more' permanent. (And I thought the silver bit was pretty permanent - this coming from a 55 year old who still believes in the tooth fairy)

I can see my existing 'farm' of external disk drives is going to develop...