Archiving your original files - which format?

[Manfred M]

It's called "digital rot", i.e. with the rapid development of new technology, the equipment to read the older technology beomes "unavailable".

Personally, my money would be on both jpeg and tiff formats. These are fairly widely used standards. While I see some advantages of DNG, the real problem is the fact that none of the major players have adopted it, it is hard to say how long it will be around and maintainable. Of course, there appear to be some potential quality issues as well.

I have little faith in the longevity of the RAW format, or even the backwards compatability of PSD. There is no guarantee that Adobe, Canon, Nikon, et al will support backward compatibility down the road.

[revi]

There is always dcraw (until now at least), which is available in source code form (C language), and handles a lot of RAW formats.
So as a safeguard, you could store a copy of the dcraw source with your backups. Source code files are text files, so no fancy
programs needed to read them.

Do check that your raw files can be read before trusting this, of course.

Remco

[darkslide]

Salut et merci - pas mal comme idée!

This is in fact a very reasonable way of doing it - just put a copy of the source code somewhere on the backup drives - good idea.

Thank you for that

[Simon Garrett]

There is always dcraw (until now at least), which is available in source code form (C language), and handles a lot of RAW formats.
So as a safeguard, you could store a copy of the dcraw source with your backups. Source code files are text files, so no fancy
programs needed to read them.

Do check that your raw files can be read before trusting this, of course.

Remco

dcraw is written in standard C. Would you know how to make that run on Windows 7? If so, you would be in the minority. And in 30 years time, when we're all using strange new devices with minimal (if any) support for the original C library, and everything would have to be re-written... Yes, in theory it could be done. But it would be way outside the capability of most users. There's bound to be someone who can do it for you - at a price.

You're quite right, it is some degree of safeguard to have a program in open source C, but don't think of it as a trouble-free long-term archive solution.

[ajohnw]

dcraw is written in standard C. Would you know how to make that run on Windows 7? If so, you would be in the minority. And in 30 years time, when we're all using strange new devices with minimal (if any) support for the original C library, and everything would have to be re-written... Yes, in theory it could be done. But it would be way outside the capability of most users. There's bound to be someone who can do it for you - at a price.

You're quite right, it is some degree of safeguard to have a program in open source C, but don't think of it as a trouble-free long-term archive solution.

Few people use dcraw directly. Ufraw is a graphical interface for it. Both run on linux,unix,windows and mac. Also OS I believe. The dcraw web page also has a list of what software packages use it. This includes Adobe and many others. There will also be some that have just copied it and maybe made minor changes to hide the fact. One day the maintainers of both packages will stop maintaining them for one reason or the other including death. Generally some one else takes over but there is no guarantee. That's why I suggested dng for archiving along with it's processing pipe if needed. That will exist as long as Adobe does and if they switch to something else they are bound to offer converters. If some one else comes along and takes over Adobe's market they are bound to offer converters. Frankly though if there are changes I suspect that they will be similar to canons cfw, cr1 etc they will be raw files with different internal layouts and compressions. Some one or the other is bound to offer conversion routines if they are needed. Substantial changes usually only happen when new technology is introduced.

As far as Ufraw is concerned even Microsoft offer the ability to run programs in a mode suitable for their earlier operating systems and still offers a dos box if some one did want to run dcraw directly. There are tutorials on the web but if it's installed separately rather than as part of another package just typing dcraw at the command prompt will list all of it's options. If ufraw is installed dcraw is likely to exist as well as a separate entity. It does on Linux,Unix and probably PC based mac's (modified Linux). Pass on windows.

Written in C. It doesn't really matter what they are written in. Windows applications are usually available as executables and if anything else is needed on the machine it will be mentioned. On linux the distributions software management will provide most things and install them directly. Compiling is relatively simple. The files will be compressed and contain readme and install text files that state exactly what needs to be done and what is needed. On linux for instance the commands ../configure, make, make install and maybe a make clean to remove work files are all that is generally needed. There are add ons to make it even simpler than that and if needed the readme or install files will mention them. Installing a compiler is a little different as there isn't one available to compile it but executables are available.

The comment relating to C in the post I quoted are not really relevant. Even microsoft use a variety of languages to produce their software. This is likely to include assembler, c and c++ and in all probability variations they keep to themselves. I'm reminded of a microsoft video on using c++ in visual studio to produce Word where they made it clear that they didn't use C++ at all. At one stage people thought they used Pascal. Interesting language as it sits on P code which is a very terse efficient language. They also have their own form of P code used to interface C/C++ applications to the software that actually drives the machine. For my sins I have spent maybe 18 years on and off writing software of one sort or another. It figures heavily in all sorts of products these days. Knowing something about it was my reason for switching to Linux.

-

[Simon Garrett]

ajohnw, I'm afraid I don't share your optimism.

You are already demonstrating that you know much more (about computer software) than the average photographer. For most people, the ability to retrieve 30-year-old computer files on obsolete media in obsolete formats, even with source code software available, is in practice very limited - or expensive.

I'm quite sure it will be possible to run dcraw and read current raw files in 30 years time, and for an enthusiast looking after their own photos it might be practical. But when a grand-child is looking over some very old photo files from grandad on unreadable media, they'll think of the difficulty and maybe won't bother.

As I mentioned earlier in this thread, my wife is a museum curator faced with the difficulty in transcribing microfiche information from obsolete formats. Possible: yes. Practical, within the resource limits of the museum, maybe not. These are microfiche copies of local newspapers from around 50 years ago, which she currently can't access. The same newspaper from 100 years ago is held on paper; she can access that!

We are beginning to face the unexpected problem that in some ways it's easier to manage records from 100 years ago than from 30. Hence the importance of standard formats (for images that means TIF or JPEG) rather than rely on being able to run current software on future systems. And for storage medium, who knows? Hope that we can keep it online, and that storage capacity rises with our need for it?

[revi]

Well, are there any other solutions than storing either program code or descriptions with the data?

And lets keep in mind that there are two problems here, that are quite separate:
1- Can we retrieve the data (as in, can we read the disks it's written on)
2- can we use the data (as in, produce a visible image from the retrieved data).

(@Simon: What you describe about your wife's problems seems to concern data retrieval. And there's no way you can
prevent that with storing extra information with your images, as that would also be unreadable.)

I suggested storing the dcraw code as that gives at least a possibility to retrieve the data from RAW files,
without going through DNG/TIFF/etc. (which are not the original data, so I'd also be quite reluctant to use such
formats as the only archived data). I did not intend to suggest that it is the solution for the archiving problem.

On-line storage sounds good, but has its own problems. Wasn't there a case recently of users losing their data through
the close-down of a storage service (MediaFire)? For the moment, I see on-line storage at best as a second off-site
back-up, and not the most secure one at that, as you as user have no control over what's going to be done to (or by!)
the service you use.

Finally, I'm not sure that full-scale bullet- and earth-quake proof archiving is even relevant to the grand-child who wants
to see his grand-parents' photos. For that case, tiff/png/jpeg is more than adequate (if there's still a way to read the
media!). A paper album might be even more appropriate for this case, as it allows easier access, and can have additional
information about the context of the images.

The situation is different for 'formal' archives (musea, government, etc,), where at least one might expect to have a
certain level of expertise available to maintain and access the archives (yes, I'm an optimist). So for them it's a lot
easier to revive data from a provided description (in any form) as long as they can read the media, which seems
to be more of a problem atm.

[Simon Garrett]

I agree with all the factors identified, I think I'm only giving different weight to some of them.

revi, agreed my wife's problem is the medium, but comparing digital with analogue is rather limited so it's a poor comparison.

My own view: I strongly expect that I'll be able to read my raw files as long as I live, and probably my son too (mind you, he's a software scientist!). However, as an extra safeguard I also archive high-quality jpegs of my processed images.

Storage is cheap, and doesn't have to be online (provided one guards against fire, theft and so on with multiple copies in multiple locations), and so why not do both:

• Keep the raw, and rely on being able to read it in the future, if only with dcraw
• Keep a copy in standard format (tif/jpeg)

As you say, the storage medium also has a potential issue of obsolescence.

[ajohnw]

That's a good point about "paper" Simon and for the sort of photo archiving you mean I suspect prints are the best option still. People do still use them. I wonder how many people have wedding photo's etc purely in electronic format. My comments were aimed at archiving for photographers not for people who just want to look at old photo's.

You have hit on the real problem though. It's not the format that the information is kept in it's the media it's stored on. I've used some interesting ones. Paper tape, punch cards and 8in floppy discs. Magnetic tape too, that is still used. I don't think media changes make much odds to people who are interested in raw files or future developments of them as the files can be moved to the new media while the facilities are available. DVD's for instance support earlier formats even CD's. One day they may not. I've mostly worked on what would be called firmware. Archives in one instance was 5 1/4 floppies in two fire proof safes. An old machine was kept around for ages to allow them to read but eventually they had to be transferred to something else. Smaller floppies. Archiving like that in my view on magnetic media is a bit farcical without some form of error correction built in and is dubious even then. There is something more reassuring about some aspects of optical media,

On the great great grand children etc times change. My son is 22 now. he could make some use of a windows PC back in the days of windows 3.1 and even some dos educational software. He could bring a PC in a shop to it's knees at around the age of 3 or 4. If I took him to a shop he took great pleasure in making them lock up. He did that all by himself. All I ever did is show him what PC bits and pieces do. These days kids are what might be called PC aware very early in life. In any case if there is need for browse old photo software some one is bound to produce it. Probably with facilities to improve is like scratch removal and noise reduction, companding etc on old lp's.

Dcraw. Whey when Ufraw is the same thing driven by via a window. The missing facilities are unlikely to be needed. I have used it to obtain grey scale actual sensor images but it can be done in a window with imageJ. It uses dcraw too.In case anyone is curious ImageJ is a very well respected image processing package. Not one for most people though including me.

Sorry that some aspects of this have been covered since I started typing this. I often have that problem as I do other things as well. Eat dinner, make tea, maybe even watch a bit of telly I don't want to miss.
-