New Tutorial Added: Archival Digital Photo Backups

[McQ]

This is currently only accessible via direct link (and not from the main page),
but I wanted to first make it available here for feedback:

Tutorial: Archival Digital Photo Backup

Is there anything you think I've left out? Something you feel differently about? Spot any typos? Let me know your thoughts.

PS: I will move this thread to the "Site Suggestions & Feedback" forum later, depending on the discussion. For additional reading you might also want to checkout an earlier thread on this topic: Best method for archiving image files?

EDIT: it's now accessible from the main tutorials page

[Dave Humphries]

Hi Sean,

Looks good to me, everything I thought of was covered just a few sentences further on.

Only thing that occurs to me now is if the person expired unexpectedly, would family or friends know (or appreciate/care) about where to find the images, so put it in your will?

Thanks,

[McQ]

Only thing that occurs to me now is if the person expired unexpectedly, would family or friends know (or appreciate/care) about where to find the images, so put it in your will?

That's a good point. On the other hand, all the worrying this article might cause about photo longevity is likely enough on its own -- without also worrying about one's own longevity

[Alis]

Hi Sean,

It is very well written and thorough. The only thing I could think of is the actual workflow, from camera memory media to the archive. It took me a while to come up with a reasonable one.

Alis

[McQ]

Alis: Thanks for the feedback. Are you suggesting to have an "in a nutshell" summary somewhere towards the end?

[Alis]

Alis: Thanks for the feedback. Are you suggesting to have an "in a nutshell" summary somewhere towards the end?

Something like that. Copying from the card, cleaning the database, naming, sorting, keywords in the metadata, and the actual back up at the end, softwares that support this process and make it easier.

Or even just a general recommendation that there should be a regular workflow that works best for each individual and that they should stick to it to prevent having duplicate files or accidental loss before back up etc.

[Colin Southern]

Hi Sean,

Just skim reading at the moment (short of time), but I did pickup "tremendouns".

[McQ]

Just skim reading at the moment (short of time), but I did pickup "tremendouns".

thanks, fixed

[David]

Hi Sean - A sub-heading reads "Choosing a Photo Backup Media". I think you need to drop the "a" as "media" is plural. I may have missed it, but would the prudent archiver also backup the backup?

Otherwise, excellent.

Cheers

David

[McQ]

Something like that. Copying from the card, cleaning the database, naming, sorting, keywords in the metadata, and the actual back up at the end, softwares that support this process and make it easier.

Or even just a general recommendation that there should be a regular workflow that works best for each individual and that they should stick to it to prevent having duplicate files or accidental loss before back up etc.

I've added a summary section at the end, as it applies to two categories of photographers: casual and discerning. Hopefully this ties everything together. Only thing I need to add is a little more on naming & other organizational issues.

[McQ]

I may have missed it, but would the prudent archiver also backup the backup?

Thanks for the feedback. A little is discussed on making multiple copies and storing these in separate physical locations, but perhaps it's not as prominent as it could be.

[dlj]

Really well done piece, and an important message for all.

Personally, I would not rely on DNG for longevity (yet: give it about another 5-10 years to see its adoption rate). I've been burned so many times in the past by relying on corporate promises, and the likes of Microsoft and Adobe in their good corporate citizenship roles. The only thing that seems to really matter is the scale of acceptance; PDF/A is just beginning to see some real consideration and adoption for document imaging, but the vast majority of usage even today still sides with TIFF. Yes, there are far better technologies, but the only real 'winner' is the file format that can be opened in many years. When there is a large enough user base for a given format, it makes economic sense for development to occur to accommodate that base.

For the safest longevity, I'd agree with everything you recommended, but put more stock at this point in TIFF16-bit (and ensuring the TIFF compression methods are 'safe' too - don't forget that a valid TIFF file could use JPEG compression; the patent for LZW has fortunately expired, and so is 'free' for all intensive purposes) for RAW data. For JPEG images, just media migration should be enough.

I like the idea of a sample workflow proposal at the end. Something like labelling the date the volume was closed/finished, and an annual or semi-annual check for any media that needs to be migrated. Maybe suggestions like using a long-term friend's or relative's home - and perhaps exchanging 'ready' volumes when visiting each other for holidays and such. Offsite storage doesn't have to be too scary or costly, but I would advise against offsite storage that remains in close proximity: hurricanes, floods and earthquakes, and even political or racial actions can impact homes for dozens of miles. There should be a local backup, and an offsite copy, for convenience and redundancy. However, if the entire process is too onerous, users will just not bother period, and so its better to tolerate some flaws in the process, for a process that's at least followed.

I'm amazed at the scope you covered in just one article, without dragging on for 25 pages: truly impressive, and concise enough to keep an average reader's attention. Wonderful to see.

[Dave Humphries]

Hi Sean,

I have become too used to your high standards and should like to echo dlj's final paragraph.

Bump time

generally takes snapshots to record events and other get together's

Is that a grocer's apostrophe at the end? Or am I exposing my 'little knowledge is a dangerous thing' in respect of English grammar?

The other thing is that I, and certainly casual photographers, will need help with "ideally SFV or MD5 checksum files", whilst I know what a checksum is, I have no idea how I'd go about this for any file, let alone a jpg.

Cheers,

[McQ]

Thanks for all the additional feedback.

In the file integrity section, I've added a technical note and a sentence on software programs for MD5 and SFV:

This way you do not need to worry about complicated RAID or parity files, but you will still need to store SFV or MD5 checksum files** along with each archived photo. There are far too many programs that can read or create SFV and MD5 files to list here; a quick search engine query will yield several free options.

**Technical Notes: A checksum is a digital fingerprint that verifies the integrity/identity of a file. SFV stands for "single file verify", and contains a list of checksums corresponding to a list of files. MD5 checksums were created to not just verify the integrity of a file, but to also verify its authenticity (that no person had intentionally modified a file). CRC checksums are much quicker to calculate than their equivalent MD5 checksum files, but MD5 checksums are more sensitive to file changes. There are other checksum file types available, but SFV and MD5 are currently the most widely supported.

I'd like to avoid delving into this topic so much as it's much more on the IT side of things. Unless there's a program or two that stand above the rest, I've also tried to keep from recommending specific software in the tutorials to keep them less commercial.

[d3debian]

RAID 1,5,10 PREVENTION A RAID 1, 5 or 10 is an array of disk drives with fault protection in case one of your drives fail. These can continue to operate even if a drive files, without losing any information. However, they can also substantially increase costs since they require additional disk drives and a RAID controller.

Typo in frame?
Is it files or failes?

Lets ignore possible future compability-problems with extractors: do you recommend not to compress files in an archive like .tar .zip etc.?

[Dave Humphries]

Lets ignore possible future compability-problems with extractors: do you recommend not to compress files in an archive like .tar .zip etc.?

Fails, I think.

Good question;
Little point zipping a jpg, but I'm not so sure on tif (non-compressed) or RAW, might be, let's see what Sean says.

Cheers,

[McQ]

Thanks, fixed the fails->files mistype.

Compressing your images within a RAR, ZIP or some other format adds another layer of necessary future software compatibility. However, ZIP and TAR files have been around a really long time. In fact, ZIP was created 6 years before JPEG. I'd find it hard to believe they wouldn't be readable on a 10 year timeframe, but beyond that it's anyone's guess.

Now...to the issue of whether to compress in the first place. A TIFF with in-built LZW compression is the most compatible, but it often makes the file size larger for photographic material. A TIFF with in-built ZIP is much smaller, but even right now it's much less compatible. My understanding is that most camera manufacturer's RAW files already employ some sort of compression. Overall though, you really won't gain that much space by compressing photo files. Plus, if a file does get corrupt, I'm not sure if recovering it will be as simple in its compressed format. With a TIFF or other image file, often times you can clone out the problem parts if it's not too bad. With a compressed file it may or may not be readable. To me it comes down to a risk-benefit analysis, and the slight benefit in terms of space savings just doesn't seem worth it...

On the other hand, one big benefit to compressing your files is that it's very easy to add parity/recovery information. WinRAR has that feature built in. I'd prefer to keep my parity files separate from the images though.

[Blazing fire]

Thanks for the great tutorial! Concise, apt, accurate, informative, comprehensive and well-organized.

I'd like to avoid delving into this topic so much as it's much more on the IT side of things. Unless there's a program or two that stand above the rest, I've also tried to keep from recommending specific software in the tutorials to keep them less commercial.

I agree that the tutorial should focus primarily on backups. However, I think it's a good idea to link the reader to relevant articles too, just to make it more exhaustive.

Raid and HDD related stuff:

1. "Storage: RAID Fast, reliable or both; FAQ and How-to" by Tim of extremeoverclocking.com. Good explaination of how to setup a RAID system.
2. "Get to know RAID" by ACNC company. Complete with diagram and explains almost all RAID levels. Raid 0,1, 5, 10 (1+0), 50 (5+0) etc
3. "What is raid" by staff.uni-mainz. Good explaination of diffferent RAID levels.
4. "RAID And You (A Guide To RAID-0/1/5/6/xx)" by Serra of xtremesystems.org. One of the more comprehensive explaination.
5. "Applications and Resources for Bit Error Recovery in Stored Data" by Speederlander of xtremesystems.org
6. "Self-Monitoring Analysis and Reporting Technology (SMART)" by storagereview.com


Checksum files for verification

I currently use a checksum program called "JSummer" by "zero-sys.net-". It is free and unlike most free program, this one the user to create a checksum file of an entire folder, not just the individual files.

There are some quirks in this program though.

During installation:

1. It does not allow me to disable the creation of a desktop and start menu shortcut.
2. It does not allow me to chose where to install it. It installs itself into the program file under the name "ZeRo-Sys.Net" by default.

Few things that the developer might wish to add:

1. Regular checking of checksum files, much like how anti-virus softwares have scheduled scans.
2. Option to disable auto-scrolling.
3. Definition of terms (glossary) for idiots like me

JSummer: http://www.zero-sys.net/portal/JSummer.html

You might wish to add this to the tutorial.

You may want to touch a bit about SMART (great exhausive site on smart) too. Just 50 words or so on what it does and how it can inform the user of errors. You may also want to talk a bit on prolonging the life of a mechanical HDD; minimise vibration, keep it cool and good DC output from PSU.

Smart:

"How does the smart hard drive warning work?" by Ken DiPietro

Using S.M.A.R.T.

Most computer users are only exposed to S.M.A.R.T. warnings during the boot-up process when they see an error that reads "Warning, SMART has detected that a hard drive failure may be imminent. Please back up your data immediately. Hit F1 to continue." However, there are several software programs that allow Windows users to get a far clearer picture of their hard drive's health, including Advanced SmartCheck and S.M.A.R.T. Monitoring Tools.

[McQ]

Wow, lots to talk about here. Thanks for the in-depth reply with additional links.

SMART Hard Drive Parameters:

The largest current customer of hard drives (Google) did a large-scale study a couple of years back on SMART parameters and how these correlate with failures. While SMART had a lot of promise initially, and can be quite helpful for large-scale statistical predictions, they found that it was very difficult to use it for predicting individual drive failures. However, individual drive behavior is what would be needed for personal photographic use. I've attached the full report as a PDF for everyone's reference...

Google found something else interesting: keeping a drive extra cool doesn't help its longevity! They showed that the failure rates follow a "bathtub distribution" with temperature; in other words, for low and very high temperatures, failure rates were highest, but for a middle range of temperatures the failure rates were similar and optimal.

Checksum files for verification:

While I try to avoid recommending particular pieces of software in tutorials, I don't have an issue doing so here in the forums. For MD5 files I use a program called FastSum, which I've found to be extremely simple to use for large batches of directories. It's not free, but very inexpensive last I checked -- a fraction the cost of an external hard drive. However, there's LOTS of good programs out there; this is just the one I happen to be using at the moment. However, since parity (and checksum) files are likely not familiar to 95% or more of photographers, it might be a good idea for me to provide a range of options in a footer within the tutorial...

I will take a look at JSummer, it looks promising. Everything else being equal, a free program always wins out over a program that costs something...

Parity files for repair:

Usenet groups created the open source PAR & PAR2 file formats, which seem to have become the leading standard. I use a program called QuickPAR to create these in batches for photos.

RAID:

This is a *whole other topic* unto itself. Lots and lots to read about on that one. Since RAID is technically not a backup, and I don't want to give the impression that it is, I am hesitant to shift too much focus towards it. I couple subtle links for further reading might not hurt though...just need to think about it. I was even considering not including RAID at all, because if you perform regular backups hard drive failures should never be an issue. After all, if the instructions appear too complicated then people won't follow the backup regimen at all. I think all of the additional information is really helpful though, and would consider focusing a lot more on it in a future article about "computer setups for photography" or something like that...

[Steaphany]

I originally emailed this to Sean who requested that I post it verbatim here:

I would like to make a suggestion which could be an addendum to your tutorial "ARCHIVAL DIGITAL PHOTO BACKUP", which could allow data storage and recovery with a > 100 Year life time, Store the Digital Photographic Data On Paper.

Over the 33 years of my electronics and software career, I have noted many data archive programs, but a technology that never seems to go main stream, which could easily provide the data longevity needed by photographers is an application which formats a file into a scan-able data pattern that is printed on paper. It is well accepted that paper media has proven to survive through out the ages while technologically based data storage's life is based on the life of the technology. The key to achieve > 100 Year data life is to take the technology out of the equation.

There is a freeware program which was developed for the Windows platform, which has the full source code available capable of this exact method of paper media data storage, PaperBack:

http://ollydbg.de/Paperbak/

As the developer clearly states, this program can easily be ported to other operating systems, so computer platform dependence is eliminated.

Now, a point that I feel needs to be made clear. Not all methods of applying ink to paper are equal. I have three ring binders of technical manuals that are over 10 years old. The content of these manuals was reproduced by high quality Xerox copiers which thermally fuse powdered toner to the paper, the very same technology utilized by laser printers. The problem is that time and temperature cycling has caused the toner to begin fusing to the facing page. If I open one of these books, each page is now a mess of what was intended merged with a mirror image from the facing page. It's readable - some what, but it's obvious that whole pieces of characters completely flaked off to the opposing page.

For this paper media data storage method to really last, the pages must be printed with an fade resistant absorbed ink into a acid free fine grained paper. This will ensure a clear image and long media life.

Now, a bit of additional information needs to be stored with photographs archived on paper, the source code for PaperBack in a simple font. If technology changes to the point were the PaperBack program is no longer available or capable of running, this source code print out would save the day. If you really feel it necessary, also include a print out of the image file format specification. That way, if TIFF or JPG are lost and long forgotten, someone could write a file reader/converter if needed.

Here is a possible data recovery scenario: 500 years in the future, an archeologist locates and opens a vault containing stacks of paper. They find the PaperBack source, the Tiff format specification, and collections of pages containing TIFF formatted data files. They scan and OCR the source code, modify it to suite the peripheral technology available, and get a runnable program. They also write a program to read the TIFF format and display the image. The collections of paper media data pages are then scanned and processed into what ever technology they have, yielding recovered 500 year old photographs, with the very same quality that existed when the data pages were printed.