Best method for archiving image files?

[Alis]

Hi Everyone,

I just read a post on this forum that made me a little bit worried. I understand from it that saving image files on DVD is not safe. How do you guys archive your files. I have a lot of those DVDs and want to use a better medium if DVD is not a good option.

Thanks,

Sedali

[McQ]

I wish there were better options for archiving digital photos, but unfortunately DVD's and external hard drives are currently the two leading options. Blu-ray discs might become a better option once the prices drop considerably, but at the moment DVD's still give you by far the best GB/$ storage.

If you save to DVD, just make sure and also save parity information so that small, accumulating errors on the disc can be corrected if they become a problem. There's a number of programs which can do this; QuickPAR is the first that comes to mind. There's also quite a bit of disc longevity/accuracy information online regarding different DVD writers and discs. Some discs last substantially longer than others, so it pays do some careful research before choosing a brand for your photo archive.

It's also a good policy to re-backup your data from DVD/CD or otherwise to the latest backup medium every 5-10 years. I did this with all my CD backups, for example, since they were moved to DVD's. This ensures that the discs do not degrade too much, and for really long time periods, ensures that your data stays on a compatible medium.

If you go the hard drive route, one option is to purchase the (cheaper) internal hard drives and store them somewhere outside your computer. Hard drives are so large and cheap these days that this is becoming increasingly popular. When transferring a large amount of data it is a good idea to use software that verifies bit-for-bit accuracy of the copy. You would be surprised how many errors a USB connection can make with GB's of data and buggy drivers. These errors can accumulate quite a bit if your transfer information back and forth, so it's something to keep in mind.

[Colin Southern]

I have my suspicions that the reliability of DVDs is directly proportional to the quality of the DVD and how it's stored.

As an archival strategy for my commercial work ...

- I keep 1 copy on my post-processing PC, which is configured with dual 400GB drives in a RAID 0 (mirror configuration) (so the data is essentially on 2 hard drives at this point).

- Burn the DNG & processed files from the shoot to 2 DVDs; 1 DVD is hung in a vertical sleave inside a metal box that's kept in an even-temperature location in the studio, the second copy is brought home and put into 1 of a number of 12 CD wallets, and stored in one of my bedroom draws (dark + even temperature/humidity).

- My post-processing PC HDD is also backed up to an external HDD (held in the studio).

When burning DVDs I always select the "verify after burning" option, and to date "touch wood" have never had so much as even the smallest glitch in reading a DVD when I've accessed them for offline access to other less important material.

The ultimate solution for serious professionals is probably to use an LTO3 tape drive - but these cost several thousand dollars (I use them in the IT industry - so far they've proven to be 100% reliable).

Hope this helps!

Cheers,

Colin - pbase.com/cjsouthern

[Davey]

I keep backups on HDD as the price/size seems the cheapest option. Don't have that many dig photos yet but lot of other work that I keep backedup on HDD. Only have two 500Gb SATA in my machine but several internal PATA drives that I plug in every blue moon temporarily and fill with backups. Can get pata 500Gb WesternDigital for under £60 in UK, and SATA WD 500 Gb equivalent for around £67 so it's cheap enough, cheaper than good DVDs anyway.

The bigger HDD are not as reliable as smaller ones but it's not an issue if it's not in constant use and you only plugin every few weeks for an hour and you look after the drives. I tend to keep in fileformat that is recoverable too so it's easier to recover lost info. My win partition is in ntfs and usually backup in this but linux part I use xfs for most and ext3 for share part (no reliable xfs driver for windows I know of). I know most will automatically use a suitable system but sometimes it's easy to forget the file system you often use migth not be the best backup file system, if you're like me it's easy to forget anyway. This is especially important for none windows users.

XFS might be lightning fast but recovering lost data is not easy. I hear it's possible even though many think it isn't but the likelyhood of good recovery and the ease of doing it make it close to impossible (I've tried with poor results). EXT3 isn't much better data recovery wise and performance it's slow (only use for 2 way transfers between OS an I don't trust NTFS3g to write to my win partition). They are both journaling FS which ntfs and some other easy recoverable FS aren't but for backup drives this isn't necessary.

[Colin Southern]

Keep in mind also though that a good backup strategy needs to cover a range of risks - writing to multiple drives gives a good degree of protection against HDD failure, but to protect against the likes of fire / theft / natural disaster etc it's a good idea to keep at least one of the backups at a physically seperate location.

Cheers,

Colin - pbase.com/cjsouthern

[Alis]

Hi Everyone,

As a followup question, I bought a couple of hard disks to back up my files, video and images. It is a RAID system so while I am using it I can be sure that I have a back up copy at all times. I want to take the disc out (it is hot swappable) and replace them when the hard drive is full.

My question is, what is the risk of leaving a hard drive on a shelf for a long time. Would I lose data just by keeping it somewhere, assuming no accidents will happen? I do not want to keep buying these full system hard drives and want to sort of be able to recycle the skeleton and just change the hard drives.

It is designed for that but I am not sure if I will lose the data if say I keep it in a safe place for 10 years. Any thought on that?

Thanks.

[McQ]

If you leave a hard drive on the shelf for a long time I would be most concerned that the hard drive may not spin up again, as opposed to the data becoming degraded. This does not mean that your data is lost --- just that it might take quite some time to retrieve, and be reasonably costly to do so. You would usually have to send the drive off to a specialty data recovery company.

Definitely look into the best method and environment for storing a hard drive though in order to minimize any degradation of the platters themselves. This would mean somewhere cool, dry and sealed. Depredation is much less of a problem if you store parity information on the drive using something like QuickPAR (as discussed earlier). I also remember once reading that every few years or so it may be beneficial to rewrite the data to the drive as is. This would involve copying to another drive, then copying back to the backup drive and overwriting everything. The idea is that this ensures all platters stay well magnetized. With solid state hard drives it's the opposite though...

[Colin Southern]

"If you leave a hard drive on the shelf for a long time I would be most concerned that the hard drive may not spin up again, as opposed to the data becoming degraded. "

I've got a small collection of old hard disk drives - some would have heen sitting around for 5 years or more. On the occasions when I have powered them up I haven't yet had a problem, although with older drives that were powered off in computers for a few days (especially in winter) we used to occasionally have an issue with what we called "sticktion" - the cure was to simply rotate the drive rapidly around it's vertical axis (or even the whole PC case) which usually got things working again (although obviously time to change the drive).

Personally, I think the best backup strategy (regardless of the medium used) is to employ a lot of redundancy. A lot of people worry about media failure, and yet in many cases the dangers from accidental damage - fire - natural disaster - theft etc are probably just as great, if not greater.

If you have your data duplicated to at least a couple of DVDs (in different locations) and perhaps a 3rd copy on HDD then I think you'd be justified in feeling reasonably comfortable about the security of your data - and if you go a step further and perhaps check that data a few times a year then in my mind that would be cause for even greater peace of mind.

Cheers,

Colin - pbase.com/cjsouthern

[Alis]

Thanks. I think I got myself a full time job with this back up issue. The problem started when I bought the 5D MKII camera. Each RAW file is ~30 MG and no DVD can keep up with the volume of data that is generated every couple of days. I guess I have to go down to sRAW 1 or 2! Now I think film photography was much easier...

[Colin Southern]

At 30MB per image yopu should get around 150 per DVD (single layer) - not sure why you're averaging 30MB per shot though - mine from my 1Ds3 (same pixel dimensions) average low 20MB (20-23MB).

If you want to have a bit more time and a bit less money, give some thought to getting an LTO 3 tape drive (800GB Compressed (average), 400GB Uncompressed) - should be good for somewhere in the vicinity of 25,000 images per tape - and tapes are relatively inexpensive. I have a couple of clients using them (HP model) and they've been absolutely bullet proof - off memory I'm 99% certain that the quoted MTBF is 250,000 hours!

I don't think film days were really any better - we just didn't make backups of negatives - so one good natural disaster or fire or theft and the images were history.

Cheers,

Colin - pbase.com/cjsouthern

[Alis]

You are right about the file size, most of them are around 23-25 but I have a lot also that are around 28 MB.

I thought about getting the tape but the thought scares me. I just look at my self and I see the guy in the movie Fargo: one innocent mistake (buying the new camera) and now I am pulled deeper and deeper into the trouble. buying professional drives was not even in my wildest dreams. It just scares me!

Thanks for the adivce, now I feel more comfortable to look at the tape drive...

[Bill44]

Based on many discussions on this subject, there seems to be a consensus that Taiyo Yuden DVD are the nearest thing to an archival quality DVD. I have no connection to this brand so this is not a commercial.

[Colin Southern]

Thanks Bill (and welcome to the forums too by the way).

Just did a quick "google" and came up with some more info about them ...

http://en.wikipedia.org/wiki/Taiyo_Yuden

In particular if was interesting that they appear to make OEM product that's rebranded for many other "manufacturers" (I'm a Verbatum fan myself - one of the ones they manufacture for).

Cheers,

Colin - pbase.com/cjsouthern

[shreds]

With all the different formats for storage over the years, there is a constant battle to ensure compatibility. About four years ago I was recommended a tape drive with 400GB as being 'all you will ever need for a lifetime'.

I am currently looking at getting a couple of 1TB HDDs so that prior suggestion has quickly bitten the dust. Ok I suppose like a lot of people here, I am now regularly saving much larger files and as DSLR cameras head into 20 MP + territory and the high end medium formats are getting to 70MP+, the need for ever larger storage capacities goes on.

The one thing with all this is that prices of technology have fallen so 1TB drives are relatively cheap, and easy to utilise. I suppose the biggest thing is actually keeping up to date on backing up, as well as having a spare backup off site.

Just hope we dont have any problems with the electricity supplies!

[Colin Southern]

The good news is that LTO 4 is now out ... 800MB Uncompressed ...

http://en.wikipedia.org/wiki/Linear_Tape-Open

Cheers,

Colin - pbase.com/cjsouthern

[shreds]

But as I can nipdown to my local computer store and pick up a 1TB for GBP 70, I see this tape storage as running behind HDD and solid state storage. Cost of drive, then cost of tapes too. Also the need to avoid magnetic sources that can wipe such tapes, and tapes snagging, mean I wouldnt go anywhere near these drives.

Visions of the old Sony Betamax?

[Alis]

And hopefully this will be available soon:

http://pioneer.jp/press-e/2008/0707-1.html

Multilayer Blue-ray megadisc from Pioneer.

[Colin Southern]

I see this tape storage as running behind HDD and solid state storage. Cost of drive, then cost of tapes too. Also the need to avoid magnetic sources that can wipe such tapes, and tapes snagging, mean I wouldnt go anywhere near these drives.

No no no no no no no no no. They're an entirely different strategy to HDDs. HDDs are great for online access - they're fast and allow for random access to data, but they're relatively vulnerable and not cost-effective when you start having to maintain multiple copies of large amounts of data.

In NZ Dollars the drive, SCSI controller, Cable and Terminator cost around $3500 (entry level, but good) - so probably around $2000 USD - but once you've got them, you've got them - you couldn't wear them out if you tried. Literally.

Once you have the setup, tapes are cheap - a fraction the cost of HDDs - a cost saving that multiples when you consider that for safety you need multiple copies.

With regard to "Magnetic Sources", you need to avoid strong magnetic sources, but in my experience there's no need to go overboard - I wouldn't place them on the magnet of a speaker for example, but putting them on a speaker box containing that speaker is unlikely to bother them. In reality the sensible thing would be to simply store most of them in a drawer or on a shelf.

The cartridges are all self-contained - I've yet to see even one malfunction.

In terms of reliability, I'd trust the tape for long-term storage over HDDs anyday - in my "other life" as an IT consultant I replace several failed HDDs every year - I've yet to replace a single LTO tape cartridge (or drive). These tapes are "enterprise grade" - I backup multi-million dollar companies on them, and sleep well at night (I've never had so much as a single read error) - I use USB External HDDs for backing up my own data (and for some of my clients) - but we're only in the minor leagues. Tape cartridges have no electronics to fail - no heads to crash - no firmware to get screwed up - they're not like the old travan drives that proved to be fairly unreliable, especially in dirty environments.

I'm not trying to "poo poo" HDDs as a backup solution - as I say, I use them myself - but if you get to the point where you have a lot of data, chances are you won't back it up to multiple sources on a regular basis with HDDs - this is where tape drives come into their own. I'm just trying to give you a bit of an insite / positioning statement about them - they're a highly reliable / cost-effective / very fast technology - and amongst larger corporations you'll find that they're used almost exclusively.

Cheers,

Colin - pbase.com/cjsouthern

[Colin Southern]

"
And hopefully this will be available soon:

http://pioneer.jp/press-e/2008/0707-1.html

Multilayer Blue-ray megadisc from Pioneer. "

400GB of data on a single disk - now THAT's scary!

Seriously, for those who handle disks with kid gloves it might work well, but judging by the condition of most re-writeable CDs I see, I think (despite my best efforts) most of my clients throw the CD like a frizby from the other side of the room when inserting it into the drive.

Cheers,

Colin - pbase.com/cjsouthern

[shreds]

In NZ Dollars the drive, SCSI controller, Cable and Terminator cost around $3500 (entry level, but good) - so probably around $2000 USD - but once you've got them, you've got them - you couldn't wear them out if you tried.

But thats the point, most people here are not global multinationals where cost is not that important, the combination of cost effectiveness at an everyday scale combined with longevity is the key.