To RAID or Not?

[TheBigE]

I have been thinking more about photo storage and seeing if I should begin looking at different methods for working on, storing and archiving my photos.

It has me wondering would anyone, if they could start over again, start with a RAID? Do you wish your storage workflow was different? I am very much a Hobbyist Photographer and have 7000 images in my LR catalog after 2 years.

Currently, I import my photos into my computer work on them in LR and when I am done send them out to WD Passport 2 TB drive and then copy those photos over too my Time Capsule 3 TB drive. At that point I reformat my SD cards in my camera. In all steps along the way I always have two copies of my photos (SD Card, computer, WD/TIme Capsule) I favor portablity overall and like to have my photos on my computer (MacBook Pro) to edit this offers me good speed in loading photos. Setting up anything more than a WACOM tablet to do photo editing to me may just be enough to slow me down.

I am not at the limits of my storage by any means, just looking long term and maybe my next investment should be a DROBO (or something similiar). I welcome anyones thoughts on what they wish they would have done different in this area.

Finally, and Yes I am very aware that I lack the "offsite" backup strategy as of yet. That is one item in which I am trying to come up with a reasonable answer.

Thanks
Erik

[Manfred M]

I am a firm believer in RAID. I have dual RAID units that have copies of all my images on these drives located at two different locations in my house. Number of images is significantly higher than what you have - probably pushing well over 50000 by now.

The reason I believe in RAID is that I have had individual hard drives go bad and with RAID the recovery is easy; just put in another drive and the system takes care of the rest. I've even had two individual hard disks go within a few weeks of one another (both on warranty). so by quickly reacting to replacing the bad drives, I averted a situation where I could have lost all the data on one of my RAID drives.

My offsite is on DropBox and I have upgraded my internet to be able to get uploads there fairly quickly. I only store processed jpegs there (to limit space requirements), so these are just the best of the best images.

[James G]

Hi Erik, I have a suspicion that your post is going to start an interesting technical discussion over the numerous backup/security strategies that are available.

My process is basically the same as yours at present. All my 'RAW' shoots older than 12 months are on backup up to 4 x 2Tb which in turn are duplicated to 3 x 3 Tb offline drives. This is my 'Master Archive'

Additionally all processed images are backed up in separate folders on a duplex basis to the same offline disk store.

I keep the last 24 months of current shoot, and all processed images on my desktop and all the image/data files are copied monthly to 2x 2 TB portable 'rugged' drives. These comprise my 'Current Archive;

On a quarterly basis (when I remember), the Current Archive is merged with the Master Archive.

On an annual basis I remove shoots more than 2 years old from my Desktop.

I've often considered setting up a RAID system, but never get around to doing more than think about it. (This is despite a long career in IT!)

My backup process kind of grew 'organically' as my image count grew, but since I've never considered it as more than my passion, and not something that my relatives or others are likely to be greatly interested in after my day is done, I've not felt the need for more security. I very much believe that nothing is 'forever' so am quite philosophical about the possibility that it could all be 'lost' for any number of reasons! I suspect that as long as I can get another camera, I'll just carry on shooting

[DanK]

My process is much simpler, but I think I will need something different once I have a larger collection of photos.

Currently, my entire collection is on a single large internal drive. My first backup is an external drive. I use a synchronization program rather than a backup program with this. As soon as I upload images, I back them up to that drive. Then I format my card(s). If I had a hard drive crash, restoring my photos would require only opening my synchronization software and dragging the top-level folder to the new drive (and waiting....)

My offsite backup is Crashplan. It costs me $59/year, I think, for unlimited storage. The initial backup is slow if you have a of photos, but after that, you can pretty much forget about it. it just chugs along in the background.

The only complication is that if I want to cull images after uploading, I have to turn Crashplan off because I can't delete images while it is accessing them.

[Shadowman]

I am a firm believer in RAID. I have dual RAID units that have copies of all my images on these drives located at two different locations in my house. Number of images is significantly higher than what you have - probably pushing well over 50000 by now.

The reason I believe in RAID is that I have had individual hard drives go bad and with RAID the recovery is easy; just put in another drive and the system takes care of the rest. I've even had two individual hard disks go within a few weeks of one another (both on warranty). so by quickly reacting to replacing the bad drives, I averted a situation where I could have lost all the data on one of my RAID drives.

My offsite is on DropBox and I have upgraded my internet to be able to get uploads there fairly quickly. I only store processed jpegs there (to limit space requirements), so these are just the best of the best images.

Manfred,

You stated you have dual RAID units, how are these connected? I know the stand alone unit would be connected through your router, how is the second unit recognized? Is it piggybacked on the first unit (through USB) or on another data port?

[William W]

. . . It has me wondering would anyone, if they could start over again, start with a RAID? Do you wish your storage workflow was different? I am very much a Hobbyist Photographer and have 7000 images in my LR catalog after 2 years. . .

Yes. If I were starting over I would be much more RUTHLESS in the first and second cull and as a result there would be far FAR fewer image files, negatives and transparencies to catalogue and subsequently store: slowly I am catching up, being about three quarter the way through the transparencies - I am up to 1999. I shed all my (business) Wedding and Portrait negatives and transparencies in 2004/5 and slowly adopted a different work flow for any professional work whereby essentially I only keep files/negatives/transparencies for a specific period of time (noted in the contract). Any teaching material I save to good resolution/size JPEG (usually two sizes one for printing and one web display/email) and I ditch the rest.

For my hobbyist captures, for example family, fun, experiments and holidays etc. I cull severely: the predicate being either good enough for the slide show presentation or good enough to give to family or friend - once the image files are given or on one of the slide show USB sticks - one copy of the original and the final edit is kept and catalogued.

With the advent of digital I have become more inclined to use exposure bracketing for (especially) landscape, architectural and opportunity (portraiture) shots when I am travelling or just walking around (as I take a camera almost everywhere), and (with humility) even though I tend to bracket usually plus and minus two thirds - I do usually use the middle exposure (the one I calculated anyway) but because digital is so inexpensive, I find this technique is good for me because frankly when under the pressure of time to grab a shot I can think less about the nuances of exposure and more about other elements of the shot: maybe that's lazy but I like to think of it as creatively using the technology so I can leverage more creativity TIME into any one walk, holiday family gathering etc.

So for example in 2013, driving for eight weeks through Italy and France, I pulled about 8000 shots, but probably about 6000 were brackets, so really that is only a total of 4000 discrete shots - about 500 per week / about 70 per day - being awake for about 14 hours per day, that's an average of 5 per hour - that's reasonably active considering (at the time) I really "wanted every single shot" - and I was driving eating and etc . . . in those 14 hours each day.

Then in the light of calm and purposeful thought, about 4 weeks after arriving home, I began culling to address my OUTPUTS and basically there were three, I didn't actually count but these are indicative numbers of what was kept:

> The slideshow USBs (300~350)
> Teaching & Reference (50)
> Family Album / Personal images (80)

So at maximum, about 500 images kept (probably closer to 400) after 8 weeks "work" and over 8000 shots pulled.

Statistics can be funny animals - we could look at those and interpret 400/8000 = "6% success rate totally mad photographer he doesn't have a clue what he is doing" - or on he other hand once reading through the explanation and the procedure, we might think "ah, he has a broad FINAL VISION at the outset and wants lots of choices and with a safety net for later fine tuned choices."

*

Entirely different hat goes on my head when shooting for money - for a commissioned portraiture for example, I probably will pull about 30 to 50 shots over a one to two hour session - each one with the aim of being "perfect" - and then from those (assuming 2 or 3 with silly eyes etc) the 27 to 47 are shown to the client for choice of "mood" "feel" i.e. the "best ones which exhibit the "personal stuff" that they have described that want from the session - and we work on those for PP - so maybe 4 or 10 are finally chosen as the Client's choice - the rest are ditched and after delivery I keep the 4 or 10 files for a period of time (was 12 months now usually 6 months) and then those files get ditched too.

*

So that's quite a lot of detail how I would start out now with a different "Storage Workflow" - if starting out now: in simple terms:

> I would store far fewer images

> I would have a set time periods for "keeping" images before ruthless culling / processing and choice of storage (not many) or ditching (lots).

WW

[Loose Canon]

Hey Erik!

It was inevitable that your OP was going to draw long, drawn-out procedures (and more) on individual back-up strategies! Mine will probably be no exception!

But my procedures are very similar to yours.

Photos to computer for selection, editing, output (archiving). Meantime, The entire RAW shoot also goes to two externals, all named as one folder. Copies now on the internal (computer) and two externals, plus I don’t mess with the camera card until just prior to the next shoot when it is reformatted fresh. This to ensure if something bad happens to the computer (also MBP), I have the shoot in three places other than the internal HDD. When in the process of post production, if it is complex and I’m spending a lot of time and steps I will additionally back the edit to the externals just to have it until the post production is finalized.

After culling, editing, outputting are all decided (in that same named shoot folder) on the computer, that same shoot folder will be backed up to the two externals overwriting the original RAW folder backups which will greatly reduce the sizes of said folders. I will keep the shoot on the internal for a time until I have slept on it plenty, then after that time it will be deleted from the internal. I like to keep my MBP lean on file storage. I backup the shoot folders to the two external backups manually and it takes minimal time.

I asked an IT friend about a RAID setup and explained what I was doing for my backup. He explained that he would backup the way I was doing because for this to work I had to keep everything on my internal when hooked to the RAID because when they synced I would lose anything (permanently) on the RAID that wasn’t on the internal. Eventually filling the disks on the RAID, replacing them, and only then, be able to clean the internal on the MBP of past shoots and starting over. Once filled I archive them anyway, label them with what's on them, move along. If an external goes out, replace it and transfer from the existing external and good to go! This made sense with the way I like to manage my machine and the fact that my machine is not a dedicated photography machine and I don’t shoot everyday on a professional basis. However, I have also talked to two professional videographers who do shoot professionally on a daily basis and they use the same basic strategy that you and I do and for much those same reasons. Admittedly, I’m not a RAID expert which is why I have made these inquiries.

Now, one of my externals is a desktop type and one is a portable (mobile). The mobile stays with the MBP always. One reason for this is because I keep a partition of it as a bootable backup for the MBP, which also gets backed up regularly and after routine maintenance (which involves Disk Warrior and is another topic). If the MBP goes south, but still has electricity, I can boot from the HDD and at least have fairly recent backup of not only the photos, but emails, all other files and activities immediately. Just boot from the backup and I am back in black! If it goes far enough south I can transfer to a new machine easily with my latest backup in tact and keep moving!

I keep permanent seperate folders of outputted photos on my computer (which of course are also backed as described on the externals). One folder is full res .jpgs and one is re-sized .jpgs (for typical web use). These don’t take near the space as a full folder of RAW’s and .psd edits and I rarely re-visit an edit later. If I do it is within the timeframe of before I delete it off my computer. I don’t have time to go back years and re-edit quite frankly, and I’d rather be shooting/editing new with the time I have.

I back final .jpg’s in full res off-line as well.

So if after all this (and if it makes any sense) I wouldn’t do it any differently unless my present situation had changed. Meaning I shoot all day every day for paying clients. And personally, I’m glad that has not happened!

[davidedric]

Ok, here's my tuppence worth.

First, I do the vast majority of my editing in Lightroom, and only store the edits in the Catalogue (yes, I know that is potentially a single point of failure!). Hence the image files never change.

So, the images from a shoot go onto a pair of RAID 1 internal hdd's, and a copy is made to an external hdd, which, since I am a low volume shooter, I never expect to need to cull.

At the same time, I have CrashPlan running in the background to give me off-site storage of that (and my other important files). I won't bore you with the extra precautions around the Catalogue, but other than that I really just make that extra sooc copy after a shoot, and I'm done.

Dave

[TheBigE]

Yes, I guess I should have guessed this would turn into a bit of a how do I backup strategy as well. It is interessing to see the different paths and options. This is very helpful.

I would consider myself a low volume shooter, that could benefit a bit more by Culling more frequently. For that I guess I am trying to see if there is a point where I need to go back into my catalog and mine some specific photo. Recently, I am thinking that will not be the case but I think I need a bit more time under my belt to convince myself of this decision.

I truly favor portability above all and like to know that I have my current shoot on my computer for post processing. It maybe a week or two before I have enough large volume of photos to download and process. . I have been known to open my computer at lunch and work on a few photos here and there, and even on a train ride or two. Further to that point I really do not have an great deal of room to set up multiple RAID systems in my Flat. Likely if I did anything it would be a DROBO Mini with SSD Drives. A single RAID does nothing more than give me protection against failure of drive. I would need two to really have a true backup system.

Would I see any performance improvement editing photos off a RAID Drive (Wired Direct via Thunderbolt) vice editing photos off my computer SSD? I am thinking there would not be an improvement.

The other option is to continue working with two external WD Passport Drives and rotate those out every 3-4 years.

I do have a photo workshop this summer in France and this was one of the reasons I was thinking about looking at going to a RAID system, I am expecting a significant increase in photo volume at that point. Likely afterwards back to the usual rate.

[DickyOZ]

Just my two bobs worth. I prefer multiple copies to RAID. My photo album hasn't yet reached a terabyte. So I keep identical copies on the two hard drives in my PC. I keep a further two copies on USB hard drives and yet another ('Keepers' only) copy in cloud storage. I have not had a disc fail in the last fifteen years, old discs are piled up in a cupboard as they are replaced by newer, bigger and better drives. I don't know if the current generation of RAID controllers insist on identical disc replacement in the event of a single failure. When a mate of mine lost a RAID disc he couldn't get an identical replacement and had to replace the pair.
Dicky.

[DanK]

A quick follow-up to an important point Terry made:

overwriting the original RAW folder backups which will greatly reduce the sizes of said folders.

One of several reasons I sync to my external drive is exactly this. With syncing, the files on the backup are exactly like those on the primary drive. This makes it trivial to retrieve a specific file, but it also saves space: anything I delete on the primary drive, as when I cull photos, disappears from the external drive. My offsite backup is a true incremental backup, which protects against accidental deletions.

Unlike Dave, I never store edits in the Lightroom catalog. I store them in sidecar XML files. These files are tiny, and having them separate means that your edits are backed up separately from your catalog. If your catalog becomes corrupted, all you have to do is import the photos and xml files to a new catalog, and all of your edits are preserved. (However, when I tried this years ago, the histories didn't appear. I don't know if that is still true, or if there is a way to retrieve them.)

[Manfred M]

The other option is to continue working with two external WD Passport Drives and rotate those out every 3-4 years.

I just had a 1-month old 3 TB WD Passport that was less than a month old fail while I was in India a couple of months ago

I also use a backup strategy when I am traveling. I have a laptop along and make two individual backups on top of what is on the laptop hard drive and on the camera's memory cards. I try to do this every night. I store the full memory cards and two backups in totally different places when I am traveling, so that it something gets lost or stolen, I haven't lost all my images.

After the WD drive failed, I had one less level of redundancy, one that I fortunately did not need.

One issue with the portable hard drive solution (especially the solutions that use mechanical desktop drives). Drop them and your images are gone as these drives are not designed to take the shock that a SSD or even a mechanical laptop drive can take.

[davidedric]

However, when I tried this years ago, the histories didn't appear. I don't know if that is still true, or if there is a way to retrieve them

Dan, to the best of my knowledge that remains the case.

Dave

[Shadowman]

Manfred,

You stated you have dual RAID units, how are these connected? I know the stand alone unit would be connected through your router, how is the second unit recognized? Is it piggybacked on the first unit (through USB) or on another data port?

I recently had an issue where my RAID unit failed after a firmware update, luckily the unit was still under warranty and I was able to restart my backup from scratch. When I sent my unit in for RMA they had to send a new one and they don't do data recovery. So a second backup seems like a good idea.

[Manfred M]

Manfred,

You stated you have dual RAID units, how are these connected? I know the stand alone unit would be connected through your router, how is the second unit recognized? Is it piggybacked on the first unit (through USB) or on another data port?

Two separate networked Drobo units is what I use. One is a Drobo 5N and the other is a Drobo FS. I have a hardwired LAN connection via a router running through the house on top of a newer wireless network.

[Shadowman]

Two separate networked Drobo units is what I use. One is a Drobo 5N and the other is a Drobo FS. I have a hardwired LAN connection via a router running through the house on top of a newer wireless network.

Manfred,

Thanks for info.