Workflow - how to save, backup and archive files?

[Adrian]

I am contemplating abandoning both Aperture and Lightroom as library / asset management systems and wonder what experience others have of a different approach.

I started off using iPhoto as a library system, when digital photography started, and most family snaps prior to 2007 were mainly stored there. When I became more interested, I adopted Aperture, though have never really been in love with the library/project/folder/album system.

At work we use Lightroom, but I am disinclined to adopt it for personal use. At home we use Mac pretty much exclusively.

On a recent masterclass, the pro teacher was advocating a different approach. If I remember correctly this was, for every shoot (or every week or whatever):

1. Import from camera into Raw original folder and keep it as an archive for each shoot or time period
2. Copy this straight away into processed folder and use this to work on (PP using whatever you prefer) or discard images
3. For print files store the processed image as a TIFF in a third folder.

These folders are all backed up into two separate locations. So every shoot would have two or three folders. Maybe he also had a JPEG folder for processed images as well: not sure. Any thoughts on a foolproof and simple file structure? Bear in mind this is amateur workflow.

Reason for change:

I have become frustrated with the fact that these commercial asset management systems are easily broken by software updates over the years. Once a library has been updated to the latest version of Aperture (or iPhoto too), then old versions of the software are incapable of opening the images. It is also very easy to end up with duplicated images in Aperture, and the software is hopeless at removing them automatically. The VAULT system is also easily broken through software changes.

I want to be able to back up my photographs automatically to both a RAID array (to guard against hard disk or SSD failure) and to an archive for true backup (as a RAID system does not really deliver backup – very easy to corrupt or delete files). I am therefore thinking of a dual RAID 1, 5 or 10 array, coupled with a timeline archive system. I am happy to buy a suitable 5 or 6 disc RAID or NAS array (e.g. Lacie big5). I would need 10Tb scalable to 20Tb gross (bearing in mind this scales down depending on the RAID configuration). I use at least three different computers privately, none of which has anywhere near sufficient on-board memory to hold my entire photo library.

I am not a fan of Apple’s time machine as I find that it is very prone to failing to open old files if there have been several software changes in the meantime. Also the system overwrites old files once the disk is full. I do have access to cloud solutions (Phanfare and dropbox storage in reasonably plentiful quantities for example) but this is a bit unwieldy unless I go for an (expensive) business solution that automates it.

Any thoughts from people who have a really good workflow solution would be very much appreciated.

Adrian

[DanK]

Mine is simple, but it may not work for you if you don't want to use Lightroom.

I work on a PC, not a mac. My primary editing software is lightroom. I use other software, including Photoshop (for now) and Zerene as needed. I export to the web and print from LR.

So, my workflow is this:

For images that require only LR: I keep the raw files and xmp files. I don't keep any jpeg or tiff. I print directly from the edited raw file, and I export using plugins that don't require storing a jpeg on my hard drive.

For images that require additional software, I re-import the image into LR. In this case, I store the file that has been re-imported, as well as the raw. The edited file, most often a tiff, also usually has an associated xmp file.

I do one backup to an external drive and a second to Crashplan.

This will work unless I decide to ditch LR (which I have no inclination to do), in which case I would be using software that can't make use of the xmp files. I have a permanent license, so it would be a nuisance but not a disaster if I do decided to this. I will simply have to export images as TIFF when I want them.

[benm]

I store my files by year/month/day. I manually back it up to multiple external hard drives. Each hard drive is kept in a different location.

I currently use Lightroom on a Mac. I back up the Lightroom catalog regularly to the same hard drives as the photos. This saves all the edits. The catalog also contains the keywords and collections that I use to organize the photos inside Lightroom and is completely independent of how I organize them on the hard drive (which is year/month/day as previously noted).

It's simple and redundant.

[davidedric]

I can't help with Adrian's workflow, but I would like to add to the back-up debate.

Back-ups are important, but unless you know that you can restore your back-up it is useless.

I am writing from a professional lifetime of systems management, and many simulated (and a few real - I still have the scars) disaster recovery scenarios.

There are almost always problems in restoring data and operations following a failure. If it is vital to you to be able to recover, it really is essential that you find a way to simulate a disk failure, for example. In essence, you need a back-up system on which to do the restore.

It's difficult, and probably expensive, to do so. I don't - but I'm a hobbyist. If my livelihood depended on it, I would.

Dave

[jcuknz]

I suppose it is rampant commercialism but it has puzzled me for ages why people use all these fancy programmes when, if you bought Windows, you were given a perfectly useful programme called Directory,Windows Explorer etc. Maybe this has changed becuase I am still happilly using XP despite my advisor telling me that techs with knowlege of XP are getting scarce .

Whatever you use you still have to have the nouse to use it for it to be effective.

Adrian ... I think your tutor is correct and it is similar to what I do. Save everything, have working copies, save work-ups and delete files not used from working folders after a period. I only have two HDs but hopefully in my limited lifetime left they will not both go bottom ups at the same time ... after that I won't be interested

[KennyF]

I have been told by a lad who is currently studying for an IT degree that unless you have 4 copies of a file in separate places then do not consider that it exists.
Now this may seem extreme but I agree with Davidedric, there WILL be a crash and there WILL be probs restoring. I keep 2copies, 1 in Picassa on my second hard drive, (picasa coz I'm a hobbyist not a pro with a mint to spend on software I edit as JPEG in GIMP) and a second on an external hard drive The external hard drive is not going to fail if my PC fails and I have a remote copy. As for filing I used to file by date but then I could find nothing because I'd think to myself " I want the pic with the time we were out in the woods and got the autumn leaves" but then maybe didn't upload it at the time or whatever. I now file by category using a mnemonic system that runs along the lines of "Sea and sand" etc it works for me as a hobbyist but probably won't for a pro or semi-pro. I agree with JC: Windows filing system is pretty adaptable and is the original of the species, try use it maybe.

[KennyF]

I have been told by a lad who is currently studying for an IT degree that unless you have 4 copies of a file in separate places then do not consider that it exists.
Now this may seem extreme but I agree with Davidedric, there WILL be a crash and there WILL be probs restoring. I keep 2copies, 1 in Picassa on my second hard drive, (picasa coz I'm a hobbyist not a pro with a mint to spend on software I edit as JPEG in GIMP) and a second on an external hard drive The external hard drive is not going to fail if my PC fails and I have a remote copy. As for filing I used to file by date but then I could find nothing because I'd think to myself " I want the pic with the time we were out in the woods and got the autumn leaves" but then maybe didn't upload it at the time or whatever. I now file by category using a mnemonic system that runs along the lines of "Sea and sand" etc it works for me as a hobbyist but probably won't for a pro or semi-pro. I agree with JC: Windows filing system is pretty adaptable and is the original of the species, try use it maybe.

[Adrian]

Guys - the question was about file structures within work flow.

I am backing up - always have. It will soon be start of the art and automated with a 5 disk raid array with full redundancy plus timeline archive, plus copy on home and work mac plus cloud for important stuff: so having a backup hierarchy is not an issue. What is an issue is determining the optimal file structure that will enable me to have rapid access to images two decades hence, without significant risk of photo software dependencies.

Adrian

[KennyF]

Re backups - Sorry lost the thread. Re filing structure? Really it has to be simple and future proof as you've said. Are u using windows or mac? if windows then I would use the windows filing system as this will be future proof as long as microsoft are around as far as I can tell. File names? what about a structure like " Biggin Hill airshow 12/3/13". then you have a way of remembering and searching by date and by what happened / what you took. I don't know if you will be able to categorize more broadly, say a folder for weddings one for events one for family and then sub folders for each shoot, that would depend on how you work and importantly how you think. I think in images, so I need to call folders by what type of image they contain others are good at where they took the shot so a folder by that name would be best. I don't like to rely too strongly on the folder structure imposed by software for reasons you have figured out already hence the thread. I hope this helps and that I have redeemed myself I hope someone else has a better suggestion.

Kenny

[Adrian]

Guys - the question was about file structures within work flow.

I am backing up - always have. It will soon be start of the art and automated with a 5 disk raid array with full redundancy plus timeline archive, plus copy on home and work mac plus cloud for important stuff: so having a backup hierarchy is not an issue. What is an issue is determining the optimal file structure that will enable me to have rapid access to images two decades hence, without significant risk of photo software dependencies.

Adrian

[Manfred M]

Adrian - use the same file structure as I use on my computer storage in my backups (after all I do want to find the files again and my structure works well enough for me).

While I have several computers; I store all of my normal documents (text, spreadsheets, etc) on one and all of my images on another. My laptop is simply a portable workstation and I never have originals there.

For my media; I have threes sub-directories; audio, video and images. General storage is by year and special projects are by event. This means, for instance, I have a separate directory in each of these three folders called Namibia2011; one for the sounds that I collected, one for the video I shot and one for the images I took. The pictures I took yesterday are in a folder called 2013-08-05.

I have a 5-disk NAS Drobo FS as my main backup (stored in a different room in the house) and for the best of the best, I will put these on a second smaller RAID box in a third location. The Drobo and NAS are isolated from the electrical mains.

[RustBeltRaw]

I also recommend using simple file structures to store your images. Properly managed, they're easy to navigate, and considerably more future-proof than dedicated photo organizing software.

After a shoot, all RAW files go from my cards to my laptop's 120GB SSD (I was an early adopter, thus the very low capacity), then get separately copied to two Seagate 750GB drives. I verify that the images are in all three spots, then clear the card, and work from the SSD to delete images I won't use, rate the rest, and edit a selection. Then I mark the backup folders with a "D" tag at the end, copy the working folder to the backup drives, and after verifying the transfer, I delete the original backup folders. After posting the images, I delete the working folder from my SSD. So I have triple-redundant copies while working, and double-redundant after. This is not a perfect system, and I would like to improve its automation and robustness.

Organization is by ISO-standard date format (YYYY-MM-DD) then " - " and a description of the event, all sorted in reverse chronological order (newest first). I try to name folders with the keywords I'll likely use to search for them, like a model's name, a location, or the teams I was shooting. Recurring projects like roller derby get dedicated folders with prefixes to keep them at the top or bottom of the order. So my photo directory winds up looking like this.

XX Rust Belt Raw
X Roller Derby
X Fire Performances
2013-08-06 - Joan Jett portraits at Harpo's
2013-08-04 - Lotus Formula 1 at Silverstone
2013-07-30 - Corvette Racing at 24hrs of Le Mans
2012-11-02 - Diving Chuuk Lagoon
...
0 Legal Documents
00 Random Miscellany

Unfortunately, some of these folder names are wishful thinking.

[Polar01]

Adrian: Here is a workflow that I found useful:
1) from camera raw files to folder named "Raw images 2013" sub folder "month"
2) at same time copy to folder "Temp images"
3) here sort images to delete or process
4) the "temp folder" is where I work on images all other as above deleted
5) move those images processed to folder, "Process Images 2013" and sub folder "month"
6) those image selected to print copied to new folder "Possible Print 2013" no sub folders.
Note: file names change, files that are processed a capital "P" is added to the file name, files that are possible print the "P" is changed to "F" (final) and if printed than the given name is added to the file name at the end. This way I can trace the image back to the original raw file.
Note: files in folder "Possible Print", will still need to be fine tuned every time the image is printed as size changes and if there is a change in stock that the image will be printed on.
This is just the way I do it, of course backup is also done on both internal and external drives.

Cheers:

Allan

[Mito]

As a relative newcomer to "mass storage" I use a year - month - day system. Within the day I have files eg. Us , Views etc where I put photos that are special to me. I use Linux as my OS and Shotwell as my primary organiser. This saves the RAW file plus createing an embedded JPG. This JPG I use for quickies to put on the web. I keep the image number on all the processed photos so I can relate back to the original. I have yet to rate the photos. All my efforts are are for my own personal use, I have not yet reached the stage of wanting to sell, that is if anybody were interested in buying.
Back up is manual to 3 separate hard discs.

[Adrian]

Thanks all.

Year / month / day is obviously logical plus some sort of event name to facilitate searches. I was not sure whether I need to go the pro route of always keeping an untouched raw file in a further hierarchy of original raw / process raw / printable tiff for files to print / jpeg for files to share as this adds at least two and possibly four files for each event.

The underlying Aperture library file structure basically files by event and date , and always maintains an original, so I guess in a way I am replicating that more manually. It may be that the smart thing to do is keep using it, but maintain two archived copies of every version of Aperture / ACR / Lightroom software from now on.

My intention was either to use a NAS array or the latest Datto cloud solution. In my office we use a 16 disk RAID, doubled up, plus Datto, so we have full redundancy on site in two different places, one of which is fireproof and isolated, plus tape with hierarchy kept off site, plus full cloud copies to controlled servers that we can recover from in about 2 hours for the whole business. This is very expensive and overkill for home use obviously, but I am keen to achieve similar piece of mind having very nearly lost all of my own wedding photos and been very badly told off by my wife! (Lacie back up disk failed due to corruption, Apple archive failed due to software change making archive inoperable without serious work, laptop dropped onto stone floor and SSD not readable for a while: all within the space of a week).

In some ways, digital is a pain. In the old days if we went on holiday we might have 70 printed pictures and we could store the negatives somewhere safe. Now we go on holiday and come back with about 1600 images / videos off three cameras and print hardly anything as it is easier to just show edited highlights on screens all around the house or carried around with us (phones /iPads). The world has changed.

Adrian

[davidedric]

Hi Adrian,

FWIW I do keep an Out-Of-Camera copy, separate from my workflow, on RAID (in this case, mirrored) disks. Not completely solid, but I decided it was good enough for me. When I ordered this computer a couple of months ago, I asked the supplier what spec. of disk I would need to replace a drive if one of the mirrored ones failed. His answer was "any 2Tb drive will do". Mmm I rather doubt that in two or three years time. So I wondered whether I should buy a third identical disk to keep in the cupboard. Then I decided that life is just too short. Even now, the only way to find out if the array will rebuild is to physically try it - but I don't plan to.

A true story. A good few years ago (long before clouds), at work we had the usual set up of a main computer suite, a "disaster recovery" suite in a different location, and back-up tapes securely stored in a third place. Then the call came (a test, fortunately!). "OK the computer suite has burned down, the operators were all outside having a fag so you still have the human resources (funny how people forget about that), activate the recovery plans." So we swung into action, sent a van to collect the recovery tapes, and with the tapes safely loaded on the way to the recovery site - it was rear-ended on the motorway. Exercise aborted! Yes we updated the disaster recovery plan, but you can only mitigate so many risks.

BTW, my wedding photos (black and white ) are a single copy in an album. Ironically, I can see it on the shelf in my computer room as I type.

Dave

[wobert]

Hi Adrian, I am a Lightroom user and learnt very early that a good file system works well in both the Windows-7 File system, and the Lightroom Catalog.
I have an internal Hard Drive dedicated only to images. (It is also backed up with software)
I import from my camera cards using Lightroom-
Convert raw files to DNG at import,
adding copyright to metadata and applying a camera profile preset to improve colours.
Images are imported to a YEAR subfolder and automatically placed in a subfolder with the DATE OF EXPOSURE.
(YYYYMMDD will sort chronologically! as does the YEAR folders.)
In Lightroom I rename each 'Day' folder with a brief descriptive name after the date. (YYYYMMDD Description)
I also apply Keywords to every image at the time of import so finding any image is much easier.
If I later pixel edit images in Photoshop or NIK (PSD, TIF, JPG, PNG, etc) that I must keep, these co-exist with the original RAWs in the same 'Day' folder.
I also have the Lightroom Catalog in a folder which will backup with all images on the hard-drive.
This system makes the folder structure logical and easy to control, but any and all changes to filenames, folder names, moving files, moving folders, etc MUST be done in Lightroom (so as not to confuse the Catalog!)



All the best- keep at it, I hope you achieve the system that works best for you.!

[Wayland]

I started my file structures way back while working with Xtree which tells you all you need to know about my age.

As such I use a fairly simple hierarchy starting with the hard drives:

C for the System and programs.
D for non image Data, documents and system backups.
M for Master image files and finished reproduction copies.
R for Raw files.
S for the Scratch drive. SSD with no permanent files.

M and R are regularly backed up onto removable hard drives and C is backed up onto D once a week.

The raw files are arranged simply into files by camera/year/month.

M is arranged by Master or Reproduction

Masters are arranged by Genre/year.
Reproductions by genre/year/purpose.

It doesn't take me long to find anything I need.

[GrahamS]

Outback Photo have published a Digital Photography Workflow Handbook which is worth every penny of the modest price. It's a download from here:
http://www.outbackphoto.com/CONTENT_..._03/index.html

I can recommend it.

[LocalHero1953]

Adrian, I'm not sure from your earlier post if you were baulking just at the idea of keeping a virgin RAW file separate from a working RAW file, or at the idea of keeping a full set of TIFFs or JPGs as well. A year or so ago, while I was newish to Lightroom, I had a software incident (human error - it happens) that reduced all my then processed digital images back to their virgin RAW state again. Fortunately, a few days' work dealt with the bulk of them, but if the same happened again now I'd be tempted to give up photography. So I now keep TIFFs of all my images after processing, as a record of my best effort of shooting and post-processing. If the same disaster happened again, I would have my back ups AND evidence of what the processed RAW should look like.

And if you wish to leave your images to posterity, what file format is likely to be around and readable longest? Probably JPG!

I also have one copy of our wedding photos, on paper in an album - and even they are fading!