Organizing Thousands of Photos: How Do YOU Do It?

[Viana]

I, like many of you, have tens of thousands of photos both film and digital. While I have 2 backups of my digi-photos, after a recent meltdown of a backup drive I came to the conclusion that I must find a better method of organizing those photos.




Question: How do you organize your precious photos: By year, by project, permanent archives—another way?




I keep the current year on my computer and have the rest on backup by years in 3 areas: Business, General and Travel. Most photos are named instead of numbered and are in their respective folders and sub-folders.




Do you have different Lightroom catalogues for different projects?




Do you convert everything to DNGs? After processing do you export photos as PSD files, TIFFs and JPGs and keep the originals?








I am really looking for a better mousetrap here.




Thanks in advance for your counsel.

[herbert]

Hi Viana,

I read a lot about LR and how to optimise it before building my new computer. Anecdotes led me to believe that LR 3 can handle catalogues as large as 150,000. Since I only shoot about 10,000 photos per year I thought that by the time this becomes a problem there will already be a new version with a solution.

The architecture of LR is based around a central SQLite database file (the catalogue), a preview for every image in the catalogue, the raw files and a tempory cache for raw files in progress (the camera raw cache). The system needs to do read/writes all the time to the catalogue but since it is a paged database the file size does not change very often. I.e. LR makes the catalogue a set size with blank bits inside that it can write to. So this avoids file fragmentation but still requires a lot of disk IO.

When working on a photo in library view LR will:

• Read the previews constantly for any image currently displayed
• Need access to the LR catalogue for any photo details.

In develop mode LR will:

• Demosaic the raw file and store it in the camera raw cache (if it is not already there)
• Use the demosaic file as the source for any develop edits
• After a develop edit, update the preview
• Need access to the LR catalogue for any photo details.

This means that there is a triangle of source/destination locations for:

1. The raw files
2. The camera raw cache
3. The previews

Since LR can read from 1/2/3 and write to 1/2/3 (although 1 only on import) the best option is to put these three locations on different discs.

One problem with LR is that the previews need to be in the same location as the catalogue file. However modern operating systems allow you to create symbolic links to files and locate them elsewhere. Consequently it is possible to locate the 3 most used files on a superfast disk:

• The catalogue (*.lrcat)
• The previews database (previews.db inside the LR catalogue previews folder)
• The root pixels database (root-pixels.db inside the LR catalogue previews folder)

My set-up is consequently as follows: A solid state drive for my operating system and the lightroom previews folder and 3 additional disks: one for the raw files, one for the camera raw cache and one for backup.

I keep my LR catalogue and the two previews DB files on a RAM disk that is loaded at start-up. I use symbolic links to trick LR into thinking they are in the correct location. I do not worry about the volatility of the RAM drive (if you turn it off you lose all data) since (a) my system in very stable and (b) I set LR to back up the catalogue each time I exit. When this happens my volatile RAM version of the LR catalogue is copied to a real hard disc. This is fine since it prompts you if you actually want to do this and can be ignored if you have not done any real editing.

I then use the 'TPG LT Backup' plugin to compress my back-ups regularly and move them to a back-up drive.

As well as an internal redundant back-up of my photos I also copy the photos and LR catalogue to two external drives on a cycle.

My workflow involves import, deletion of bloopers, then keywording. I do not need to organise my folders because LR does it all for me with metadata. (Note that google uses metadata to look after all the web so I think 150,000 photos is fine for my computer.) I can find any photo I want with keywords of which I have 3-5 on average per photo including colours of flowers, places, categories (landscape, portrait), animals and peoples names. Stuff you would want to use to find photos again.

I do not convert to DNG since this means that the file has been demosaiced from the original raw data. Since algorithms get better there may be a time in the future that you want to do this again. Another argument is that DNG stores changes to XMP inside the file making file backups very long and tedious. If you use seperate sidecar files for XMP then the backup only has to copy the XMP file since the raw is unchanged. The argument for DNG is that the format has better compression and so you save about 20% disk space.

As for my computer I have a i7 with 16Gb of ram (3GB for the RAM disk). However since a lot of LR tasks are run on a single thread you will not see a lot of difference with 2,4,6,8 cores as long as your clock speed is high. LRs main bottleneck is disk I/O. Get that sorted and you will wave goodbye to the 'Loading...' message.

I think that is good for starters. If you have any questions about computer set-up or workflow then please ask.

Regards,

Alex

[Mark von Kanel]

Bugger! thats a lot of words for "use lightroom, its great!" (scott kelby does a good book on lightroom, it will do everything you need it to. but you already know this dont you ??

[herbert]

Bugger! thats a lot of words for "use lightroom, its great!"

I couldn't stop myself...

Lightroom is great BTW.

[Glenn NK]

I use LR exclusively - for organizing and processing (well I do some very minor work on the occasional image in Photoshop Lite).

I personally DL photos to my computer based on dates. So for every single day on which any number of images were captured, a folder is created for that day.

On the occasion when I have more than one date on a single card, then there is a folder for each day. Then when I import them into LR, they remain in these folders. Incidentally the Canon software does this automatically when photos are copied from the CF card to the HDD.

For my use this folder system is quite workable, but for a wedding photographer this would be a disaster, and they would likely have a folder for the Jones wedding, the Smith wedding etc.

The folders could also be organized into topics like birds, scenery, closeups, people, etc. It's up to you.

Of course like any PC system (windows or mac), there could be sub-folders too.

Spend some time thinking about how you would sort them if you were sorting slides or a photo album.

Glenn

[Viana]

Thanks for all the info everyone.


I shall of course continue to use LR and probably my old organizing system of year folders with named sub-folders. I recently tried the dated folders and found that to be very annoying and confusing.


With keywords and metadata everything should be “findable.” Now, if I can just find the time to apply those to all the old photos. . .


The jury is still out for me regarding DNGs.

[Glenn NK]

Now, if I can just find the time to apply those to all the old photos. . .


The jury is still out for me regarding DNGs.

1) Finding time - let me know where you find some - I need some too.

2) DNGs: I thought about it, but there doesn't seem to be an advantage as DNG is essentially another form of RAW. The upside is that in time the DNG may become a standard, but so far it's not.

Glenn

[Colin Southern]

2) DNGs: I thought about it, but there doesn't seem to be an advantage as DNG is essentially another form of RAW. The upside is that in time the DNG may become a standard, but so far it's not.

I use DNG exclusively ...

- If you don't, then EVERY edit to EVERY file has to either be stored in Adobe's internal database (which its probably fairly safe to say that many people don't even know about, let alone backup), or they're stored in a seperate .XMP "sidecar" file which MUST be kept with the original RAW file forever. Lose / corrupt the database or disassociate the XMP files from the RAW originals and the unlucky recipient has just lost ALL of their RAW adjustments for those file(s)

- DNG files are generally a bit smaller

- Since my computer has to transfer the RAW files from the memory card anyway, it's no additional overhead to have it convert them to DNG whilst it's doing it (and rename them and add metadata in the process)

- It also gives the ability for files to be opened in earlier versions of Photoshop

Just a few thoughts

[Mark von Kanel]

I use DNG as well, i use 2 cameras , canon and a nikon and like just to have one flavour of image file, keeps thing neat, i never save any jpegs because its easy to make more at a push of a button, so why use the space?

[Black Pearl]

I import them in to a new folder via Adobe Bridge - naming the folder with an event or place - then they are fairly easy to search for in the future.
Bridge will spot if a folder exists with the same name and put the images into it automatically.

Bridge also makes for a very good and fast browser, linking into your image editors - Photoshop/NX2.

Good tip if you own a Mac is to use the Spotlight, type in the name you're looking for and OSx will point you straight to it.

[Colin Southern]

I import them in to a new folder via Adobe Bridge - naming the folder with an event or place - then they are fairly easy to search for in the future.

I do the same, with the exception of putting a date code at the beginning, so the folders sort in date order eg: "11-11-19 - Back Beach Shoot with Kayla"

[rlebleu]

I like Idimager.

Idimager's downloader reads my SD cards containing Pentax RAW files, verifies the download, calls Adobe DNG Converter to convert the raw files to DNG's, and files them in a Date/time structure \yyyy\mm\dd and renames them to yymmdd-original filename. The reason for renaming them is my camera's filenames _IGPXXXX repeat after 9999 images, and I want unique filenames in my database.

It then copies and verifies the original image files into a similarly structured backup folder without any conversions at all so I always have an original file saved.

In the process, it applies any captions, headlines, copyright information etc and attaches a label ( Idimager's version of keywords - but keywords are possible also).

Images are sorted and found by label name most of the time, but they can be sorted by any metadata collected.. for example camera name and shutter speed...

If you already have a library, you can simply 'Import' the images into Idimager's database from their existing locations.

There is a great more to this, but you can learn more at their web site...

If you are interested have a look at www.idimager.com

Dick

[darekk]

Perhaps Adobe Lightroom would be the best choice, Adobe is probably the best company producing graphical software, but I don't have Lightroom (only Adobe Photoshop Elements) and store relative paths to thumbnails in the Microsoft Access database. This is very universal format, known very well from many years. Thumbnails on the disk are organized by date only, but described in the database. Then records can be filtered by subject. I have also created forms displaying thumbnails (see attachment).