Lr5 snafu

[Manfred M]

Setting up a storage strategy (including how you set up your folders) doesn't have to be complicated. I store everything chronologically, but have separate folders for special events, trips, etc, where again I use a chronological approach, but cluster these files together.

I also have a backup strategy where I store duplicate files in at least 2 and in some cases three places that are not right beside each other. I do use RAID (and RAID like) drives for backup, so when disks fail, I can quickly restore my files.

When it comes to using Lightroom; I stronly suggest you do not depend on the catalog system for storing your edits; you are now well aquainted with the weaknesses (in my view "fatal design flaw"), with that tool. Save your edits as either dng files or (my preferred approach) as sidecar .xmp files (done in the catalog settings).

[Mark von Kanel]

and when you move them for goodness sake do it in LR or it will lose them all again!

[Manfred M]

and when you move them for goodness sake do it in LR or it will lose them all again!

Umm. Perhaps not as this is what caused all the problems in the first place.

As I (and others, including Colin) have been saying for years, the monolithic design of the Lightroom catalog system is its fatal flaw and people should be willing to work around it so that this does not happen again. The only "safe" way to use Lightroom is to have multiple catalogues (one for every project?) and then store all edits as either dng files or as .xmp sidecar files, otherwise this is going to happen again.

[Colin Southern]

The only "safe" way to use Lightroom is to have multiple catalogues (one for every project?) and then store all edits as either dng files or as .xmp sidecar files, otherwise this is going to happen again.

In a moment of spontaneous sarcasm I'm temped to say that the only way to safely use LR is to uninstall it and use Bridge!

OK - on to something more helpful ...

If anyone is interested, I organise my shooting into 2 folders:

The first is PEOPLE where folders are organised as such ...

PEOPLE

SMITH, John

12-03-12 - Studio Shoot
13-04-19 - Location Shoot (Back Beach)
14-05-27 - Location Shoot (Isel Park)

JONES, Mark

11-10-12 - Studio (Product Shoot)

and

LOCATION

2012

12-01-22 - Barcode
12-05-30 - Towering Inferno

2013

13-09-03 - Rocks Road

blah blah blah

Keep in mind too that how we find information is changing; these days if you're after information on something you probably WON'T try to locate a website by guessing the name - you'll probably just use a search engine - probably Google. In a sea of data, it works well. Why then do you try to find image collections by navigating when - assuming you've put appropriate key words in the title - don't you just search your PC for it? I do that even on my own website now; I'll be looking for an image - can't find it (too many there) so just ctrl+f - type in the name - bingo. Ironically, if it's not in that section, I can usually get Google to find it in a second by searching on a few key words (and restricting the search to a particular site).

Case in point - my image "Barcode" ...

http://lmgtfy.com/?q=barcode+site%3A...m%2Fcjsouthern

First hit

[Manfred M]

In a moment of spontaneous sarcasm I'm temped to say that the only way to safely use LR is to uninstall it and use Bridge!

I must confess, that thought crossed my mind as well...

[Colin Southern]

I must confess, that thought crossed my mind as well...

But we cant say that because then all the LR5 affectionados will jump on us from a great height to point out how wrong we are because hundreds of zillions of people from all over the world use it with out any problems (cough, and only some lose all of their work!, cough) - and then others will assure us that all they need to do is backups to be safe (not realising that relies on them spotting a corruption before it infiltrates all of their backup sets) - and then other will argue just because they like arguing ...

[Mike Buckley]

There really is another alternative: don't use the cataloging function of Lightroom as a catalog. Instead, use it only to import the images. Then adjust the images, export the images in a file format(s) that meets your needs, and remove the adjusted images from the catalog. It really does work like a charm.

I am not going to get in a discussion about this, as the last one was exceptionally dissatisfying. Try it. If you like it, great. If not, great.

[Glenn NK]

There really is another alternative: don't use the cataloging function of Lightroom as a catalog. Instead, use it only to import the images. Then adjust the images, export the images in a file format(s) that meets your needs, and remove the adjusted images from the catalog. It really does work like a charm.

I am not going to get in a discussion about this, as the last one was exceptionally dissatisfying. Try it. If you like it, great. If not, great.

Mike:

That sounds familiar:

Maybe I'm doing things too simply too - from my camera the images are DL'd to the computer using EOS Utility. It puts them into folders with dates (2014_08_03 is the newest one from today). I suppose I could get ambitious and have a "flower" folder, a "people" folder, a "landscape" folder, but I never got around to it, and besides I have a pretty good memory as to when image were taken. Occasionally I call one "Summer Holidays 2014" etc.

Upon exiting LR, I very regularly do an Integrity Check and Optimize Before Backup.

Virtually all the software problems I've ever had can be put down to user error (mine). (ducks and runs).

Glenn

[Colin Southern]

Mike:

That sounds familiar:

Maybe I'm doing things too simply too - from my camera the images are DL'd to the computer using EOS Utility. It puts them into folders with dates (2014_08_03 is the newest one from today). I suppose I could get ambitious and have a "flower" folder, a "people" folder, a "landscape" folder, but I never got around to it, and besides I have a pretty good memory as to when image were taken. Occasionally I call one "Summer Holidays 2014" etc.

Upon exiting LR, I very regularly do an Integrity Check and Optimize Before Backup.

Virtually all the software problems I've ever had can be put down to user error (mine). (ducks and runs).

Glenn

But do you store edits in the LR database (ala "catalog") or in DNG / Sidecar files?

[oldgreygary]

I'm slightly surprised that it is being said that the database structure is a weakness of Lightroom or indeed any system. As a former IT professional I have worked on software such as Informix, Progress and Oracle which are all database based software systems which small, medium and large business's rely on. Using a database is pretty much common practice and as long as you are aware of it and have a solid backup strategy then it shouldn't matter.
You could go down the argument of having your originals on a seperate drive and what happens if that gets corrupted? That's the same scenario?
Whether you use Bridge, Lightroom or any other software for that matter corruption, loss of data is always going to be an issue. It is more about how you manage that by having a solid recovery plan in place.

Gary

[Colin Southern]

I'm slightly surprised that it is being said that the database structure is a weakness of Lightroom or indeed any system. As a former IT professional I have worked on software such as Informix, Progress and Oracle which are all database based software systems which small, medium and large business's rely on. Using a database is pretty much common practice and as long as you are aware of it and have a solid backup strategy then it shouldn't matter.
You could go down the argument of having your originals on a seperate drive and what happens if that gets corrupted? That's the same scenario?
Whether you use Bridge, Lightroom or any other software for that matter corruption, loss of data is always going to be an issue. It is more about how you manage that by having a solid recovery plan in place.

Gary

Not that easy Gary.

If folks store edits in the database - and a database corruption occurs but isn't noticed because it only affects blocks of images that are currently "out of sight and out of mind" then when the issue is discovered (a) it's probably permeated all current data sets and (b) even if it hasn't then you still lose edit that have been done post backup.

Yes - databases are used extensively in the real world - but ones like SQL are built on far more robust structures with technologies like transaction-based processing and logs that can be re-run against the database.

They're both databases but worlds apart.

[oldgreygary]

Fair and valid points Colin. I should have said in my post that I don't particularly favour anyone software solution over another. As a gnarled and probably slightly cynical ex-IT professional I like to think that I take an objective view of software that I use. By the way, I use or have used Lightroom, Photoshop, Photoshop Elements and Bridge
While the larger business systems are more bullet proof i.e. transaction based processing with recovery logs etc. I don't think that people need be afraid of information held in a database. I can certainly see your point that a huge amount of work can be lost if the database is 'silently' corrupted. But I would think that would suggest a major software or hardware flaw?

I guess that's probably the main point about Lightroom which is, is there an awareness there for everybody who uses it of the potential pitfalls? Hence my comment about backup/recovery strategies.

I'm not a professional photographer so perhaps that gives me more scope to be able to try different software without worrying too much about the consequences. If my livelihood depended on it then I would be more inclined to find a solution that was stable and repeatable and gave me the best prospect of having access to my images. In that case then maybe Lightroom might not be the best solution.

Anyway, I'm hijacking the post with some unnecessary rambling.....so I'll stop there!

Cheers for now

Gary

[Colin Southern]

Fair and valid points Colin. I should have said in my post that I don't particularly favour anyone software solution over another. As a gnarled and probably slightly cynical ex-IT professional I like to think that I take an objective view of software that I use. By the way, I use or have used Lightroom, Photoshop, Photoshop Elements and Bridge

Many days I'd like to be an ex IT professional too (been at it since the early 80s).

While the larger business systems are more bullet proof i.e. transaction based processing with recovery logs etc. I don't think that people need be afraid of information held in a database.

They're great when they're working properly - and by and large things are pretty reliable these days -but that's cold comfort to those who get bitten. Even the very title of this thread suggests that William has had an experience that's way too close for comfort.

I can certainly see your point that a huge amount of work can be lost if the database is 'silently' corrupted. But I would think that would suggest a major software or hardware flaw?

Um, no and no.

If the software is set to write changes to the DNG or sidecar file (and not store it in the database) then no edits should be lost - however - this isn't the program default. And no - it wouldn't take a major software of hardware flaw - even something as insignificant as a momentary powercut or a buggy kernel-mode driver can result in an unexpected shutdown - at which point even transactional file systems like NTFS can result in file corruption. In that respect I see it as being a bit like driving without a seatbelt -- may not bite you in the bum today or next week, but if you drive enough days in a row - with longer and longer journeys each time - then eventually chances are you'll have an incident. Backups are a starting point, but I suspect the vast majority of photographers don't have an effective off-site strategy - and even then it's by no means foolproof. I've seen people get themselves into all kinds of mess with backups and restores. I've had to restore SQL databases sitting on certified dedicated server-grade hosts with ECC RAM - continuous AC-DC-AC power conditioning - in air conditioned rooms; and if that's still needed then how many years work should an end-user entrust to a simple database engine running on budget price hardware - no UPS - co-existing alongside many other pieces of software - where it's backed up to a memory stick left sitting plugged in to the machine once or twice a year. Unfortunately, that's pretty close to the reality for many. Some of course are better - so they backup to an external drive once a month and think they're safe because they keep it in a fireproof safe (which WILL NOT protect the drive in the event of a serious house fire).

I guess that's probably the main point about Lightroom which is, is there an awareness there for everybody who uses it of the potential pitfalls? Hence my comment about backup/recovery strategies.

To be honest, I don't think 1 in 20 photographers really understands the risks; and if you don't understand the risks then clearly people can't make informed decisions about them. In my 30+ years of doing this I've seen numerous hard drives fail - I've seen data wiped out by viruses - I've seen a safe that 3 men would struggle to lift stolen from a mezzanine floor (crooks were looking for money - but all it contained were their backup tapes) - I've seen people with onsite backups unable to access them for a year after our Christchurch earthquakes. I've seen databases pass integrity checks - then fail end of year rollovers (in essence needing 11 months work re-keyed) (yep - it was corrupted, but not detected - then over-wrote all of the daily / weekly / monthly backup sets). I've seen all 3 tape backups of a multi-user system fail on restore. I've seen a 3 dentist practice lose 2 weeks of work because they only used 2 tapes ... the dentist in charge thought the receptionist was alternating them ... and the receptionist thought the dentist was alternating them. (I kid you not) (3/4 of an hour into the restore "bink -- unable to read from tape -- OK?" (no, not OK, but only the one button to choose from) (what saved their bacon was a trial backup I'd done on the other take 2 weeks before after changing the tape drive).

If people really want to keep safe, then here's my advice.

1. Don't keep all you eggs in one basket - keep your edit data with the images they pertain to.

2. Keep at least 3 copies of all important data - one of which must be kept offsite (I use Google drive) (and now also Carbonite) (but there are many other effective strategies).

For what it's worth, I once lost 6 months of accounting data because "the doctor didn't take his own medicine". Major wakeup call. Now I backup to excess; funny story ... not long after I started doing proper backups I corrupted my accounting database again (I've got a temperature sensitive solid state drive in my system). This time I just laughed - and had it restored from Google drive in about 3 minutes.

I'm not a professional photographer so perhaps that gives me more scope to be able to try different software without worrying too much about the consequences. If my livelihood depended on it then I would be more inclined to find a solution that was stable and repeatable and gave me the best prospect of having access to my images. In that case then maybe Lightroom might not be the best solution.

I know what you mean, but when I've been involved with people who'd lost memories of a lifetime (kids growing up - overseas trips etc) I can tell you it's quite sobering, even though they're not "professional photographers". For one lady the choice was "lose all those memories" or "cough up around $3000 to have them recovered by drivesavers". Her decision was to put the drive away somewhere safe so she could have it recovered if she ever could afford the $3000 or so. It's sobering, and it's real world. And it's getting worse as the amount of data increases vastly (I was telling someone today that when I put my first server in for a particular company - and cut-across the data from their old Novell server - the total data that I moved across was ... wait for it ... 600MB (not a typo). These days they have on one server some 60+ mailboxes - some in excess of 32GB. Scary).

[oldgreygary]

Thanks for your detailed and informative response. I think that a perception of digital technology is that it is 'perfect'. As you quite rightly point out in your response there are many scenarios for things to go wrong. I don't think that the way data is stored and backed-up or recovered is fully understood. It's something that's brushed under the carpet until things go wrong! Interestingly, I attended a lecture on storing images. The speaker worked in conjunction with museums etc. There conclusion was that the best way in there scenario to archive photographs was printed output!

Cheers for now

Gary

[Mark von Kanel]

Horses for courses really.

Had LR not been around when i re started togging then i may have used bridge but as a beginner LR is attractive because it give everything you need to start out with. having said that ive just checked the file structure and all of my images are stored in the same place by light room in the DNG format and ive just opened bridge and it found all of my files with edits, so ill not worry about it.

[Colin Southern]

Thanks for your detailed and informative response. I think that a perception of digital technology is that it is 'perfect'. As you quite rightly point out in your response there are many scenarios for things to go wrong. I don't think that the way data is stored and backed-up or recovered is fully understood. It's something that's brushed under the carpet until things go wrong! Interestingly, I attended a lecture on storing images. The speaker worked in conjunction with museums etc. There conclusion was that the best way in there scenario to archive photographs was printed output!

Cheers for now

Gary

I think there's a lot of "too technical for me" going around. At the coal face though I can assure everyone that "digital" is far from perfect; we're getting just as many problems in 2014 that we got in the 80s ... they're just different problems.

[Colin Southern]

Horses for courses really.

Had LR not been around when i re started togging then i may have used bridge but as a beginner LR is attractive because it give everything you need to start out with. having said that ive just checked the file structure and all of my images are stored in the same place by light room in the DNG format and ive just opened bridge and it found all of my files with edits, so ill not worry about it.

So long as you have an offsite copy (at a minimum) you should be good to go then. If not, just sign up with Carbonite and upload through your sat connection!

[Mark von Kanel]

So long as you have an offsite copy (at a minimum) you should be good to go then. If not, just sign up with Carbonite and upload through your sat connection!

The 10mb connection has been withdrawn at 20,000 euro a month they started to fidget! we are down to 2mb MIR 512 CIR now..... but i have been taking steps with my main catalog at home, im now raid 5 and will look at cloud stuff for paying customers as im now starting to get some!

We used to back the ships server up via VSAT but it didnt work so we gave up! belive it or not i am only just installing a backup NAS now and the chief engineer has insisted that it go in the rack room with the server........ Ive begged him to let me put another in a different part of the ship, but so far no deal.....let just wait for the fire eh?

[ajohnw]

It is possible to produce rock solid data bases but there is little likely hood of finding one in Lightroom or any similar package. I'd hazard a guess that they wont be found in pc based smaller company accounts packages either but on the other hand banks and it isn't just a matter of saying oh well we will fit a back up power supply etc.

The real bottom line is that software can only be as good as what is put in it followed by the fact that people make mistakes even the people who write the software that is used to produce software. Even the chip it runs on can have mistakes in it.

John
-

[Colin Southern]

The 10mb connection has been withdrawn at 20,000 euro a month they started to fidget! we are down to 2mb MIR 512 CIR now..... but i have been taking steps with my main catalog at home, im now raid 5 and will look at cloud stuff for paying customers as im now starting to get some!

Just remember that even a RAID 5 setup can still lose everything in the event of a fire (seen it) - flood (seen it) - virus (seen it) - theft (seen it) - "finger-trouble" (seen it) - power-spike (seen it) ...

We used to back the ships server up via VSAT but it didnt work so we gave up! belive it or not i am only just installing a backup NAS now and the chief engineer has insisted that it go in the rack room with the server........ Ive begged him to let me put another in a different part of the ship, but so far no deal.....let just wait for the fire eh?

I was contracted to document the aftermath of a ship fire. Scary stuff. Distance is about your only hope. The acidic smoke gets in to absolutely everywhere.