Facinating lens-to-lens image quality comparison tool.

[Manfred M]

Larry - you make a good point that I would like to expand on a bit. Virtually every test I read result seems to be based on shooting with a single lens. That in itself is a flawed methodology; we don't know if the manufacturer has suppled a "cherry picked" that has been selected because it is just about "perfect" and is unlikely to end up in an average user's hands.

Conversely, we don't know if the opposite is true and the distributer has sent over a real "dog" that just sneaks in under the wire as to being acceptable. A single point on a chart is somewhat meaningless, yet people make all kinds of value judgements based on that single data point...

A real test would involve comparing a number of different identical units from different manufacturing runs, to see the variance between different units. I find some of Roger Cicala's postings (LensRentals.com) interestng because he will sometime mention the differences he sees between different units of the same lens model out of the rental fleet.

[NorthernFocus]

Ok, it sounds like there are quite a few people arguing against the use of test data from sites like the one linked here. So where does one start in the search for information to make a decision on a lens selection. I'm in the middle of making a decision on a 28mm lens for Nikon FX format. There are at least a dozen options and that isn't even considering older models. So how does one decide? Narrow the field a bit, I'm not willing to spend over $1000 on it. OK, what's next? I'll be shooting a high resolution camera. In the three classically discussed performance parameter, my order of importance for my style of photography in this range are sharpness, CA, and distortion. I couldn't really care about vignette. So what's next. How do I narrow it down. Or more accurately, how would you folks narrow it down?

[Manfred M]

How do I narrow it down. Or more accurately, how would you folks narrow it down?

1. Budget - how much am I willing to spend?

2. Zoom or fixed? If zoom, what is the range. I would look at where this fits with the rest of my lenses. I personally go for lenses that have less of a range rather than more.

3. Maximum aperture - I tend to go for fast glass. On a zoom I tend to prefer a lens that has the same aperture across the entire zoom range.

4. Model age - Nikon's newest lenses tend to be the best optically, but probably the worst mechanically.

5. Internal focus motor (G-type) or older screw drive? At 28 mm I might look at a D model with screw drive. At 28mm I might consider a manual focus lens too (I probably would not buy one that is not AF).

6. VR? Again - depends on use, but I would probably look at a non-VR model on a wide-angle lens.

7. OEM or third party - I tend to stick with Nikkors because some of the third party manufacturer's lenses do not age well and may not update lens firmware to handle newer camera bodies that come out after the lens has been discontinued. Nikon (so far) seems to manage backwards compatibility.

8. Lens construction - virtually all of my lenses have the rear dust seal. Older lenses are often better built; although this is not necessarily true for Sigma and Tamron, as a lot of their older glass was built to a lower price point. Some of the newer glass seems to have a better build. I find Tokina will often exceed OEM build quality.

9. Filter size - I prefer lenses with 77mm filter threads. I don't like step-up rings as they are sometime difficult to remove (binding issues). At 28mm, filters should not be a problem.

10. If I still can't decide between a couple of candidates, I might look as some reviews.

[Brev00]

Dan,

I didn't mean to completely argue against lens testing data. I think it is worth reading and seeing if there is some kind of consensus. I do like to look at user reviews on Amazon and B & H. It is nice if there are a number of reviews. Then, I look at the stars. More important than the average is the break down. Lots of 1's and 2's with people reporting failures is a pretty bad sign. Sometimes, users are more appreciative of a lens than the critics. I can also see what level the user is at and put some weight on those who seem to be knowledgable. I have never been in your boat. Most of the time, there are maybe two main competitors. The one exception was when I bought my wide angle lens. I rented one and was not impressed. I bought another and it seemed like a bad copy. Returned it. The process of elimination was my method as I eventually got the Tokina 12-24. So, renting is good if possible. Buying if unsure is not so bad if you can return it. Looking at flickr group streams has been helpful for me, too. I have discussed lenses at DPreview with some members there. There are some pretty good photographers there. One exceptional photographer used the Tokina so that was helpful. I also read photo magazines like NPhoto which review lenses. They often show charts but also give anecdotal information about handling and stuff like that. So, there really is a lot to draw on, but, in the end, it is also a matter of intuition.

[NorthernFocus]

Manfred, so no consideration for performance? That simplifies things to be sure. By your criteria it narrows it down to the only current Nikon 28mm. The only incentive to consider older used Nikkors would be if they performed better than the newer model. But how would one know anyway? So easy peasy. Buy the one model available.

[Manfred M]

Dan - With the higher end glass I tend to buy (FX) I generally don't get too wrapped up in the actual performance, as I assume it will pretty reasonable. I tend to look at features I need first. In fact, I can't remember any review where they actually say anything terrible about a lens...

Kind of like buying a car. I will have a maximum price point in mind, and then will look at features. The very last thing I will research is the "performance", because I won't buy something that doesn't have the features I'm looking at. If it has the features I want, and the reviews tell me that the vehicle is terrible, then back to the drawing board. On the other hand, if it has the features and the reviews are decent, I will go for it. I've never bought anything because of the reviews; the device, whether it is car, phone, camera or lens has to meet my needs from a features standpoint, first and foremost.

[NorthernFocus]

Dan - With the higher end glass I tend to buy (FX) I generally don't get too wrapped up in the actual performance, as I assume it will pretty reasonable. I tend to look at features I need first. In fact, I can't remember any review where they actually say anything terrible about a lens...

Kind of like buying a car. I will have a maximum price point in mind, and then will look at features. The very last thing I will research is the "performance", because I won't buy something that doesn't have the features I'm looking at. If it has the features I want, and the reviews tell me that the vehicle is terrible, then back to the drawing board. On the other hand, if it has the features and the reviews are decent, I will go for it. I've never bought anything because of the reviews; the device, whether it is car, phone, camera or lens has to meet my needs from a features standpoint, first and foremost.

I think we're on the same page. Nowadays there are very few lenses produced by the reputable companies that are pure crap. So at the end of the day, performance differences can be minutia compared to other considerations. Minutia that is likely obscured by technique anyway.

But every once in a while, a situation does arise wherein objective performance data can help make the decision complicated. As in trying to evaluate whether the performance difference in a prime lens or a zoom that covers the same range is meaningful. One thing that's for sure, getting picky gets expensive...

[RichardTaylor]

As a hobbyst who mostly publishes to thew web and never prints abpove 10x8 (or A4)
What I do when looking for a new lens (and it has been since 2009 since I purchased the last one).
#1 What do I need (not want).
What am I going to use it for?
This leads to focal lenght(s), maximum aperture, mechanical performance (especially how it focuses), and possibly weight.

#2 What is my budget?
Sometimes this may need to be revised.

#3 Then I read the reviews (but not usually the super technical ones) including user reviews, and talk to other photographers.

#4 Then I will make a decision, however I will have a definite leaning towards Canon (I am a Canon shooter) lenses.

I agree that technique, and atmospherics etc, is usually the deciding factor in the IQ at least in a lot of my photographs.

One thing is that buying "cheap" is not always the best investment in the long run. This is from personal experience in looking for a long tele zoom. However buying your first "expensive" lens can be big pill to swallow.

[Black Pearl]

1. Budget - how much am I willing to spend?

2. Zoom or fixed? If zoom, what is the range. I would look at where this fits with the rest of my lenses. I personally go for lenses that have less of a range rather than more.

3. Maximum aperture - I tend to go for fast glass. On a zoom I tend to prefer a lens that has the same aperture across the entire zoom range.

4. Model age - Nikon's newest lenses tend to be the best optically, but probably the worst mechanically.

5. Internal focus motor (G-type) or older screw drive? At 28 mm I might look at a D model with screw drive. At 28mm I might consider a manual focus lens too (I probably would not buy one that is not AF).

6. VR? Again - depends on use, but I would probably look at a non-VR model on a wide-angle lens.

7. OEM or third party - I tend to stick with Nikkors because some of the third party manufacturer's lenses do not age well and may not update lens firmware to handle newer camera bodies that come out after the lens has been discontinued. Nikon (so far) seems to manage backwards compatibility.

8. Lens construction - virtually all of my lenses have the rear dust seal. Older lenses are often better built; although this is not necessarily true for Sigma and Tamron, as a lot of their older glass was built to a lower price point. Some of the newer glass seems to have a better build. I find Tokina will often exceed OEM build quality.

9. Filter size - I prefer lenses with 77mm filter threads. I don't like step-up rings as they are sometime difficult to remove (binding issues). At 28mm, filters should not be a problem.

10. If I still can't decide between a couple of candidates, I might look as some reviews.

Give that man a round of applause for possibly the best post I've seen on CiC.
There are so many considerations to choosing a lens that the finite, pixel level performance is virtually insignificant.

[Black Pearl]

1. Budget - how much am I willing to spend?

2. Zoom or fixed? If zoom, what is the range. I would look at where this fits with the rest of my lenses. I personally go for lenses that have less of a range rather than more.

3. Maximum aperture - I tend to go for fast glass. On a zoom I tend to prefer a lens that has the same aperture across the entire zoom range.

4. Model age - Nikon's newest lenses tend to be the best optically, but probably the worst mechanically.

5. Internal focus motor (G-type) or older screw drive? At 28 mm I might look at a D model with screw drive. At 28mm I might consider a manual focus lens too (I probably would not buy one that is not AF).

6. VR? Again - depends on use, but I would probably look at a non-VR model on a wide-angle lens.

7. OEM or third party - I tend to stick with Nikkors because some of the third party manufacturer's lenses do not age well and may not update lens firmware to handle newer camera bodies that come out after the lens has been discontinued. Nikon (so far) seems to manage backwards compatibility.

8. Lens construction - virtually all of my lenses have the rear dust seal. Older lenses are often better built; although this is not necessarily true for Sigma and Tamron, as a lot of their older glass was built to a lower price point. Some of the newer glass seems to have a better build. I find Tokina will often exceed OEM build quality.

9. Filter size - I prefer lenses with 77mm filter threads. I don't like step-up rings as they are sometime difficult to remove (binding issues). At 28mm, filters should not be a problem.

10. If I still can't decide between a couple of candidates, I might look as some reviews.

Give that man a round of applause for possibly the best post I've seen on CiC.
There are so many considerations to choosing a lens that the finite, pixel level performance is virtually insignificant.

[Colin Southern]

Frankly, I think they'd do the world a favour if they posted down-sampled results of the entire image and not 100% crops.

As Lance Armstrong said, "it's not about the bike". Ok, so in his case it turned out to be more about the drugs, but hopefully at least one person gets my point!

I'm sure a lot of work has gone into that comparison site, but in my opinion it's next to worthless because ultimately it's encouraging people to "major in minor things". The image quality difference between any two lenses at 100% magnification is pretty much the LAST thing people should be giving weight to when making a lens purchasing decision. And if people can't see that then we're all doomed.

Quite depressing actually.

[Colin Southern]

There are so many considerations to choosing a lens that the finite, pixel level performance is virtually insignificant.

Robin, you get my (seldom awarded) "this man gets it" award for today.

[ajohnw]

Frankly, I think they'd do the world a favour if they posted down-sampled results of the entire image and not 100% crops.

As Lance Armstrong said, "it's not about the bike". Ok, so in his case it turned out to be more about the drugs, but hopefully at least one person gets my point!

I'm sure a lot of work has gone into that comparison site, but in my opinion it's next to worthless because ultimately it's encouraging people to "major in minor things". The image quality difference between any two lenses at 100% magnification is pretty much the LAST thing people should be giving weight to when making a lens purchasing decision. And if people can't see that then we're all doomed.

Quite depressing actually.

I would say it's more depressing that manufacturers churn out more and more pixels and may not turn out lenses to match and that people are prepared to accept that and often show no interest in the fact. Also that a few people can't accept that some have interests and views that differ from their own. Tongue in cheek comments because all people look at what their gear produces who ever they are. If they didn't they would have no idea if the above comments were fact or fiction so had best shut up.

For some tests results are part of their interest. Some go a little further and do pointless things like compare older models with new for no other reason than curiosity. Pointless because it wont do anything about how lenses are. They will remain as they are. The signs are that they get better and then get steadily worse at various times in their history. That is part down to what people buy and also what they use there cameras for. Many lenses and cameras are complete over kill for what many people actually use them for. Trouble is that there tends to be a big jump in quality if some one strays too far back from them and enters the world of only being interested in small prints and moderate views on a pc screen.

Having used cameras and lenses for rather a long time one of the facts that I find most interesting is price. The Japanese turned a number of optical things into more mass market items items rather than let them remain mostly in the realms of the rich. They didn't do that by retaining the same quality and life. Personally I was glad. Just like many many other people I could afford to buy them. The same process is always going on in many areas to keep prices down and profits up.

Personally I feel it's best to live with the real world and there is no harm in taking close looks at what they churn out. It's rather difficult to see why any one should object to that.

I also have an attitude about so called kit lenses and people often being keen to get rid of them. It would be a bit stupid of manufacturers to churn out items that come with cameras that will give disappointing results. In general they don't but increasingly with time software will be curing more and more problems. Same on dearer lenses too.

The other thing that makes me roll about laughing is a refusal to accept the amount of effort manufacturers put into producing quality jpg's straight out of the camera. They do this because that is where they see the market is. Us lot in some respects don't really count. I can quote a comment from some one who spends time hacking Canon firmware about the bits that they don't hack - There are so many ifs and buts that people don't stand a cat in hells chance of matching it. Then there are people about who correctly reckon that the main thing about a camera is to go out and use it. Then they give the impression that people must use raw dooming them to a rather steep learning curve or various most of the work done by the package adjustments. Cameras are always getting better and better at doing this sort of thing themselves. Modern jpg's can most definitely be titivated. The extra stops myth in many respects goes back to older cameras.

Rant over.

John
-

[NorthernFocus]

I find it rather fascinating that all of the negative comments about the site (and others) which provide quite good technical data are actually comments about human nature and about saving photographers from themselves. While I appreciate everyone's concerns on our collective behalf, I for one rather enjoy exercising at least some degree of self determination.

Frankly, I think they'd do the world a favour if they posted down-sampled results of the entire image and not 100% crops....

If one lives in the world of web resolution images and small prints, I completely agree. But in my world I find full frame images reduced to we resolution to be virtually useless for evaluating equipment.

[NorthernFocus]

...The extra stops myth in many respects goes back to older cameras...

Care to enlighten us on said myth?

[ajohnw]

Care to enlighten us on said myth?

I did use the word mostly. I'm glad some one else seems to have a sane attitude to information by the way.

Much of the myth goes back to Canon who for some reason best known to themselves had the habit of cutting of significant amounts of the highlight end in jpgs and more was available from raw. If some one goes though Canon camera reviews on dpreview they will find comments to that effect on many especially as the cameras go back in time. They even often show this via shots from raw. Sneaky people though - they generally choose the shot carefully and don't show any dark end.

I'd guess they went this way because there dark end performance was so good. People have formed the opinion that they get down to the sort of noise levels their cameras produced by using dark frames. They don't. It's was purely down to the sensor. Also if the camera does capture the light end it's presentation in the jpg will be natural. Things aren't so simple when other arrangements are used as they are in most cameras these days 12 stops into an 8 stops colour space just doesn't fit. Not that the camera really has 12 or the display 8. Both suffer at the dark end.

Over time more and more work has been done by mapping more and more stops into jpg's. A pretty typical figure these days is 9 stops with the facilities to have even more on many. One I have seen manages the best part of 11. The jpg's can sometimes be "bent" to extend into the dark end as well. They do this by playing with tone curves just as people do when developing raw one way or another. They may use what Adobe think is best, something they concoct themselves or what ever. The PP challenges that are left from either raw or jpg are often very similar. Often a jpg is likely to be simpler or given some peoples opinions of what is a good range of light levels to shoot in they may even be near perfect.

Basically I think people are inclined to stick to what they do and don't notice change. All sorts of things have changed. Noise is no longer as it usually was, some cameras will produce fashionable HDR straight out of the box. It seems others will process the dark end for you. In fact there are so many facilities kicking around the jpg area in cameras that it would take a significant amount of time to find out which to use for what.

And photography supposed to be about framing and choosing the shots etc.

John
-

[Colin Southern]

But in my world I find full frame images reduced to we resolution to be virtually useless for evaluating equipment.

And I find evaluating equipment in that context to be a virtually useless contribution to photography. I've been trying to think of a meaningful comparison -- the best I've come up with is "evaluating 100% crops as an comparative indication of lens performance is about as meaningful to photography as chemical analysis of different brands of petrol is to a cars performance". Yes - in theory in makes a difference, but in practice is makes no significant difference what-so-ever.

IMO opinion the encouragement of the over-analysis of specs by the public general is one of the biggest modern-day hoaxes around. I just wish more would stand up and say "the emperor has no clothes".

[ajohnw]

I much prefer 400% crops I have done myself but if buying I take any info I can find. I also right click to see what has been used to process images as some sites really do take liberties in that area. At least they show that great results can be produced from what might be a mediocre lens - if you know how. Some do give straight shots.

Pixel Peepers is a popular site but again it pays to right click. Just google pixelpeepers lens model. Much used by bird photographers - hardly surprising really. I pixel peep to see how much I can crop.

Another very useful one is googling lens model tests and looking at images - some one some where with no axe to grind will have probably put it through it paces.

Sites like this one aught to be long gone surely but actually I think it gets a fair few visits.

http://www.normankoren.com/Tutorials/MTF5.html

Also googling resolution targets always brings up lots of results. I gave up going that far when certain UK magazines gave up providing them for free from time to time but others haven't.

Out of interest I just googled Nikon "18-105mm" DX and picked 3 from the web

http://imaging.nikon.com/lineup/lens..._35-56g_ed_vr/

http://www.photozone.de/reviews/410-nikkor_18105_3556vr
http://www.ephotozine.com/article/ni...s-review-15049

The Nikon one shows how useless MTF charts are and the difference between the 2 technical ones are interesting

I then switched to an image search and found this one - truly useful even though it's just images

http://www.cameralabs.com/reviews/Ni..._results.shtml

and yet another which is still useful as there are a wide variation of tones.

http://www.whatdigitalcamera.com/equ...html#slideshow

There will be many others. I'm stuck with the lens now anyway. Basically it's not TOO bad like many. That is why I bought it with a fair amount of confidence and hopefully I will be lucky as lenses do vary full stop.

John
-



Like KR I take manufacturers MTF charts with a pinch of salt. Canon dream and Nikon measure - a lens, as do others. Pity no one provides each lens with one for that very same lens. It wouldn't cost them much to do just that.

[Colin Southern]

Heck - when I bought my Canon EF70-200mm F2.8L IS USM II, I did it without consulting a single MTF chart or looking at any 100% crops. It's been great - love it - would buy it again in a heartbeat.

But now I'm not too sure - I saw a 100% crop that suggested another lens might be slightly sharper - and to make matters worse it was on a Nikon. So now I'm going to have to consider selling up everything and switching. But what if I do that and then Canon makes a better one .

Signed, lost and confused.

[NorthernFocus]

And I find evaluating equipment in that context to be a virtually useless contribution to photography. I've been trying to think of a meaningful comparison -- the best I've come up with is "evaluating 100% crops as an comparative indication of lens performance is about as meaningful to photography as chemical analysis of different brands of petrol is to a cars performance". Yes - in theory in makes a difference, but in practice is makes no significant difference what-so-ever.

IMO opinion the encouragement of the over-analysis of specs by the public general is one of the biggest modern-day hoaxes around. I just wish more would stand up and say "the emperor has no clothes".

I envy you, Colin. Or more appropriately curse myself for getting more liberal minded with age. Life was so much simpler when I didn't waste time considering others' perspectives. But your meaningful comparison is a good one regarding the petrol. I've always thought it rather absurd that performance auto drivers think they have to burn higher octane gasoline than I do on my daily commute to the office. For that matter why do they have to drive different cars than us at Indy or LeMans? But in that context I have to voice a difference of opinion. Your example of the spec analysis hoaxe pales in comparison to the billions spent on the meaningless boost in octane for which motorists pay a premium every year. And don't even get me started on diesel versus gasoline engines...