Are lens reviews reliable?

[Panama Hat & Camera]

Ted, thanks for your comments!

Any test of a single lens is meaningless in terms of Quality Assurance.

I agree with you.

Roger Cicala of LensRentals is the only tester I've read that publishes tests of multiple lenses (of the same model) in statistically significant quantities.

He is doing the correct thing!

I suppose the only alternative is to find many separate reviews of the same lens model and to analyze like parameters, e.g. MTF50, rather than subjective opinions ... good luck with that, eh?

When I read the review of PCMagazine, I had already purchased my 18-300mm f/3.5-6.3 lens. I realized that aperture recomendations of PCMagazine (medium to small apertures) for my lens are exactly the opposite of aperture recomendations of DxOMark and PhotographyBlog (large to medium apertures).
So, I started looking for other tests and, to my surprise, I realized that there were different recommendations for the best aperture. Because of this, I decided to study the subject and post the thread.
The MTF (with Imatest software) was used by PCMag, ephotozine and Lenstip (but with different results).
I will do my own tests using various apertures and focus distances to find out what is best for me.
Regards,
Antonio.

https://www.lensrentals.com/blog/201...h-translation/
https://wordpress.lensrentals.com/bl...s-test-either/

[Manfred M]

I realized that there were different recommendations for the best aperture.

This should tell you that there is no "best aperture"...

It is the aperture that happens to work for the shot you are taking for a given subject, ISO setting and shutter speed.

There are "issues" like maximizing or minimizing depth of field that come into play. People don't buy "fast glass", which is usually viewed as a maximum aperture of f/2.8 and faster for zoom lenses and f/1.4 and faster for fixed focal length lenses) to shoot at f/8 or f/11, even if they are sharpest at those apertures. People buy "fast glass" so that they can shoot wide open. In fact lens designers understand this and make the assumption that this is how people will use those lenses.

Lens test results never seem to indicate the distance to the object being tested to get the sharpest results. This might be part of the reason for the different sites reporting different results. In my view, there are too many variables here for most lens test results to be meaningful.

Unless you are shooting on a very sturdy, heavy duty tripod (rather than hand-holding) or using very high shutter speeds, you will never get the best performance out of the high end lenses because there will be some softening due to camera movement, even when using image stabilization.

Finally, unless you do large prints, you will never be pushing the lens limits either because when you down sample so that they display on the standard 1920 x 1080 HD capable computer screen, you will have thrown away most of the data anyway. Unless you are printing at least A2 / 17" x 22" size on a full-frame camera, don't worry about it...

[Panama Hat & Camera]

I'm afraid I'm a sceptic when it comes to this type of information. I am very much of the same opimion as Ted and Trev.

I tend to the view that my shooting technique, particularly as I get older is probably the most significant issue when it comes to the quality of my images.
When it comes to 'choosing' a new lens, I prefer to simply try it. The companies I generally buy from are quite willing to give me an hour or two to try, and I have on occasion hired a lens for the day for a trial.

And, truth be told, despite having a science/technical past (Chemist/Biochemist/IT), I'm not sure I really understand what the variances in the technical measurements mean in reality...... One photographer's blur is arguably another's good bokeh so ......

James, thanks for your comments!
You are right!
Please click on the link https://theonlinephotographer.typepa...are-wrong.html and read the article "All Lens Tests are Wrong".
Regards,
Antonio.

[Cogito]

Antonio,in truth, the only way to evaluate a lens is to use it! If you can't get to a lens retailer then your only option is to buy from a reputable online store. Then if the lens is not to your satisfaction, return it!

[Panama Hat & Camera]

This should tell you that there is no "best aperture"...

It is the aperture that happens to work for the shot you are taking for a given subject, ISO setting and shutter speed.

There are "issues" like maximizing or minimizing depth of field that come into play.

In the tests cited in my thread, the depth of field was not considered on the analysis of sharpness of the images.

Lens test results never seem to indicate the distance to the object being tested to get the sharpest results. This might be part of the reason for the different sites reporting different results. In my view, there are too many variables here for most lens test results to be meaningful.

You are right! It makes no sense to assume that the sharpness of the lens at short shooting distance is the same at infinity.
Cheers,
Antonio.

[Panama Hat & Camera]

I personally like opticallimits.com for lens reviews - good combination of objective tests and hands-on experience.

Raphael, thanks for your comments.
I agree with you. I will do my own tests using various apertures to find out what is best for me.
Regards,
Antonio.

[Panama Hat & Camera]

Tony, thanks for your comments.
I agree with you. I will do my own tests using various apertures to find out what is best for me.
Greetings,
Antonio.

[Manfred M]

In the tests cited in my thread, the depth of field was not considered on the analysis of sharpness of the images.

In my personal work, depth of field considerations, whether I am looking for shallow depth of field and a soft background or an image where everything is sharp is probably 80% - 90% or more what drives the aperture. The rest of the time I am either looking to blur or freeze motion and depth of field considerations are less critical.

That is why I consider your comment a bit amusing. Sharpness is what I concentrate on in almost all my work.

[george013]

Dof can't be an issue in testing the sharpness of a lens. Dof is by definition BLURRING but of an acceptable value depending on various factors.

George

[rpcrowe]

I totally agree with Manfred's comments regarding the need for many reviewing sites and especially YouTube blogger reviewers to present their findings positively so that they will continue to be in the pipeline of gear to review.

However, along with that concept, both reviewers and bloggers need hits on their sites to keep alive. Sometimes they can get hits by echoing the general reviews of other sites and bloggers - especially if they hone in on a very sensitive area. Such as all the reviewers did on the reputed poor dynamic range performance of the Canon 6D Mark ii when it was introduced.

Another example of the "jumping onboard the sinking boat" is that virtually every reviewer and blogger complained that if you used the flip-up screen of the Sony A6500 and mounted an external microphone on the hotshoe, the mike would block the flippy screen. That has been solved very simply and inexpensively by using a small bracket for the mike. There is now a plethora of brackets available for this use so the original complaints of the reviewers and bloggers are no longer valid.

There are also some reviewers and bloggers who are prejudiced towards or away from certain brands of equipment. They are generally called Canon, Nikon, or Sony Fanboys!

[Manfred M]

Dof can't be an issue in testing the sharpness of a lens. Dof is by definition BLURRING but of an acceptable value depending on various factors.

George

Agreed George, but you have definitely missed the point here.

Many people seem to think that sharpness is the only attribute that they need to look at when judging a lens. Unfortunately, you are not alone in this rather one-dimensional analysis of lens performance. People that understand photography tend to take a much more nuanced view and look at much more than absolute sharpness when they look at which lens would work best for them.

Lens tests are meaningful in establishing these sharpness parameters when used in a lab situation using a 2-dimensional test target. Image stabilization is turned off. I can't think of anything less representative of "real world" shooting unless you are going to spend most of your time reproducing flat art work. Yes, the tests identify the theoretical maximum sharpness of the specific lens being tested. To get a statistically significant view a large number of lenses, of the same model, would have to be tested. That simply does not happen so trying to extrapolate the results to any specific lens is simply not meaningful. To go a step further we might also suspect that the lens manufacturers might be "cherry picking" the lenses being sent to the major lens test sites to ensure a high test score.

Going a step further, real world photography is generally done using 3-D subject matter and in non-ideal conditions. Most people hand-hold their cameras and when I see people shooting with tripods, most are not what I would call "sturdy". I have done side-by-side comparisons of pictures taken with "kit lenses" and pro lenses (using the same shooting parameters; focal length, ISO, shutter speed) in these less than ideal situations and from a pure sharpness standpoint, the images are often indistinguishable at 100% magnification, especially when shot at f/8 or f/11. Other factors are noticeable and I will get to those.

When knowledgeable photographers buy "fast glass" (I define what I mean by that in #22), they are planning to shoot those lenses "wide open". If you want to shoot at the sweet spot range of f/8 - /f11, save yourself money and go with low end lenses and you will get sharp images. If you are interested in soft, interesting out of focus areas (i.e. bokeh) and are not overly concerned with absolute sharpness, then you will be in the market for lenses that do this and will be shooting at or close to wide open.

The problem is that bokeh is difficult to quantify and words like "smooth" or "creamy" are harder to quantify than the number of lines per mm, this stuff gets mentioned, but only in passing. Part of the reason that so many people are picking up some of the old manual lenses from the 1960s and 1970s are specifically for those aesthetics. These will generally give far more interesting images than the ultra sharp lenses that the manufacturers are producing today.

[george013]

Agreed George, but you have definitely missed the point here.

Many people seem to think that sharpness is the only attribute that they need to look at when judging a lens. Unfortunately, you are not alone in this rather one-dimensional analysis of lens performance. People that understand photography tend to take a much more nuanced view and look at much more than absolute sharpness when they look at which lens would work best for them.

Lens tests are meaningful in establishing these sharpness parameters when used in a lab situation using a 2-dimensional test target. Image stabilization is turned off. I can't think of anything less representative of "real world" shooting unless you are going to spend most of your time reproducing flat art work. Yes, the tests identify the theoretical maximum sharpness of the specific lens being tested. To get a statistically significant view a large number of lenses, of the same model, would have to be tested. That simply does not happen so trying to extrapolate the results to any specific lens is simply not meaningful. To go a step further we might also suspect that the lens manufacturers might be "cherry picking" the lenses being sent to the major lens test sites to ensure a high test score.

Going a step further, real world photography is generally done using 3-D subject matter and in non-ideal conditions. Most people hand-hold their cameras and when I see people shooting with tripods, most are not what I would call "sturdy". I have done side-by-side comparisons of pictures taken with "kit lenses" and pro lenses (using the same shooting parameters; focal length, ISO, shutter speed) in these less than ideal situations and from a pure sharpness standpoint, the images are often indistinguishable at 100% magnification, especially when shot at f/8 or f/11. Other factors are noticeable and I will get to those.

When knowledgeable photographers buy "fast glass" (I define what I mean by that in #22), they are planning to shoot those lenses "wide open". If you want to shoot at the sweet spot range of f/8 - /f11, save yourself money and go with low end lenses and you will get sharp images. If you are interested in soft, interesting out of focus areas (i.e. bokeh) and are not overly concerned with absolute sharpness, then you will be in the market for lenses that do this and will be shooting at or close to wide open.

The problem is that bokeh is difficult to quantify and words like "smooth" or "creamy" are harder to quantify than the number of lines per mm, this stuff gets mentioned, but only in passing. Part of the reason that so many people are picking up some of the old manual lenses from the 1960s and 1970s are specifically for those aesthetics. These will generally give far more interesting images than the ultra sharp lenses that the manufacturers are producing today.

I'm not missing the point. I'm just saying dof isn't a property of the lens. It's a value that's related with the output and viewing conditions.
I don't need much more words for that

George

[Manfred M]

I'm not missing the point. I'm just saying dof isn't a property of the lens. It's a value that's related with the output and viewing conditions.
I don't need much more words for that

George

The shooting aperture and how much of the subject fills the frame at capture are important inputs to DoF. Viewing distance impacts the Circle of Confusion, so it is a factor too; but the capture side is more important.

There is a great photographic site found on the internet called "Cambridge in Colour" that has a very good explanation of DoF.



https://www.cambridgeincolour.com/tu...h-of-field.htm

[george013]

The shooting aperture and how much of the subject fills the frame at capture are important inputs to DoF. Viewing distance impacts the Circle of Confusion, so it is a factor too; but the capture side is more important.

There is a great photographic site found on the internet called "Cambridge in Colour" that has a very good explanation of DoF.



https://www.cambridgeincolour.com/tu...h-of-field.htm

Manfred, go back to post 25 of Antonio.
This thread is about lens reviews. Dof is a range of out of focus parts of the image that's still acceptable for a certain output and viewing conditions. And everybody knows that for making a photo you need a camera with a lens etc. etc.
Is it so difficult to understand simple things?

George

[xpatUSA]

... how much of the subject fills the frame at capture are important inputs to DoF ...

OT but could you please expand on that, Manfred? I can't find anywhere that says DOF is directly dependent on Framing.

Is it to do with shooting distance, or perhaps magnification?

Thanks ...

[Manfred M]

Manfred, go back to post 25 of Antonio.
This thread is about lens reviews. Dof is a range of out of focus parts of the image that's still acceptable for a certain output and viewing conditions. And everybody knows that for making a photo you need a camera with a lens etc. etc.
Is it so difficult to understand simple things?

George

It is about lens reviews but part of the issue that I (and others) have views regarding lens reviews is how good / relevant / thorough / consistent they are.

One area that "good" reviews cover is bokeh, which is important to most people that are interested in "fast" lenses. Some create pleasing bokeh and others have bokeh that is intrusive and not very appealing. These are characteristics of the lens design itself, not a random outcome. Bokeh is closely related to depth of field as it is a description of the appearance of the out of focus areas. Leica lenses have traditionally had two features that users loved about them; high resolution (effectively very good micro-contrast) and extremely beautiful (generally described as "smooth and creamy") bokeh.

Other than mentioning the term, CiC does not cover bokeh in its tutorials. One of the best articles I know on the subject is found at Luminous Landscape, which has been a pay site for a few years. At $1 US per month it is the biggest photographic bargain out there. Dr Harold Merklinger's article on bokeh can be found here, if you are a site member.



https://luminous-landscape.com/redis...tanding-bokeh/



Once again you do not seem to understand certain aspects of photography and rather than trying to understand it you make things up. Please don't do that as it does nothing for your credibility.

[Manfred M]

OT but could you please expand on that, Manfred? I can't find anywhere that says DOF is directly dependent on Framing.

Is it to do with shooting distance, or perhaps magnification?

Thanks ...

Try the CiC article I pointed out. Framing of the subject is what this is all about, so as long as the subject takes up the same area of the frame (for a given aperture) the DoF will be more or less the same. Focal length of the lens will affect the out of focus areas in front of and behind the subject so that is a variable.

https://www.cambridgeincolour.com/tu...h-of-field.htm

I've also found that bokeh is more apparent with a longer lens, so magnification plays a part in the prominence of the bokeh.

[xpatUSA]

Try the CiC article I pointed out. Framing of the subject is what this is all about, so as long as the subject takes up the same area of the frame (for a given aperture) the DoF will be more or less the same.

Sorry, Manfred, I must disagree and the the link seems not to address my concern.

For example, if I equally frame objects of different sizes e.g. 2m and 1m, the DOFs are different by a factor of 5, which is not "more or less the same":





Obviously, there must be a misunderstanding, perhaps about "same area of the frame"?

Or, by "the subject" did you mean the same subject situated at the same distance?

[dem]

For example, if I equally frame objects of different sizes e.g. 2m and 1m, the DOFs are different by a factor of 5...

Obviously, there must be a misunderstanding, perhaps about "same area of the frame"?

What Manfred said is true for the same subject and only when dof is shallow (when focusing distance is much shorter than hyperfocal as in macro, portraits, wildlife). Same framing means same magnification.

[george013]

Sorry, Manfred, I must disagree and the the link seems not to address my concern.

For example, if I equally frame objects of different sizes e.g. 2m and 1m, the DOFs are different by a factor of 5, which is not "more or less the same":





Obviously, there must be a misunderstanding, perhaps about "same area of the frame"?

Or, by "the subject" did you mean the same subject situated at the same distance?

If you double the focal length and double the focus distance you will have the same framing of the subject.
https://www.cambridgeincolour.com/tu...h-of-field.htm, chapter CLARIFICATION: FOCAL LENGTH AND DEPTH OF FIELD.
To me completely off topic.

George