Are lens reviews reliable?

[Panama Hat & Camera]

As I wrote in the thread A DSLR superzoom camera, my decision of buying the lens Nikkor 18-300mm f/3.5-6.3 was influenced by the reviews published by DXOMARK, Photographyblog and Ken Rockwell.
These reviews indicated to me that the Nikkor 18-300mm f/3.5-6.3 was sharper than the Nikkor 18-300mm f/3.5-5.6.
I discovered recently other reviews with the same opinion, but I found another review that told me otherwise: PCMag.
I can not compare the two lenses, so I decided to compare the reviews on the lens I have (mainly the analyses on the resolution of the image). I noticed that recomended apertures and opinions varied from review to review. The methods of avaliation used in the reviews also varied:
- Imatest was used by PCMag, ephotozine and Lenstip (but with different results);
- DxO procedures was used by DxOMark;
- Photo evaluation of a real world subject (total frame area and 100% crops) was used by Photography Blog.
The results were presented in different units (lines, lpmm, P-Mpix) but I don't know how to do the correspondence between lines (or lpmm) and P-Mpix.
The spreadsheet bellow summarizes the results from the reviews [the DxO data values have a specific column of apertures (f-numbers)].
The yellow cells corresponds to aperture that each review recommends for each focal lenght.
The text and numbers in bright yellow corresponds to apertures recommended by the reviews.
The text and numbers in light yellow corresponds to secondary apertures recommended by the reviews.
The text and numbers "between quotes" corresponds to my readings or my interpretations of the results of the reviews.
The values ​​in light red are below the minimum acceptable quality.
The data values of DxO were obtained from the graphs of profiles (see figure).





https://www.pcmag.com/review/352400/...3-5-6-3g-ed-vr
https://www.ephotozine.com/article/n...r-review-25602
https://www.lenstip.com/426.4-Lens_r...esolution.html
https://www.photographyblog.com/revi...ew/sharpness_1
https://www.photographyblog.com/revi...ew/sharpness_2
https://www.photographyblog.com/revi...ew/sharpness_3
https://www.photographyblog.com/revi...ew/sharpness_4
https://www.photographyblog.com/revi...ew/sharpness_5
https://www.dxomark.com/Lenses/Nikon...kon-D500__1061

I would appreciate the comments, corrections and help (about English Language and technical aspects).
Cheers,
Antonio.

[Manfred M]

If you want "facts", I find that both DxO Mark and Roger Cicala at LensRentals (https://wordpress.lensrentals.com/blog/) are worth reading. Both these sites have technical people, rather than photography writers, doing tests and writing up reviews. Another site I like is Luminous Landscape who are essentially good photographers writing for photographers. The only issue with that site is that it is a "pay site" and one has to have a subscription (only $US 12 per year).

Don't pay too much attention to Ken Rockwell. He can be an amusing read as long as one does not take him too seriously. He is a prime example of a writer who seems to confuse opinion with fact. The one fact that I do agree with him on is that review sites get equipment loaned to them by the manufacturers or distributors of the gear that they are testing. If one stops writing "good reviews", the website's business model is broken because they won't get any equipment to test. Here DxOMark and Lens Rentals are immune as neither of these sites are primarily review sites.

DxO Labs rent (?) lenses and bodies to create profiles for DxO PhotoLab and LensRentals is primarily a rental company with some techs that write blogs. Neither site lives and dies based on the hits to their websites.

[Tronhard]

Further to this... Statistically, unless one is prepared to test a valid number of lenses, a test is only really a test of that one lens. I have seen numerous comments from users on both DPREVIEW and the Canon USA forum that their lens was or was not sharp. In some cases the advised that the lens was fixed by the manufacturer.

While manufacturers doubtless put great effort into quality and consistent assembly, there are bound to be minute variations in the elements and their assembly. Add to that the chances of a lens being damaged during handling (especially the non-pro ones) - recently a courier in NZ was filmed TOSSING packages marked Fragile between vans, if they hit the deck of the target van that was bad enough, but sometimes they missed or bounced off and hit the ground to boot. I once saw a bag with fragile items that I could not check in being tossed onto the loading belt - it got to the top and then fell about 6m to the ground. Needless to say I was onto the airline right away as soon as I landed and got compensation for the damage.

[MrB]

Lens tests

[Shadowman]

Less than five years ago whenever I wanted to evaluate a lens on paper; I would look to Popular Photography magazine (no longer published), they would use a series of evaluations included an SQF (subjective quality factor) which included a table which evaluated sharpness by aperture and print size, they would also include sample images. I haven't purchased a lens in the last five years so haven't really delved too deeply in new offerings.

https://www.popphoto.com/lens-test-t...f-28-di-vc-usd

[Clactonian]

Lens tests are a good guide or starting point, but given all the variables, manufacturing tolerances with both the lens and camera, differing sensor designs, the way you use the lens etc., the definitive test will always be the way it works for you on a specific camera. If your chosen lens doesn't perform as expected then return to sender.

[russellsnr]

Hi, The trouble I find is that you look at a review for a lens XXXX and whoever you look at the reviews vary so much, one says not a sharp a lens another great very sharp lens and so on. I was looking at buying a Canon 100-400mm version one lens as the version two is outside my pocket, some say the version two is a much better lens others say they love the version one and would not trade up. I don't have the option of going and taking a lens on hire but if I ever get back to the UK I would certainly do so. Russ.

[Tronhard]

I had the 100-400MkI and the images I got with mine were fine for my objectives. However I am not a fan of push-pull lenses for the type of photography I do, although I still have the Canon 28-300L, since there is no alternative.

The thing is almost any lens will have flaws in design, manufacture and handling. But for me the suitability of a lens (or camera for that matter) is dependent on what I am trying to produce. I am not a massive pixel peeper - I find that in the real world people look at images on line on on portable devices, or printed from a comfortable distance. If the lens and camera produce what I want for my purposes I am fine with that. Still, each to their own...

[xpatUSA]

Any test of a single lens is meaningless in terms of Quality Assurance.

Roger Cicala of LensRentals is the only tester I've read that publishes tests of multiple lenses (of the same model) in statistically significant quantities.

I suppose the only alternative is to find many separate reviews of the same lens model and to analyze like parameters, e.g. MTF50, rather than subjective opinions ... good luck with that, eh?

[Astro]

I'm afraid I'm a sceptic when it comes to this type of information. I am very much of the same opimion as Ted and Trev.

I tend to the view that my shooting technique, particularly as I get older is probably the most significant issue when it comes to the quality of my images.
When it comes to 'choosing' a new lens, I prefer to simply try it. The companies I generally buy from are quite willing to give me an hour or two to try, and I have on occasion hired a lens for the day for a trial.

And, truth be told, despite having a science/technical past (Chemist/Biochemist/IT), I'm not sure I really understand what the variances in the technical measurements mean in reality...... One photographer's blur is arguably another's good bokeh so ......

[Meandmypup403]

I personally like opticallimits.com for lens reviews - good combination of objective tests and hands-on experience.

[Panama Hat & Camera]

Lens tests

Thank you Philip, I corrected the title of my thread, but I could't correct the title of replies.
Regards,
Antonio.

[Panama Hat & Camera]

If you want "facts", I find that both DxO Mark and Roger Cicala at LensRentals (https://wordpress.lensrentals.com/blog/) are worth reading. Both these sites have technical people, rather than photography writers, doing tests and writing up reviews. Another site I like is Luminous Landscape who are essentially good photographers writing for photographers. The only issue with that site is that it is a "pay site" and one has to have a subscription (only $US 12 per year).

Don't pay too much attention to Ken Rockwell. He can be an amusing read as long as one does not take him too seriously. He is a prime example of a writer who seems to confuse opinion with fact. The one fact that I do agree with him on is that review sites get equipment loaned to them by the manufacturers or distributors of the gear that they are testing. If one stops writing "good reviews", the website's business model is broken because they won't get any equipment to test. Here DxOMark and Lens Rentals are immune as neither of these sites are primarily review sites.

DxO Labs rent (?) lenses and bodies to create profiles for DxO PhotoLab and LensRentals is primarily a rental company with some techs that write blogs. Neither site lives and dies based on the hits to their websites.

Manfred, thanks for your comments and the link "https://wordpress.lensrentals.com/blog/". I began to read the articles and now I have another point of view on the tests of lenses.
Cheers,
Antonio.

https://www.lensrentals.com/blog/201...h-translation/
https://wordpress.lensrentals.com/bl...s-test-either/

[MrB]

Thank you Philip, I corrected the title of my thread, but I could't correct the title of replies.
Regards,
Antonio.

Sorry Antonio, it is not meant as a correction of your title. I wonder if you and others realise that, in Post #4, it is a link to a relevant article on another web site?

Philip

[Panama Hat & Camera]

Further to this... Statistically, unless one is prepared to test a valid number of lenses, a test is only really a test of that one lens. I have seen numerous comments from users on both DPREVIEW and the Canon USA forum that their lens was or was not sharp. In some cases the advised that the lens was fixed by the manufacturer.

While manufacturers doubtless put great effort into quality and consistent assembly, there are bound to be minute variations in the elements and their assembly. Add to that the chances of a lens being damaged during handling (especially the non-pro ones) - recently a courier in NZ was filmed TOSSING packages marked Fragile between vans, if they hit the deck of the target van that was bad enough, but sometimes they missed or bounced off and hit the ground to boot. I once saw a bag with fragile items that I could not check in being tossed onto the loading belt - it got to the top and then fell about 6m to the ground. Needless to say I was onto the airline right away as soon as I landed and got compensation for the damage.

Trev, thanks for your comments.
You are right, a lens test is just a test of that one lens.
I wrote this thread because the recommended apertures varied greatly from one review to another. I will do my own tests using various apertures to find out what's best for me (I usually use the maximum aperture when I'm taking photos of birds), but don't worry: I won't post my tests here!
Cheers,
Antonio.

[Panama Hat & Camera]

Less than five years ago whenever I wanted to evaluate a lens on paper; I would look to Popular Photography magazine (no longer published), they would use a series of evaluations included an SQF (subjective quality factor) which included a table which evaluated sharpness by aperture and print size, they would also include sample images. I haven't purchased a lens in the last five years so haven't really delved too deeply in new offerings.

https://www.popphoto.com/lens-test-t...f-28-di-vc-usd

John, thanks for your feedback and the link "https://www.popphoto.com/lens-test-t...f-28-di-vc-usd".
I used to read Popular Photography magazine when I was young (and took pictures with the Olympus OM-2 camera).
Cheers,
Antonio.

[Panama Hat & Camera]

Lens tests are a good guide or starting point, but given all the variables, manufacturing tolerances with both the lens and camera, differing sensor designs, the way you use the lens etc., the definitive test will always be the way it works for you on a specific camera. If your chosen lens doesn't perform as expected then return to sender.

Mike, thanks for your comments.
I agree with you. I will do my own tests using various apertures to find out what is best for me.
Cheers,
Antonio.

[Shadowman]

John, thanks for your feedback and the link "https://www.popphoto.com/lens-test-t...f-28-di-vc-usd".
I used to read Popular Photography magazine when I was young (and took pictures with the Olympus OM-2 camera).
Cheers,
Antonio.

I had a subscription until it went out of business, I still had issued owed me and they started sending me Popular Science. Luckily I paid my subscription by check or else they would've continued billing my card until I put a stop on it.

[Panama Hat & Camera]

Sorry Antonio, it is not meant as a correction of your title. I wonder if you and others realise that, in Post #4, it is a link to a relevant article on another web site?

Philip

Yes Philip!
I saw the link "https://theonlinephotographer.typepad.com/the_online_photographer/all-lens-tests-are-wrong.html" and read the article (very good).
The only thing I do not realize is how the link is hidden in your post.
Thank you very much.
Antonio.

[Panama Hat & Camera]

Hi, The trouble I find is that you look at a review for a lens XXXX and whoever you look at the reviews vary so much, one says not a sharp a lens another great very sharp lens and so on. I was looking at buying a Canon 100-400mm version one lens as the version two is outside my pocket, some say the version two is a much better lens others say they love the version one and would not trade up. I don't have the option of going and taking a lens on hire but if I ever get back to the UK I would certainly do so. Russ.

Russel,
After reading the review of PCMagazine, I started to look for other tests and, to my surprise, I realized that there were different recommendations for the best aperture. So I decided to study the subject and post the thread. I will do my own tests using various apertures to find out what is best for me, but do not worry: I will not post my tests here!
Cheers,
Antonio.