Comparing low light performance of different cameras

[facts_please]

The ability of a camera to take low noise images under low light conditions is extremely valuable. Given that almost all digital camera these days have more than enough megapixels, low light performance is arguably one of the most important factors to consider when purchasing a new camera.

The purpose of this thread is to discuss methods for testing and comparing the low light performance of different cameras. What testing method(s) would enable a prospective buyer to judge which camera performs better under low light conditions, and how much better? What exactly should be compared, and how would you go about doing it?

This is not as simple as it may appear. Many respected camera review sites test and compare low light / high ISO performance of cameras, but it is not at all clear (at least to me) what their tests are trying to achieve, or even if their comparisons have any real meaning at all.

I will start the discussion by giving some of my own ideas on this topic. On 15/2/09 I started a thread entitled ‘Large Sensors and Image Noise’, specifically to provide some background information for this topic, and much of what I write here is explained in greater detail in my first posting to that thread.

Consider the common situation where a prospective buyer would like to know how camera A compares with camera B, in terms of low light performance. An obvious example would be choosing between a top end point and shoot (say a Canon G10), and a basic SLR, for example a Canon EOS1000D. To my mind, two key questions that should be included in such a comparison are :-

(a)
How much intrinsic low-noise advantage (if any) is provided by the larger SLR sensor? The comparison should show two images, one from each camera, which directly illustrate the lower image noise provided by the larger sensor. The key concept here is that both sensors must receive the same number of photons, and a method of doing this is given in the thread ‘Large sensors and Image Noise’. This is an important measurement because it cannot be predicted from the specifications.

(b)
How much better will the SLR be when the conditions are really tough, and both cameras are flat out capturing all the light they can get? In other words, what advantage is gained by the better camera as a whole, including the lens system? Express this advantage in terms of stops, and show two images, one from each camera, that illustrate this advantage. Again, see the previous thread for details of taking this measurement.

I contend that these are key questions that the prospective buyer would like answered, and the (admittedly few) online tests and comparisons that I have seen do not seem to answer either question!

Comments?

Any agreement or disagreement that these are important key questions?

Any suggested alternative testing/comparison methods?

Any comments on what the low light (high ISO) tests in dpreview.com, for example, actually mean?

[Colin Southern]

Any comments on what the low light (high ISO) tests in dpreview.com, for example, actually mean?

Hi Colin,

I think that while the "lab rats" were busy doing their testing, the rest of the photography fraternity were out shooting somewhere in the real world

Given that almost all digital camera these days have more than enough megapixels, low light performance is arguably one of the most important factors to consider when purchasing a new camera.

For some perhaps - I couldn't care less about it to be honest (and I do a LOT of low-light shooting).

For me, the important factors are robustness & construction quality - weather sealing - AF performance - metering options and firmware options.

In my opinion, noise characteristics of all the latest SLR cameras qualify for the "more than satisfactory" category; sure, the differences can be seen in photoshop in "100% crop tests", but for most "real-world" situations they're just not an issue. If manufacturers continue to improve low-light performance and improve noise characteristics then that's fine - but it's not anything that's holding me (or any other photographers that I know) back in the meantime.

If anything, LOWER ISO options would be far more use to me.

[facts_please]

In my original posting on this thread, I indicated that some common tests used to compare low light performance of different cameras appeared to be of dubious value.

For example, one well respected website uses the following test. Basically, photographs are taken of a standard object, or studio scene, over a wide range of ISO sensitivities, for example from ISO80 to ISO3200, and a crop is shown from a small part of the scene. Not surprisingly, the image appears increasingly noisy at higher ISO values. Frequently this is done for several cameras, so that the viewer can judge which camera produces the ‘cleanest’ most noise free pictures. Generally speaking, the more expensive cameras, and SLR style cameras in particular, produce cleaner images at higher ISO numbers. As the ability to produce noise free images under low light conditions is important in the real world, much significance is attached to these tests, with the reviewers not at all shy in pronouncing which camera and image is best, and invariably give credit to the larger sensor for the superior result.

HOWEVER, JUST WHAT, EXACTLY, DO THESE TESTS MEAN????

To answer that question I need to fully describe the test that is commonly used. When comparing several cameras, all cameras are set at the same indicated ISO. All cameras view the same scene, and I presume (and certainly hope) that all cameras are placed at the same distance from the scene. The field of view is the same for all cameras, meaning that the same sized scene exactly fills the sensors of all cameras, and presumably this is done by adjusting the lens focal length (zoom) on each camera. For each camera, the correct exposure is then obtained by adjustment of shutter speed or aperture.

Frequently a prime lens, not normally supplied with the camera, is used and in addition the test makes no attempt to exploit the full lens aperture, so I must presume that the test is intended to tell us something about the noise performance of the sensor, rather than the camera and lens system as a whole. Certainly the reviewers invariably attempt to interpret the result in terms of the sensor, rather than the lens.

So far, so good, except that if the sensor is being tested, then it is necessary that the same amount of light is delivered to each sensor under test. After all, it is totally obvious that if substantially more light is given to one sensor, then that camera will produce a cleaner image, and the test would be meaningless ….

Unfortunately, when the test is carried out as described, the sensors in the different cameras being compared most definitely DO NOT receive the same amount of light, and in some cases by a factor of 3 or more. Believe me that this is so – it can be predicted theoretically, and it can be confirmed by examining the quoted f-number and shutter speed which were in fact used to obtain the correct exposure. If anyone is interested or doubtful I can talk more about the details, but for the sake of brevity just believe me for now. I hope it is thus obvious that these test tells us nothing about the low noise performance of the sensors, and as discussed the test also tells us nothing about the lenses, so just what do these tests tell us?

As always, I seek comments from others re my reasoning here.

[Dave Humphries]

Colin (facts_please),

NB This was written before reading your post #3 in this thread. (crossed in the post)

You wrote; "Any suggested alternative testing/comparison methods?"

One method I find quite illustrative in magazine reviews is a 100% crop of a shot of a subject taken at the low (100) and high (3200) ISO settings. One would hope this is achieved in windless conditions at a constant aperture and varied shutter speed to offset the ISO. When done for brand A vs brand B, one appears to get something that one can compare.

One thing though; the engineer in me does recognise that the high ISO performance in good light demonstrated by the comparison above is not the same as low light performance, when other factors could come into play, like fixed pattern noise, etc. There were also candle light shots in the review, but these were probably not 100% crops and the differences between them are difficult to judge in print.

However, I don't have the desire you have to get to the technical bottom line on things like this, I view the magazine (or internet) results with a fair degree of scepticism and weight them in my mind accordingly. Although I followed your sensor size thread, I'll admit to having only a tenuous grasp of the detail - it appeared to make sense at the time, but I couldn't answer questions on it afterwards!

My interest is as follows:
I am currently weighing the pros and cons of going 4/3 against APS-C for a DSLR and the above comparison shows the 4/3s camera (E-30) to be significantly worse than the APS-C ones (50D and D300). However, I also factor into this that I currently have a very tiny sensor, (but slightly larger lens) 'bridge' camera and have to noise reduce all my images anyway, either in ACR or Neat Image, so while I am hoping for an improvement I could live with it being similar. The other feature advantages on the newer cameras, in particular the E-30, outweigh the noise performance for me - something I know I can fix for a small drop in IQ in PP. If I couldn't frame the shot properly because I couldn't see the viewfinder, the noise figures are irrelevant if I have chopped of the subject's head.

To paraphrase what Colin Southern has said (in other threads); too much pixel peeping and attention to minute detail will turn the average photographer to a whimpering blob unable to decide what day of the week it is, let alone what camera to buy. Clearly you have the ability to cope with this level of detail, but many of us do not.

It will be interesting to see whether your master-plan to educate us all achieves its goal - if it does, in a way mere mortals can apply and gain benefit from in day-to-day decision making, then I'll be the first to thank you - unless someone beats me to it

You might like to consider introducing yourself in the Introduce Yourself & Welcome Other Members thread so we know a bit more about you. As you may have noticed, C-in-C threads run more like conversations than two sides of a point of view and this is easier when we know who you are. I hope you don't mind me raising that and that the above comparison method adds to the thread content.

Regards,

[Dave Humphries]

Hi again Colin,

Unfortunately, when the test is carried out as described, the sensors in the different cameras being compared most definitely DO NOT receive the same amount of light, and in some cases by a factor of 3 or more. Believe me that this is so – it can be predicted theoretically, and it can be confirmed by examining the quoted f-number and shutter speed which were in fact used to obtain the correct exposure. If anyone is interested or doubtful I can talk more about the details, but for the sake of brevity just believe me for now. I hope it is thus obvious that these test tells us nothing about the low noise performance of the sensors, and as discussed the test also tells us nothing about the lenses, so just what do these tests tell us?

Ah see, I fell into the trap, this is interesting to know, but in terms of application in the real world, doesn't it just support Colin's idea of such figures being viewed as "more than satisfactory" and, due to the unreliability of comparison methods, something to be taken as of dubious value?

Is there an alternative?

Regards, Dave

[Davey]

You might as well rule out true objective testing since scientific method wont work because you'd need to use the same lens on each camera really for it to be a dead on accurate test. The next best thing is of course using similar lenses and minimising difference because lens factor is an unavoidable variable but as you say it still plays some part in the light gathering ability.

I think most tests don't stand up in the real world, not just for cameras but for many things. Most are misleading and poor representation of real world applications due to abstract nature of the tests. My prefered method would be quite simple, try it yourself and if it suits your needs and meets/exceeds expectations then great, if not choose another potentially promising one. Sounds too simple but thinking about it what better test of low light level performance in the conditions you shoot than field testing it, that way results are exactly what you are after. So real world benchmarks (especially from people testing in similar manner to what I'd use and without major brand bias etc) for anything usually interest me more than abstract reviews, however "reliable" such artificial reviews are they often have little bearing on actual performance I observe.

Even if you built a camera body to house the test sensor that had the same lens, same scene, rigged to so same amount of photons hit the sensor etc etc. Basically the only variable being the sensor. What good would it do? Unless you are looking at making a camera line and are interested in the sensor for that performance spec it's irrelevant since variables like lens and other factors will always come into play, might even be the best noise free at low light sensor is the worst because it's in a body that only takes lenses which don't produce a final result in low light compared to another sensor (albeit one with poorer snr) that is in a more fitting body that takes lens better suited to low light photography. Coupled with other needs it gets more complex.

I'd say I agree with all your points, except that sensor performance in low light being the most important feature. I think the most important feature even for night photography is highly subjective. Everyone has different needs and no one thing is more important. Essentially it's a blend of budjet, ergonomics, current/future accessory compatability, familiarity, size & weight, robustness (including operating environ related stuff like weather sealed lenses, from stuff for diving/marine photography to sandy conditions or extreme temperatures), image quality and specs (from sensor size/cropping issues to noise), software/hardware considerations. I'm sure there are many more I've missed too. I also agree an accurate test is needed but all things considered it doesn't really have much bearing on the current situation of there are no 100% perfect tests and unlikely to be any due to it being more complicated than just sensor performance. Until body and lens choice (and other specs) becomes standardised for all cameras (which clearly wont happen) field testing and considering all factors relevant to you is the way to go in my opinion.

[facts_please]

Firstly, thanks to Dave, Davey and ColinS for your comments.

One common comment has been that 'low light performance' is maybe not that important for many users anyway.

That depends very much on what camera you own. As this is an enthusiast website, a high proportion of members own SLRs, and agreed that these are pretty good. However, in general many people also own compact point and shoot camera, and I can tell you from experience, you find yourself so, so often wishing for more light with one of these. For a significant group, the size and weight of an SLR really is a drawback, such as when your camera is typically carried all day in a backpack on outdoor expeditions, and these people (I am one of them) really want to know how much advantage an SLR will give before buying one.

More importantly, I would like to generalize the concept of good 'low light performance'. The broader, and more important concept, is that it is really, really useful to at all times have the option of getting more light through the lens and into the camera. The advantage that this gives you can be 'used' in four ways, probably more.

(a) With light to play with, you have the option of higher shutter speed.
(b) With light to play with, you have the option of high depth of field.
(c) With light to play with, you have the option of using lower ISO to obtain lower image noise
(d) With light to play with, you have the option of taking quality low-light shots without flash

You can (nearly) always get rid of the light by stopping down the aperture, using higher shutter speed, or lower ISO, but if you camera can't gather enough light to do what you want on any given shot, then you are stuffed, or you will need to compromise.

With all that in mind, from here on I will speak not of low light performance per se, but of the camera's potential ability to gather as much light as possible. That IS important, really important, and my discussions are ultimately leading to methods of testing, predicting and comparing cameras on that basis.

Assuming, of course, that no one shoots me down in flames in the meantime.

To DaveH, I am hopeful that my final conclusions and summary will be simple, useful, and very much related to the real world. As you say, useful information is that which we 'can apply and gain benefit from in day-to-day decision making'. For nutters like me though, the underlying technology and theory is also interesting in it's own right.

You might like to consider introducing yourself in the Introduce Yourself & Welcome Other Members thread so we know a bit more about you. As you may have noticed, C-in-C threads run more like conversations than two sides of a point of view and this is easier when we know who you are. I hope you don't mind me raising that and that the above comparison method adds to the thread content.

I'll think about that - I doubt I am typical. And of course, the more people contribute to the thread, the better. Thanks. Um, what's a 'C-in-C' thread?


PS to ColinS
The superb digital cameras that we enjoy today came about because of the efforts of engineers and 'lab rats'.

[Dave Humphries]

Hi Colin,

Don't worry that you are not typical, we have all sorts here, from vastly differing backgrounds and cultures, I'm sure we can cope with a scientist too. What you are doesn't really matter here, just a love (if that's not putting it too strongly) of photography.

"C-in-C" is shorthand for Cambridge in Colour, where we are now, and a thread is what we are in now (but I suspect you knew that last bit). The conversational reference is just that we like to get a bit chatty now and then and we don't take ourselves too seriously. Seems you don't either, so welcome to the forum.

I look forward to the next instalment.

Regards,

[Colin Southern]

More importantly, I would like to generalize the concept of good 'low light performance'. The broader, and more important concept, is that it is really, really useful to at all times have the option of getting more light through the lens and into the camera. The advantage that this gives you can be 'used' in four ways, probably more.

(a) With light to play with, you have the option of higher shutter speed.
(b) With light to play with, you have the option of high depth of field.
(c) With light to play with, you have the option of using lower ISO to obtain lower image noise
(d) With light to play with, you have the option of taking quality low-light shots without flash

You can (nearly) always get rid of the light by stopping down the aperture, using higher shutter speed, or lower ISO, but if you camera can't gather enough light to do what you want on any given shot, then you are stuffed, or you will need to compromise.

Doesn't work that way for world-class landscape shooting - especially where water is involved.

(a) Higher shutter speeds aren't an option because they don't smooth out water or allow transient objects (birds / planes etc) to "disappear".

(b) We need a high depth of field - so we're shooting at F11 to F22 anyway.

(c) ISO is usually at 100 (50 is available but because it's done in firmware you lose a stop of DR which isn't usually acceptable).

(d) Flash isn't needed for landscape anyway. Low light performance can be a blessing at weddings and the like, but generally levels of light that are that low aren't usually that flattering anyway - and it's difficult to use fill flash with that amount of sensitivity because the flash has to output a minimum level of brightness, which nukes the exposures at extreme ISOs.

[facts_please]

However, in general many people also own compact point and shoot camera, and I can tell you from experience, you find yourself so, so often wishing for more light with one of these.

More importantly, I would like to generalize the concept of good 'low light performance'. The broader, and more important concept, is that it is really, really useful to at all times have the option of getting more light through the lens and into the camera. The advantage that this gives you can be 'used' in four ways, probably more ....

Doesn't work that way for world-class landscape shooting - especially where water is involved.

(a) Higher shutter speeds aren't an option because they don't smooth out water or allow transient objects (birds / planes etc) to "disappear".

(b) We need a high depth of field - so we're shooting at F11 to F22 anyway.

(c) ISO is usually at 100 (50 is available but because it's done in firmware you lose a stop of DR which isn't usually acceptable).

(d) Flash isn't needed for landscape anyway ...

Yes, sometimes you have enough light, or even too much. I stand 100% behind what I wrote though. An expensive camera+lens gives you the OPTION of getting more light into the camera, and for most photographers that option is very useful indeed. If it were not so, there would be no market for fast, expensive lenses ....

On a totally different subject, I personally don't like the blurred, 'smoothed out' look of moving water, it just looks so unnatural to me, but that is personal opinion only - there is no right or wrong.

[royzone48]

try Bjorn Rorslet

[Davey]

Hehehe you got the point and shoot bit right for my model at least. I use a finepix s5600 and I can tell you it's low light performance isn't very "performing". I'm happy with it though since it's my first digial camera and I'm very new and it's just a hobby it makes a great learning camera. I'm not scared of using it in the snow or the like and it's light enough to carry everywhere (much lighter considering it's an upgrade to my old pentax 35mm film cam, considering I always carried 3 lenses when out with it (usually wide angle, standard portrait and telephoto or macro lens) and 1 flash unit I travel MUCH lighter).

Even though for me real life over all / multi factor is what I'd personally base a purchase on I'd still be interested in whatever results you come up with even if it's more interest than purchase driven consideration (I've already narrowed my future upgrade down to a nikon d80 as that seems to fit what I want next, be a while yet though and who knows your result may sway me). I'm sure many people here would also be interested too, some like me just for the fun of the theory of it. After all everyone (well maybe not everyone) loves knowing how stuff works even if it has no direct relevance to them other than enjoyment (my favourite present as a kid was a meccano set because it had a screw driver in, I took EVERYTHING it fitted to pieces to work out how they worked. I did, but they tended not to go back together so well ).

As for an intro I think the community here is spread across such a wide range that there is no typical member and your contributions will add yet more to the ever increasing diversity and appeal here.

[Colin Southern]

Yes, sometimes you have enough light, or even too much. I stand 100% behind what I wrote though. An expensive camera+lens gives you the OPTION of getting more light into the camera, and for most photographers that option is very useful indeed. If it were not so, there would be no market for fast, expensive lenses ....

Again, doesn't quite work that way in the field - I have and use lots of expensive and fast glass - but even for non-landscape shooting, the speed doesn't usually enter into it a heck of a lot. For Canon, "Fast and Expensive" means L-Series (which is all I shoot with) - the extra speed is almost a by product; what makes the lenses a "no-brainer" for me is build quality - ruggedness - weather sealing - superior optics. In reality once you start getting into the faster speeds (<F2.8) (a) your DoF can become so narrow (often just a few mm at F1.2) that you just can't use the lens for "conventional" photography (ie the desire to have all parts of a face in focus), and (b) much faster than F2 outside on a sunny day you run out of shutter speed, requiring something above 1/8000th @ ISO 100.

On a totally different subject, I personally don't like the blurred, 'smoothed out' look of moving water, it just looks so unnatural to me, but that is personal opinion only - there is no right or wrong.

Luckily for me my customers love it

At the end of the day the camera doesn't work like a human eye, and doesn't capture things the way we see them - moreover humans don't REMEMBER scenes as they saw them either. It's just another art form.

[Bill44]

When I got my D90 I did a very scientific low light performance test. Walked into my fairly dark loungeroom, opened the lens right up and wound the iso up to 3200, checked that I wasn't up into the multi-second exposure time and let 'er rip. Had a look at the shot on the 'puter and said "She'll do".
As I said, very scientific.

[Colin Southern]

When I got my D90 I did a very scientific low light performance test. Walked into my fairly dark loungeroom, opened the lens right up and wound the iso up to 3200, checked that I wasn't up into the multi-second exposure time and let 'er rip. Had a look at the shot on the 'puter and said "She'll do".
As I said, very scientific.

I reckon if you were a farmer Bill, you'd be the sort to tie it up with a piece of bailing twine and get the job done!

[Dave Humphries]

Colin (f_p),

I might be oversimplifying here, so correct me if I am wrong.

If the important point you are trying to get across is that it is the amount of light hitting the sensor that matters most, then surely the likes of my Fuji S6500 and Davey's Fuji S5800 should perform better in low light than most point and shoots because although our sensor is the same size as most of those, our lens' are much bigger physically. Judging by the front element and barrel size, they are probably similar to say 4/3 DSLR lenses.

Now I haven't compared mine for low light/noise/high ISO against one of the smaller compacts, but as Davey says, what is noticeable is that we do get noisy pictures.

What am I missing?

Thanks,

[facts_please]

DaveH: If the important point you are trying to get across is that it is the amount of light hitting the sensor that matters most ...

That is EXACTLY the point I trying to get across, yes.

, then surely the likes of my Fuji S6500 and Davey's Fuji S5800 should perform better in low light than most point and shoots because although our sensor is the same size as most of those, our lens' are much bigger physically. Judging by the front element and barrel size, they are probably similar to say 4/3 DSLR lenses.
Now I haven't compared mine for low light/noise/high ISO against one of the smaller compacts, but as Davey says, what is noticeable is that we do get noisy pictures.

All else equal, a point and shoot camera with a physically larger lens will do better than one with a smaller lens. Everyone is so hung up on sensor size that the basics have been forgotten - you cannot get around the fact that to get low noise images in low light needs a decent piece of glass out the front to gather enough light for the sensor, and this fact is true quite independently of sensor size. However, lenses with a large zoom range as per the S5600 throw away a lot of light, so you really just need to run the numbers. I presume you only find the noise objectionable at higher ISO, is that correct?

I am in the process of writing up an example comparing the light-gathering ability of the S5600, D80 SLR and Olympus E30 SLR, which I think you will find interesting. Even better, the calcs are real simple so you will easily be able to compare any other camera you want.

[Dave Humphries]

Can't wait!

[Davey]

true it's only high iso that is a real problem. An unprocessed (or straight developed raw with minimal process I mean, ie raf to tiff with no real improvements made with fuji software) image even at lower isos (but typically long exposures for me hence the particular noise problems) I can see noise at 100% on screen BEFORE processing. It's no more noticable than it was on 35mm especially since not visible on print sizes I do even before processed. In fact noise wise it's equal at my given view/print size to a friend I lived with images from a good cam (he had a D3).

After processing (even lightly) and scaling to view size for monitor or printing (no larger than a4) the noise is far from problematic really. Just the initial OMG factor when I tried a few low light iso 800 and 1600 shots to see how well I suspected my cam wouldn't fare hehehehe. It's rare for me to need fast action night shots so I tend to stick with 15sec exposures on iso 64 or iso 100 and noise isn't a problem in final result despite that if I look at it close up on screen it's noticable. The key to my issue with high noise is several factors including

1. I look too closely compared to others (can be a little too perfectionist at times verging on ocd hehehe). Scale of viewed /printed image neutralises

2. I tend to shoot of a night, [low light] + [picky mofo] + [sensor x lens factors] =

3. Rating noise at 100% based on the preprocessing preview. The mid stage actual image is cleaner than initial preview never mind finished article. I like omelettes as much as the next guy but raw eggs just ain't my thing if you get what i mean, the stage at which the review is made is critical. Anyone who says they hate bread although they've never eaten it cooked, their experience is based on eating dough is a little inaccurate for rating if they like bread.