The Swan and the Histogram

[Nicks Pics]

Hi folks

Although you have seen better Mute Swan photos recently I got some great photo ops when the lake partially melted and waterfowl showed up again for the open water. Anyway, all comments/critique on this welcome, though IQ is not up to the standards of some, partially because it was shot on a bridge camera, but I am particularly interested in thoughts about the histogram I have posted.

I thought there needed to be more contrast between the swan and the ice


So I applied a curve something like this


Then it looked like this


But the histogram got gaps in it, though I'm not sure why


It looked something like this before


I processed this in Lightroom from RAW first, saved as TIFF, for importing into Gimp, then when I was done I exported to Jpeg.

So I am wondering if you think adding the contrast is a good idea even if it makes that happen to the histogram, and what I can do to prevent that from happening to the histogram. Gimp can't accept my RAW files, so I save them to TIFF, would there be a better format? It doesn't do more than 8 bit. Is that the problem?

Just a note: There was more visual difference between the before and after swan in the editor than when I exported to Jpeg, not sure why.

Thanks all for viewing!

[Downrigger]

Thanks for posting a great problem! I haven't a clue unless it's that the original (camera) file is limited in size, but I'm looking forward to an informed answer.

[Shadowman]

Not sure if it's related to the gaps in the histogram, but you've got a white subject on a reflective surface, lots of bounce creating a series of grays. The rich colors of the swan's head keeps the image looking fresh. Nice effort.

[davidedric]

Don't know, I'm afraid. The only times I have seen gaps appearing in a histogram is when processing a fairly low res, I think 8bit, jpeg.

However it seems that the gaps mostly do not appear in the tones that your curve affects. Strange.

[Geoff F]

That often occurs when there is a bit of a shortage in good detail but usually doesn't cause any problems. Not sure I can explain the science though.

I might be tempted to boost the highlights slightly more. Move the top right corner of the Curves line fractionally to the left.

[Brownbear]

Hi Nick,

Nice image!

I can't answer your question about the spiking in the histogram but I can advise that I tried a curves adjustment just like yours on a few of my swans photographed under similar conditions, and no spiking was seen, even when I exported the image as a TIFF from LR to apply the curves adjustment in Photoshop CC... (Nikon D80 and Nikon D7100 images)

So I've sent you a raw file (private message) with a pic of a swan photographed under similar conditions for you to play with so you can figure out if it is your camera or post processing.

[ajohnw]

It's just the way the GIMP shows the histogram and nothing to worry about. What's happened is that you have changed the contrast = the rate tones change at so some have moved out of the display bit space. Some packages - might I say all when working 3x8bit don't show this but it still happens.

Unless the GIMP has been upgraded recently it will have mentioned that it is down converting any 16bit TIFF that is loaded so you 16bit info has gone. My version no longer mentions this when I load 16bit png so I assume all of that data is retained in it's internal 32bit floating point format. I have read that the developers are having problems with the TIFF library. As things change over time these sort of things sometimes have to be rewritten at some point. The TiFF library is ancient. PNG is an open source format - they may favour everyone using that. Hard to know.

Actually I am not sure that all of the use of a 32bit floating point workspace have been cleaned up. I suspect some 2.8 versions are part done, 2.9 will be more done and things should be sorted on 2.10. Until then as always when using the GIMP general tone behaviour has to be sorted out from raw and GIMP just used to polish things up in this area. It's surprising just how many gaps can be in a histogram before anything becomes visible in the image

If you want to try the next release it can be found here for windows but beware of the warnings.

http://nightly.darkrefraction.com/gimp/

To be honest you would be better off playing with Rawtherapee to avoid these effects. It's perfectly happy to accept all that is thrown at it in it's entirety and has curves - 2 in fact.

On colour changes it sound like you colour management isn't functioning as it should. There is a section in GIMP's preferences for setting that up. Some strange things can happen in this area - not just with the GIMP either. I have try to use system colour profile ticked, it's actually installed local user but still works. It's also possible to locate the correct one off the disk from the same preferences panel.

John
-

[Loose Canon]

Hi Nick!

What these gaps represent is the data loss of smooth transitions between tonal blending. Meaning that with the curves adjustment you have re-mapped the tonal values (spread them apart) and this causes the gaps.

There are ways you can avoid this. Correct setting of the black and white points when you convert your RAW is a biggie and checking this should be one of the first things you do for any shot. That way you won’t have so big an adjustment in a pixel editor and your contrast concerns may be addressed before you even get to the editor. You could also address any contrast enhancement in the RAW converter. Your histogram will fill the range from 0-255 before you bring it into the pixel editor. With this shot, correct B&W point settings in RAW conversion gave a big, fat histogram in the editor. The same curves adjustment only caused the tonal gaps in a relatively small part of the histogram. Right there is a big improvement.

The image appears underexposed to begin with and when corrected noise is starting to rear its ugly head. I didn’t check your ISO setting here. And won’t. I can already tell. Proper exposure is another biggie. Minimum editing (yeah right!) is of course the main way, which means getting it right at the time you pull the shutter trigger! I don’t know anyone that gets it right every time so we just aspire! But the issues you are looking at Nick started long before you got the image to the point where you posted and are cumulative.

If this phenomena is exaggerated by processing, banding/posterization will occur.

With a good work-flow starting in the RAW converter what it means in the Real World is usually nothing! Or for sure minimally enough to be “nothing” to sweat.

Add your adjustments on a separate layer in the pixel editor. That way during the editing process you can go back and re-adjust if necessary. The “gaps” will (may) appear in the histogram during editing. If everything looks good with the edit visually, it is fine. The gaps will not show up when you flatten the image and convert to an output format.

Its good to know cause and effect, but the proof is always in the pudding so to speak!

[Nicks Pics]

Thanks every one for your comments! I would like to respond more when biscuits are not calling me.

[Dave Humphries]

Hi,

I can't better the advice Terry has given, so I won't try.

A bit of background may help understanding ...

A histogram is merely a graphical representation of the number of pixels at each and every luminance level.

So just think of it as a bar graph - the sort with vertical columns - unfortunately, it doesn't look much like a bar graph because all the individual columns (one for each luminance level) are pressed up against each other and they're all the same colour, but that's all it is.

The height of each column scaled in proportion to the maximum number of pixels (at the most prevalent level), the absolute numbers don't matter, so the vertical axis is not scaled numerically.
The horizontal axis is scaled; 0 on left (darkest digital value possible) and 255 on the right (brightest digital value possible).



I appreciate this may be a difficult concept to grasp mentally if you're not used to it - the tutorials here may help.
https://www.cambridgeincolour.com/tu...istograms1.htm
https://www.cambridgeincolour.com/tu...istograms2.htm

Or you could abstractedly think of it like this;

Imagine a black and white image made up from wooden blocks
Each pixel is a wooden block
Each different level of luminance is a subtly different shaded block, with white ones, black ones and 254 other shades of grey in between.
When you view the image, they are stacked as the camera took the image, in a rectangle, with the top left image element at the top left corner and so on.
Now knock 'em all down
Sort them into piles of the different shades (good luck with that!)
Now carefully stack all the black ones on the left (in one tall column)
Then add the next brighter shade of grey beside them (to right); in a new column just touching
Repeat 254 times - until you stack all the pure white blocks on the extreme right
Some columns will be tall, some less so
Now stand back - turn off the room lights - and switch on a light behind the stacks
There is your histogram - a real physical one in silhouette

Anyway ...

When you capture an image of a typical scene, there usually will be some pixels at all levels between the brightest and darkest pixels in the image - unless shooting say; a very evenly lit chessboard

Under exposure will cause the brightest pixels to not reach level 255, this is shown by the flat area on the right hand side of the histogram.
Flare will cause the darkest pixel level to be somewhat above 0, as shown by the flat area on the left.
In other words; it is a low contrast image.
In my analogy above, this doesn't mean less blocks, just more of the same shades, with no white or black ones at all.

As Terry says, if this is not addressed while the image is being edited in 16 bit mode (e.g. RAW), when you stretch the contrast; to put the darkest pixels nearer (or on) 0 and the brightest nearer (or on) 255, it will simply drag the levels apart, leaving gaps. Meaning there are no pixels at certain luminance levels.
In my analogy above (but with a low contrast image), think of spreading the columns apart so the ones that should have been white are in the place they should be (extreme right) and the ones that should have been black are fully over to the left. Now spread the others out, but remember you cannot have them at half levels, they must be in one of the 256 possible places, which leaves gaps

So imagine the original image, which has some pixels at level 127 and some at 128, since the two columns nestle against each other there is no gap.
If the contrast stretch is completed and/or displayed on an 8 bit histogram, it is possible that the pixels that were 127 are shifted to 126, while those at 128 stay put, the revised histogram now shows a gap one pixel wide at level 127. Sometimes the gap can be wider, you'll see there are some of these in the example provided, over to the right.

You will also note that some subsequent image editing operations will remove the gaps - anything that interpolates pixel values to achieve an effect; e.g. sharpening, image rotation, etc., will make the histogram gaps reduce or disappear. Please note however that it may not remove/reduce any visible problematic banded/posterized area in an image, because the interpolated pixel levels can be anywhere in the image and hardly be noticeable.

HTH, Dave

[Nicks Pics]

Hello folks! You all deserve more of a response than a comment about biscuits

Thanks every one for your comments! I would like to respond more when buscuits are not calling me.

I received some very helpful responses and learned a lot about what is happening here, why, and about how the histogram works.

Dave,
That was an excellent explanation of the histogram and how it relates to the image I posted. I learned a lot from that, and was able to understand it I think, so very much appreciated.

Hi Terry!
Thanks for your great reply. I learned a lot from that! I gather from your suggestions that I should try to do all/most of my adjustments to contrast/exposure in the RAW converter before exporting to the pixel editor. I didn't notice that that image was under-exposed (much, if at all) but now that you mention it I notice that most of my images from that day did not fill the right side of the histogram. Should I try to expose that all my images to fill it from end to end would you say? What ISO do you think I used? High or low? Generally I hear that using high is recommended if necessary to get a proper exposure. Anyway, thanks again for all your helpful advice!

John,

Thanks for your helpful reply, I appreciate that you seem to be familiar with Gimp, There were a few things you mentioned about it I would like to follow up on.

.Thanks for posting a great problem! I haven't a clue unless it's that the original (camera) file is limited in size, but I'm looking forward to an informed answer.

Thanks for stopping in Mark
Hope you've found some good reading in this thread Glad information can be shared here.

Thanks again to all for your comments and helpful insight!

[ajohnw]

GIMP's TIFF library. All open source software uses it to deal with TIFF's if it needs to. There is nothing wrong with it. The problem is down to interfacing it with GIMP's new greater bit depth image processing engine. There is nothing wrong with it's functionality.

Histogram gaps when curves are used / contrast is being increased. When they matter the effects will be apparent on the image - posterisation. This can happen when any bit depth is being manipulated. There isn't enough information in the basic data to allow the tones to be increase smoothly so bands appear. Also colour errors are likely to crop up especially when recovering dark areas. Lets say some grey sits at 10,10,12 rgb. The extra 2 in b is unlikely to be noticeable. If these values are brightened up to say 20,20,24 the blue will be far more noticeable. The 2 bits extra in the initial value can just be an error generated by the camera. I'd hope not at these number but this does happen when tones have low bit counts.

The only answer really to colour variations in different pieces of software is colour management. This needs a colorimeter to calibrate the display. If this isn't done colour managed software will just use some default. Some software may not be colour managed at all. The only answer to that is to colour manage the whole system by default. An alternative to a colorimeter might be found on tftcentral's monitor profile data base - if the monitor is listed and the profile comes with brightness, contrast and r g b settings. They are useless without the settings. Some other site posts some on it without the settings. Look on the reviews on their site and settings are usually missing.

John
-

[jprzybyla]

Each of the histograms is showing and under exposed image. The data in the white part of the histogram is missing.

[Loose Canon]

Hi Nick,

I gather from your suggestions that I should try to do all/most of my adjustments to contrast/exposure in the RAW converter before exporting to the pixel editor.

Yes.

I didn't notice that that image was under-exposed (much, if at all) but now that you mention it I notice that most of my images from that day did not fill the right side of the histogram. Should I try to expose that all my images to fill it from end to end would you say?

You won’t be able to do that necessarily. In Dave’s excellent explanation he noted that how the histogram “fills” is a function of the tonal values in the scene. Unless you add lighting or somehow change the scene, it is what it is. Some scenes will fill a histogram, some won’t.

What I would say is, generally speaking, up the exposure in camera at the time you shoot such that your histogram (on the camera image review screen) is moved more toward the right side. Just below clipping. This is necessary to give the widest leeway to reduce the appearance of noise at higher ISO values, or if your camera has a smaller sensor, when you set your white/black points in post. Especially in a low light, low contrast situation.

Just because your camera meter zeros out doesn't mean that is necessarily the best exposure. There are a lot of other factors that can enter the mix.

[jprzybyla]

Hi Nick, to follow up on my comments about the image being underexposed... the exposure meter in most (maybe all) cameras only is smart in bright light with a mid-tone background and subject using Matrix or Evaluative metering. In your image of the swan the histogram shows no data in the area where the whites are found. In this situation I would capture one image, take a quick look at the histogram and set +1 EV to raise the exposure and move the data in the histogram to the right. Capture another image and check the histogram to make sure I was not clipping the whites. The histogram then would tell me whether I should add more exposure, subtract exposure or leave it alone. Always if you have the time check the histogram on the back of the camera. Get the exposure right in the camera and the rest of post processing is easy.

[Nicks Pics]

Thanks for the info John, Joe, Terry. Very helpful.

So just because you got me curious about histograms a little now, I'm thinking about Terry's answer here:

You won’t be able to do that necessarily. In Dave’s excellent explanation he noted that how the histogram “fills” is a function of the tonal values in the scene. Unless you add lighting or somehow change the scene, it is what it is. Some scenes will fill a histogram, some won’t.

If some scenes will, and some scenes aren't expected to fill the histogram from right to left, how do you know that an image is under exposed if it does not use the right side of the image? Do you in either case (whether it does or does not fill the histogram) expect the data to be centered in the middle of the histogram?

[ajohnw]

How much of the histogram is filled / it's shape depends entirely on the range of light levels in the scene.

John
-

[DanK]

[Edited with new information]

I am not certain, but I think this thread may be conflating several issues: type of editing (parametric vs. pixel editing) and bit depth (8 vs 16 bit), and luminosity banding vs. color banding.

In my experience, stretching the histogram in my raw editor (Lightroom) does not produce gaps in the histogram. Or, to be more precise, LR doesn't display gaps in the histogram. That is all I know.

In contrast, stretching the same histogram in photoshop can produce gaps. Moreover, if I understand photoshop correctly. this does not depend on reducing the image to 8 bits. Someone please correct me if I am wrong. Here is an example:

Original image:



LR histogram pre edit:



LR histogram after pulling up whites to expand the histogram:



I then chose 'edit in photoshop cc' for the original image. The working space was set to prophoto. Isn't this enough to insure that PS is working on the image in 16-bit? This produced the same histogram as the first:



I then spread the histogram with a curves adjustment:



Here is the histogram after the adjustment:



Now, for the part that was completely new to me: I reapplied the curves adjustment, this time using a luminosity blend mode. This produces spikes but no banding:





This suggests to me that even in prophoto, stretching the histogram far enough with a normal blend mode, which entails stretching each of the color channels, causes color banding in the histograms, which in turn causes luminance banding, whereas stretching luminosity doesn't.

Am I missing something here?

[ajohnw]

How much "stretch" is needed to cause banding to occurs depends on the bit depth of the actual image that is being worked on. Providing the workspace as defined earlier is equal or bigger than that it has no influence on when banding occurs. The catch with it being the same is that an image might be manipulated to the extent that some of it moves out of the workspaces range so is gone for ever.

So raw files can be stretched more than jpg's because they have a greater bit depth. In both case if some one is working in sRGB/aRGB etc data in the workspace is being colour managed into the display gamut. Mods are made and the changes colour managed into the workspace. This can also be done by storing the modifications as they occur and applying them one after another when the information in the workspace is being colour managed into the display gamut.

Spikes are just another example of the same thing. The bit's can't be manipulated in a manner that results a smooth curve. Again they don't matter providing they don't have any visual effect on the image.

Where people are most likely to have problems in this area is when they repeatedly make contrast adjustments with curves, levels or even a simple contrast adjustment especially with the GIMP. It's intended to be used with a separate raw converter and those are perfectly capable of making curve style adjustments while the full bit depth is available. This is where things like this should be corrected. Colour temperature as well or any other factors directly concerning the conversion.

John
-

[Nicks Pics]

Dan, thanks for your post. Yes, I think we are talking about two different subjects in a way, but I think they may be connected. When I import a 16 bit image into GIMP it does give me a notification that it will be converted into 8 bit. Probably Adobe Photoshop is capable of handling 16 bit, but I'm not sure. Thanks for pointing out the luminosity blend mode. I may check that out.

John, Thanks for your post. I think you have a point about the work flow.

Thanks all for your input!