Camera profiling - matching values on Macbeth CC passport

[spittle]

I created a camera profile using HDRShop. I took a series of 20 exposures at 1/3 stop increments of a couple of grey cards (one in sunlight, one in shadow) and a colorchecker (CC) passport in sunlight saving out the resulting curve. Note: the curve is created by sampling the whole values on the whole image and not just the colour checker or any specified area.

My source image were processed in dcraw using the following:

-v -H 0 -o 0 -w -M -q 3 -W -g 1 0 -6 -T

My in camera white balance was set to 6500k and I outputted RAW colour.

As a comparison I loaded one of my tiffs into HDRShop with a linear curve, and the same tiff again using the camera profile.

Visually the profiled one looked as if the blacks had been lifted.

Image from dcraw (converted in Nuke to sRGB for viewing purposes):



Same image loaded with profile:



The profiled one looks a little fishy to me so I tried to figure out how I could verify the info.

I tried this:

1) found sRGB values for the grey swatches on the CC passport, linearized the values and calculated the % difference in Excel.

2) sampled grey swatches on tiffs (the original linear ones), multiplied the values on the photo so that swatch three matched that of the CC passport.

3) calculated % difference between each swatch and compared results.

non linear CC grey values, converted to 0-1, % difference:

245 0.95703125 122.50%
200 0.78125 124.22%
161 0.62890625 133.06%
121 0.47265625 147.56%
82 0.3203125 167.35%
49 0.19140625


Linearized grey swatch CC values and % difference between them:

0.90008 152.26%
0.59113 163.31%
0.36197 183.08%
0.19771 219.65%
0.09001 287.57%
0.0313 #DIV/0!

Converted RAW
0.91957015 146.27%
0.628664784 161.87%
0.388373952 196.44%
0.197710051 182.79%
0.108161262 248.69%
0.043492413 #DIV/0!


Converted Raw with response curve applied
0.878818434 145.33%
0.604697662 159.90%
0.378172693 191.28%
0.197710009 178.82%
0.110564632 211.14%
0.052366742 #DIV/0!

I would have thought I would be able to get closer to the results as I thought I've been quite careful with my processing workflow.

I'm aware that the colour spaces (RAW compared to sRGB are different) but on previous tests the difference between these two spaces tended to be with the colours more than the luminosity. I should probably try again and stick with sRGB space initially.

Any ideas?

[revi]

How did you handle images with clipped zones (shadows pure black, or light parts saturated in one or more channels)?

[spittle]

With ignorance I guess. I'm an optimist.

Would this be the point where I need to calculate the exact saturation and black points for the raw processing?

[Manfred M]

Honestly, I cannot fathom your methodology, and so far as I can tell what you are doing makes no sense from a technical standpoint. You wonder why your results are not coming out the way you would have expected; look at what you are doing.

1. If you are trying to generate calibration curves with the Colorchecker Pro, then only it should be included in your image you are analyzing and each colour patch should be validated individually against the known colour value of that patch. That will give you the data to create a decent calibration curve for each channel.

2. You cannot output "raw colour"; that simply does not exist. Your raw data is simple the data that the camera has captured; it's not an image or anything else. It does not have a colour space. The colour temperature setting on your camera does not affect it in any way. It must be converted to an image file using a raw converter and a white balance / colour temperature value is assigned at that time. Many editors will take the white balance / colour temperature that you set for the jpeg and use it as the starting point for white balance, but that is neither here nor there. The easiest way I know is to ensure each of the colour channels reads the same when you sample a neutral gray patch (and you have lots of those in your image).

3. Neither the black points nor white points seem to be close to being clipped. Your gray values (taken from the lower gray card) seem to be about 5 units light in the blue channel (i.e. a touch too much yellow), but the red and green channels look to be right on.


Performing an analysis on incorrect data seems to be rather meaningless to me. Figure out your colour theory and experimental methodology before you get stared. That should save a bit of time and frustration. Other you are dealing with the age old issue of "garbage in = garbage out". Why aren't you using the software and methodology that comes with the Colorchecker Pro? It has a proven track record and is used by many photographers

[spittle]

Honestly, I cannot fathom your methodology, and so far as I can tell what you are doing makes no sense from a technical standpoint. You wonder why your results are not coming out the way you would have expected; look at what you are doing.

1. If you are trying to generate calibration curves with the Colorchecker Pro, then only it should be included in your image you are analyzing and each colour patch should be validated individually against the known colour value of that patch. That will give you the data to create a decent calibration curve for each channel.

2. You cannot output "raw colour"; that simply does not exist. Your raw data is simple the data that the camera has captured; it's not an image or anything else. It does not have a colour space. The colour temperature setting on your camera does not affect it in any way. It must be converted to an image file using a raw converter and a white balance / colour temperature value is assigned at that time. Many editors will take the white balance / colour temperature that you set for the jpeg and use it as the starting point for white balance, but that is neither here nor there. The easiest way I know is to ensure each of the colour channels reads the same when you sample a neutral gray patch (and you have lots of those in your image).

3. Neither the black points nor white points seem to be close to being clipped. Your gray values (taken from the lower gray card) seem to be about 5 units light in the blue channel (i.e. a touch too much yellow), but the red and green channels look to be right on.


Performing an analysis on incorrect data seems to be rather meaningless to me. Figure out your colour theory and experimental methodology before you get stared. That should save a bit of time and frustration. Other you are dealing with the age old issue of "garbage in = garbage out". Why aren't you using the software and methodology that comes with the Colorchecker Pro? It has a proven track record and is used by many photographers

1) I'm using this calibration process which doesn't specifiy that you need to only photograph a chart

http://gl.ict.usc.edu/HDRShop/HDRSho...tutorial2.html

this method was also outlined here: http://www.amazon.com/High-Dynamic-R...+range+imaging

2) Sorry I'm not fluent with this terminology. By Raw colour I mean no colourspace. I used the in camera white balance which was applied to my image after reading this):
http://www.radiance-online.org/piper...er/000140.html

I figured perhaps this is why a lot fo the RGB colour spaces use 6504k as a standard <- I admit I haven't quite wrapped my head around this but figured by using this white bakance I'd be on the right path.

3) That's the reason I came here - to try to get help with this. I'll try the Xrite software but figured the calibration method that I linked to would be robust.

[Digital]

I really am going to show my ignorance here. Just what are you out to prove?



Bruce

[spittle]

I'm trying to prove that my images are linear.

The purposes of the imagery is to match real world lighting intensities inside of a 3d package and I'm trying to somewhat follow methods described in the book that I referenced before. But the techniques don't discuss specific software.

[xpatUSA]

The profiled one looks a little fishy to me so I tried to figure out how I could verify the info.

I tried this:

1) found sRGB values for the grey swatches on the CC passport, linearized the values and calculated the % difference in Excel.

Any ideas?

David,

Please pardon the snipping. One question:

As you know, the Passport sRGB values are published for an imaginary illuminant (CIE D65) at some specific illuminance.

How are you accounting for the numeric difference between the imaginary illuminant and your scene illumination?

[revi]

1) I'm using this calibration process which doesn't specifiy that you need to only photograph a chart

http://gl.ict.usc.edu/HDRShop/HDRSho...tutorial2.html

this method was also outlined here: http://www.amazon.com/High-Dynamic-R...+range+imaging

Sorry, but from what I can see you did not follow that method:

1 - They suggested 4-5 images, which would cover at most a 2EV range of exposure difference.
You used 20 images with 1/3 EV between them, which would cover 6 2/3 EV of exposure difference. Which means that the only useable pixels are in the center of the exposure range, and that alone can get you in trouble with the calibration.

2 - That method is used to calibrate deviations from γ=2.2, you feed it images with a γ=1.0. The poor solver will have to work hard to correct the deviations from gamma=2.2 ...

I admit that I had to read it two or three times (and most of it between the lines, at that) to get an idea to what they wanted as input. Not good for instructions for a calibration method :/

2) Sorry I'm not fluent with this terminology. By Raw colour I mean no colourspace. I used the in camera white balance which was applied to my image after reading this):
http://www.radiance-online.org/piper...er/000140.html

As a colour space defines how the RGB values map to a visible colour, you cannot have an image with no colour space: as soon as you display an image, you have a mapping of RGB values to a visible colour. And in practice, if none is defined in your image, sRGB will be used.

I figured perhaps this is why a lot fo the RGB colour spaces use 6504k as a standard <- I admit I haven't quite wrapped my head around this but figured by using this white bakance I'd be on the right path.

3) That's the reason I came here - to try to get help with this. I'll try the Xrite software but figured the calibration method that I linked to would be robust.

Those calibration methods have different aims. From what I saw in your link, the HDRshop calibration tries to correct for deviations from a gamma curve, with no regards for colour accuracy per se. It also seems to want developed images (i.e. no raw data), supposedly with the in-camera conversion (they are not really explicit about that, to say the least) and with a supposed gamma of 2.2.

The Xrite calibration is interested in matching the colour swatches to the theoretically expected values. So, different tools for different goals.

But in any case, whatever calibration you want or need to use, the first things to do are to understand the purpose of the calibration, and follow the instructions...

[chauncey]

I must confess that my knowledge of using the Passbook is limited to using it in LR for people pics...
clicking here and there until I get the WB that I think works best on that series of images.

Beyond that, unless you consider your work to be documentary in nature, proper WB is purely a
subjective decision, relegated to one's, probably faulty, memory of the scene in question.
And, as illustrated by Ansel Adams, tis indeed a rare photographer that post processes a image
the same way twice.

So, I ask, what is the value of trying to match your camera with a color chart when it's artistically
unnecessary, or, just requires a couple of clicks in PP when it is needed?

[Iliah Borg]

> I'm trying to prove that my images are linear.

The sensor having linear response throughout the working range of the characteristic curve does not mean the image is linear - because of flare, glare, and some other, less important, effects like cross-talk (optically end electronically).

Your setup is very prone to glare and flare, you even have a specular highlight on the metal clip on the strip.
Let's start with the values. Let's work with ColorChecker Passport 6-patch grey step wedge that goes in the following progression, white to black:
-0.15 EV; -0.77 EV; -1.51 EV; -2.37 EV; -3.50 EV; -4.91 EV
The above is based on "traditional" metering, 2.45 EV from the spotmeter readings to the clipping of the highlights. To relate it to a particular camera, one needs to determine the headroom in highlights. This takes 2 shots: one that is fully blown out, to determine the maximum in raw; the other one of a uniformly lit neutral surface (like a grey card), exposed as the camera spotmeter suggests. Bringing both shots into RawDigger, one by one, determine the average levels on the blown-out and grey card shots, divide, and take the log base 2: Headroom in highlights, EV = log2(AverageMaximum/AverageGray). You will need the AverageMaximum for the next, too.

The reference RGB values for the same 6 patches, obtained through the conversion of the spectral reference, in gamma = 1, scaled to 0..255, green channel:
229.95; 149.6; 89.73; 49.57; 22.6; 8.48
Btw, you can obtain the EV scale above through the following: log2(255/GreenChannelValue); log2(Green1/Green2) will amount to the step in EV between the patch having Green1 and the patch having Green2.

Knowing the AverageMaximum for the camera (ISO-dependent on Canons and some other make/models, btw), rescale each green channel value above by dividing it by 255 and multiplying by the average maximum.

From here:
- sample the raw values of the grey patches through RawDigger,
- determine the underexposure factor as the ratio between the reference white and raw white,
- multiply the raw values for the green channel by that factor,
- check the linearity by plotting against the reference RGB values.

You will probably see how the black patch is affected by flare, especially if it is 1-2 stops underexposed.

I know it is a lot of words to digest, but it is easier to do than to describe

--
Best regards,
Iliah Borg

[Iliah Borg]

> You cannot output "raw colour"; that simply does not exist.
"Raw colour" means no colour profile is assigned.

> Your raw data is simple the data that the camera has captured; it's not an image
It is an image, in most cases it is a rasterized TIFF image with gamma=1 and no colour spaced explicitly assigned; it is impossible to convert a "non-image" into an image with simple arithmetics.
--
Best regards,
Iliah Borg

[Manfred M]

David - I've read some of the links that you have posted and now question the proposed methodology even more.

You are trying to prove your images are linear? Sensors are linear by design (or at least the linear part of the response curve will be used; as an example, you don't care if the near IR and UV are non-linear, as there won't be any data recorded as these wavelengths are filtered out before the light hits the sensor) and this is output by your camera in a raw file. Go and convert that raw data into an image file, and you will have a gamma adjustment applied to it. So the methodology you are using is to get a processed image file that has a gamma curve applied to it and then to try to reproduce any gamma errors to back that out again?? Somehow this does not make a lot of sense to me...

Raw data has neither WB nor gamma assigned to it, although this is something that has to be done when you create an image file and these values are "baked in" at the time that the image is created, regardless of the image format you are using.

Have you read this article? https://www.cambridgeincolour.com/tu...correction.htm

Let's talk a bit about experimental methodology. It's really critical to work under consistent conditions and ONLY adjust one variable at a time. If some of the variables can be shown to have little to no impact on what you are doing, then you are likely safe in ignoring those, but you have understand the magnitude of their impact before you make that determination.

The first step to calibrating anything would be to light the subject with a consistent light known (daylight?) light source. Placing your samples outdoors in the sunlight simply does not cut it. If I were trying to do what you are, I would do this in a studio setting using studio flash (~5500K). Consistency and repeatability are absolute requirements when you are doing calibration work.

Also, I'm pretty sure that the methodology in the article you linked to cannot work. By changing aperture setting you are introducing a host of other variables into the "calibration" images that can and will throw any attempt to measure the same point across a number of images as meaningless. Assuming that one is using a fixed focal length lens (to eliminate the impact of focal length changes):

1. Vignetting will found in wide aperture images and disappear as you stop down. This means parts of your image will be darker between the shots. Do you compensate for this in your methodology, if so how?

2. Diffraction will affect pixel distribution at small apertures, so different values should be expected between large and small aperture images. How does the calibration methodology account for this?

3. Depending on the lens used, you could be introducing other variables related to, sharpness (most lenses tend to have a "sweet spot" in the f/5.6 - f/11 range), depth of field impact, aberrations and other lens design "artifacts" that may be impacted by aperture changes, etc.

Conclusion: Using different aperture settings will likely introduce measurement errors that are independent of the variable (gamma) that you are trying to measure and are likely going to result in data that does not give you meaningful results.

One of the articles you mention has a 2006 date. The underlying constraints / assumptions may or may not be valid any longer. Many of the "affordable" cameras at the time did not have raw capabilities and output was limited to the sRGB colour space. This has certainly changed over the past 9 years.

[Iliah Borg]

The gamma is indeed often misunderstood. Nature has linear gamma, so our eyes are expecting images in linear gamma (just like sensors record it); but the output media often has non-linear gamma. So we "raise" the gamma only to compensate for the media "lowering" it; the image on the media is in near-linear gamma (slightly higher, like 1.15 to 1.3, to compensate for the loss of contrast due to flare).
--
Best regards,
Iliah Borg

[Manfred M]

The gamma is indeed often misunderstood. Nature has linear gamma, so our eyes are expecting images in linear gamma (just like sensors record it); but the output media often has non-linear gamma. So we "raise" the gamma only to compensate for the media "lowering" it; the image on the media is in near-linear gamma (slightly higher, like 1.15 to 1.3, to compensate for the loss of contrast due to flare).
--
Best regards,
Iliah Borg

Carefully said, your statements regarding our eye and nature need to be challanged. Neither our eyes nor the light we see things under (predominantly sunlight) are linear.

The way we see (i.e. our eye / brain combination) nicely masks this from us human beings, but not from our linear camera sensors.


Attribution: Nick84 - http://commons.wikimedia.org/wiki/File:Solar_spectrum_ita.svg





The response curve of the cones in our eyes is hardly linear either; humans are extremely sensitive to the mid frequencies (green receptors) and have decent response at the red end, but our blue vision is relatively weak. Look at the frequency response curve of the human eye in this article.

http://www.physicsclassroom.com/Class/light/U12L2b.cfm

For a decent view of what gamma is all about, I would like to direct you one of the CiC tutorials, as well as a White Paper that Jeff Schewe, a well known resource on colour in photoography put out for Adobe some years ago. Some of the discussion around the Camera Raw convertor are a touch dated, but the underlying physics and principles are not.

https://www.cambridgeincolour.com/tu...correction.htm

https://www.adobe.com/digitalimag/pd...kflow_sec2.pdf



Another useful link is: http://www.brucelindbloom.com/

[Iliah Borg]

>your statements regarding our eye and nature need to be challanged.

Do you disagree with "Nature has linear gamma, so our eyes are expecting images in linear gamma"?
So, if the subject is lit with twice the luminous intensity, an exposure meter does not recommend to half the exposure?

Linear relation between the luminous intensity of the light source and the amount of reflected light from a subject is a fundamental law of Nature. Our eyes are expecting a linear scene to be presented, be it in Nature or on a media, with gamma = 1. That is why in photography gamma of the paper compensates for gamma of the film, for total gamma close to 1. The response of our eyes enter the equation at a later stage.

[xpatUSA]

Originally Posted by Iliah Borg
The gamma is indeed often misunderstood. Nature has linear gamma, so our eyes are expecting images in linear gamma (just like sensors record it); but the output media often has non-linear gamma. So we "raise" the gamma only to compensate for the media "lowering" it; the image on the media is in near-linear gamma (slightly higher, like 1.15 to 1.3, to compensate for the loss of contrast due to flare).
--
Best regards,
Iliah Borg

Carefully said, your statements regarding our eye and nature need to be [challenged]. Neither our eyes nor the light we see things under (predominantly sunlight) are linear.

The way we see (i.e. our eye / brain combination) nicely masks this from us human beings, but not from our linear camera sensors.

Good references Manfred but the words in these two posts seem to be being used to describe different things.

For example, most surfaces have a linear radiometric reflectance characteristic for any given wavelength. That is to say flux out = reflectance times flux in, irrespective of the spectral characteristic of the of the light and irrespective of the spectral reflectance of the surface. Therefore the term 'linear' should always be qualified. And, with respect to wavelength-dependent parameters, I would prefer to see the inclusion of the word "spectral", or "spectrum" for the ungrammatical amongst us

We can not really just say that our sensors are linear, either:

Their response to illuminance (weighted irradiance) is indeed linear (in terms of electrons captured vs. lux) in the preferred range of use but their spectral response is highly non-linear.

I could go on and on about the abuse of the "gamma" in these fora but I'll try to keep it short. Used by itself, the word means nothing, other than the name of a Greek letter. The word as used in these fora refers to several different characteristics which should always be stated whenever it is used.

Rant over. You're both right in the intent of you posts, IMHO.

Welcome to the forum, Iliah, [looks like we were typing at the same time]. I'm sure that you personally have no need of tutorials, if your posts elsewhere are anything to go by

[Iliah Borg]

I'm speaking just of gamma being linear in the scene. In photography, to illustrate and compute tonal mapping, one of the tools is Jones diagram (also known as quadrant method). Mees explains this in his Photographic Reproduction of Tone (Kodak, 1921) One can also find the graphs and explanations in Focal Press books such as "Basic Photo Science", "Manual on Photography", "Advanced Photography", and "Way Beyond Monochrome". The last one is of particular interest in this respect as it starts with two lines in Q1 - one ideal, and one accounting for flare and glare in the camera (lens, chamber, etc).

The usual misconceptions in relation to gamma stem out of mixing up two things, Nature and our perception of Nature; as well as not including the closed circuit (full diagram of tone reproduction, from scene to presentation) into discussion.

[Cantab]

I've been following this discussion with interest -- despite, or perhaps because, of having absolutely no relevant expertise. For me, an important point is made at the very beginning of the CiC tutorial on gamma:

"Our eyes do not perceive light the way cameras do. With a digital camera, when twice the number of photons hit the sensor, it receives twice the signal (a "linear" relationship). Pretty logical, right? That's not how our eyes work. Instead, we perceive twice the light as being only a fraction brighter — and increasingly so for higher light intensities (a "nonlinear" relationship)."

[Iliah Borg]

I've been following this discussion with interest -- despite, or perhaps because, of having absolutely no relevant expertise. For me, an important point is made at the very beginning of the CiC tutorial on gamma:

"Our eyes do not perceive light the way cameras do. With a digital camera, when twice the number of photons hit the sensor, it receives twice the signal (a "linear" relationship). Pretty logical, right? That's not how our eyes work. Instead, we perceive twice the light as being only a fraction brighter — and increasingly so for higher light intensities (a "nonlinear" relationship)."

Yes, but this has nothing to do with gamma. Gamma is the method of compensation for the non-linearity of capture and non-linearity of presentation. Generally speaking, a poor method, to boot.