NEF Processing

[george013]

This was after I had bought my first DSLR, so back in 2010. I already owned the full Adobe CS suite, so did not need another editor, but at the time I was interested in tethered shooting. At the time Lightroom, which I have used since the original beta release, did not have this functionality.

I can't agree with your definition of what a raw convertor does. It assembles an image from the raw data, which includes the demosaicing, fixing the gamma and the colour temperature. All the other functionality we see in the conversion software is editing functionality plus whatever other functionality the supplier decided to build in.

Nikon Capture is a raw convertor, parametric editor and tethered capture tool. Adobe Lightroom is a raw convertor, parametric editor, tethered capture tool and cataloging tool.

Maybe you're mixing up with Camera Control, the thethering program of Nikon. I'm not aware of a tethered functionality in CaptureNx.

LR, CaptureNx and more are called converters. They take care of the transformation of the Color Filter Array to a RGB-raster image. And they include tools that work on that RGB raster image. See my diagram.

John,

It's still not very clear to me if you are comparing a RAW and a JPG with each other. I don't think you can in this situation.
I don't know what camera you have. Am I right you can only edit a NEF file, that's written away as a JPG?

George

[Shadowman]

Maybe you're mixing up with Camera Control, the thethering program of Nikon. I'm not aware of a tethered functionality in CaptureNx.

LR, CaptureNx and more are called converters. They take care of the transformation of the Color Filter Array to a RGB-raster image. And they include tools that work on that RGB raster image. See my diagram.

John,

It's still not very clear to me if you are comparing a RAW and a JPG with each other. I don't think you can in this situation.
I don't know what camera you have. Am I right you can only edit a NEF file, that's written away as a JPG?

George

George,

The In-Camera editor converts to JPG from the RAW file, the right side image was imported into Capture NX and then saved as a TIFF. This was shot with the D750 and all Nikons with the NEF processing capability can only be used on RAW files. Additionally, both images were converted from the same RAW file. The other comment I made about the feature is that it can save card space for those who normally shoot RAW +JPG Fine.

[george013]

George,

The In-Camera editor converts to JPG from the RAW file, the right side image was imported into Capture NX and then saved as a TIFF. This was shot with the D750 and all Nikons with the NEF processing capability can only be used on RAW files. Additionally, both images were converted from the same RAW file. The other comment I made about the feature is that it can save card space for those who normally shoot RAW +JPG Fine.

I thought so.

In the NEF file there's an embedded jpg included. If you use a browser like Irfanview it shows you that file. And you can save it. Well, you need to put them on a pc first.
Not sure, but can you save them to jpg without editing?

George

[Shadowman]

I thought so.

In the NEF file there's an embedded jpg included. If you use a browser like Irfanview it shows you that file. And you can save it. Well, you need to put them on a pc first.
Not sure, but can you save them to jpg without editing?

George

George,

Yes you can, you have to make sure you haven't modified Picture Control settings or it will apply those to the converted image. Also, different camera models have different levels of controls, the D750 has full controls, the D60 only has quality and size, WB, EC, and Picture Controls. The D750 also has NR, colorspace, vignette, and D-Lighting.

[Manfred M]

Maybe you're mixing up with Camera Control, the thethering program of Nikon. I'm not aware of a tethered functionality in CaptureNx.

Sorry, it's been too long since I looked at it, but you are right. I am confusing Phase One Capture One that has has tethered capabilities with the Nikon Capture software.

And they include tools that work on that RGB raster image.

No George, this is not how editors work and your diagram is not correct. You have been told this many times, but don't seem to want to understand that. With parametric editors the only time there is anything resembling rasterized data is when it is output to your computer screen through the screen driver / video card. Instructions on how to manipulate the data (parameters) are the only data that is stored and the only rasterization that occurs is at output.

With a pixel based editor, the edits are done destructively or parametrically (depending on what is happening on a specific layer; for instance an adjustment layer), so here the output is a little more complicated, but is governed by layer order. Again the screen driver takes care of issues (such as out of gamut colours, scaling, etc.) and via the video card, these are displayed on screen.

[Black Pearl]

Robin,

I think the key to utilizing the in-camera processing is that you are doing it to a RAW file, sure you can do it to a jpeg but the options are limited to a few quick filters; at least with the NEF tool you can choose more editing options to apply to the image. Also, it just might save some space on the card rather shooting RAW/jpeg fine you can choose which RAW files need that quick adjustment.

Thats exactly what I am doing and have just described?

[george013]

Sorry, it's been too long since I looked at it, but you are right. I am confusing Phase One Capture One that has has tethered capabilities with the Nikon Capture software.




No George, this is not how editors work and your diagram is not correct. You have been told this many times, but don't seem to want to understand that. With parametric editors the only time there is anything resembling rasterized data is when it is output to your computer screen through the screen driver / video card. Instructions on how to manipulate the data (parameters) are the only data that is stored and the only rasterization that occurs is at output.

With a pixel based editor, the edits are done destructively or parametrically (depending on what is happening on a specific layer; for instance an adjustment layer), so here the output is a little more complicated, but is governed by layer order. Again the screen driver takes care of issues (such as out of gamut colours, scaling, etc.) and via the video card, these are displayed on screen.

We've been through this before. Photo editing is pixel based. A picture of x by y pixels, that's the playground. Every pixel can be addressed individual.
Before editing you have to load a picture in memory. That can be the result of a converter, another program passing an in-memory image or just loading from disk. That RGB raster file is where the editor works on, you can see it changing, as a photographer you work visual. All pixel based.
A parametric editor keeps track of what it is doing. It's building a list of commands and the used parameters.
Saving a non parametric editor usually overwrites the source file with the result of the editing. A parametric editor is adding that list of commands to the source file in some way. When opening that file again all those commands are done so you get the end result of the last session.

My diagram is still correct.

George

[Manfred M]

We've been through this before. Photo editing is pixel based. A picture of x by y pixels, that's the playground. Every pixel can be addressed individual.

If you are writing about a pixel based editor like Photoshop, yes. We can and do have access and control of individual pixels if we want that level of control.

If you are writing about a parametric editor like Lightroom, NO. Equations and formulae are applied to the data and individual pixel level control by the user is not possible. All the user controls are the parameters for the individual equations.

If you are thinking digital output on a screen or inkjet printer, where the drivers have to address individual pixels or nozzles, I can see why you might think this way. If you look at commercial chromogenic printers that use traditional photographic papers (Kodak Endura and Fuji Crystal Archive papers), these are an analogue process and individual pixels cannot be addressed.

My diagram is still correct.

You are the only one who seems to believe this.

[xpatUSA]

My diagram is still correct.

George

It is now:

[Manfred M]

It is now:

I think you are oversimplifying here Ted. The "lots more" box needs a very large section dedicated to it. The A/D converter does create data output, but it is not an image file, that's a secondary step using a dedicated image processing chip. Each camera company has its own proprietary name; my D810 has an Expeed 4 chip.


https://en.wikipedia.org/wiki/Expeed

[xpatUSA]

I think you are oversimplifying here Ted.

Is there a Smiley missing Manfred, or is that a serious critique of my tongue-in-cheek fix of the by-now infamous flow-chart?

The "lots more" box needs a very large section dedicated to it.

Quite so. My favorite remains Norman Koren's rather detailed chart:

The A/D converter does create data output, but it is not an image file, that's a secondary step using a dedicated image processing chip. Each camera company has its own proprietary name; my D810 has an Expeed 4 chip.


https://en.wikipedia.org/wiki/Expeed

My cameras' three A/D converters produce a recognizable image straight from the sensor, albeit somewhat subdued. No 'demosaicing' needed.

[Manfred M]

Sorry I forgot the Smiley's Ted. They have been added now.

Koren is a well known authority and while his website is a tad dated, much of the information there is still quite relevant. I see to remember seeing a plug for CiC on his website.

Your cameras are unique with the Foveon sensors. Their advantage is that they work very much like colour film, so none of the issues associated with creating an image from a Bayer filter apply to it (demosaicing and interpolation). The downside to the Foveon are also well known; colour accuracy is not as good as with the Bayer filter and low light performance is not great either.

[george013]

I don't understand why some people want to make things so difficult. Afraid for simplifications?? The diagram is about the use of files., not how the exceed processor works. Can you explain that with that expeed diagram?? Please don act that silly.
Even Tedś addition of "lots more" didn't change the idea of that diagram. He probably missed that idea.

If you are writing about a parametric editor like Lightroom, NO. Equations and formulae are applied to the data and individual pixel level control by the user is not possible. All the user controls are the parameters for the individual equations.

Probably the same formula is used in parametric and non parametric editors. The same pixels are changed on that RGB raster image so you can see directly what you're doing. But in a parametric program the design of that program is to save that used formulae in a list which can be stored on disk. Maybe adding a parametric list on the disk block might satisfy you?

Beside that, why is this so important or this thread?




George

[xpatUSA]

I don't understand why some people want to make things so difficult. Afraid for simplifications?? The diagram is about the use of files, not how the exceed processor works. Can you explain that with that expeed diagram??

Hilarious!

Please don't act that silly. Even Tedś addition of "lots more" didn't change the idea of that diagram. He probably missed that idea.

Being rude to the Moderator will not help you make your point, George.

And speculating on whether I "missed that idea" doesn't go down well with me either.

My editing of your diagram was intended as a joke and your grossly over-simplified and incorrect diagram will itself remain an on-going joke for me and some other members here.

Here are some questions for you to ignore, as you usually do:

Why do you put "color filter array" when not all cameras have one?

Why do you put "Jpg" when everybody and his dog writes "JPEG" for the file type?

Why do you put "tiff" when everybody and his dog writes "TIFF" in caps for the file type?

Why do you put "scherm" when so few people know what that means?

Why do you put "editer" when everybody and his dog writes "editor"?

Why do you still refuse to correct these simple errors?

Eh?

[DanK]

And they include tools that work on that RGB raster image. See my diagram.

Sigh. It's a shame to see both a repetition of the same argument and the same degeneration of the tone of the discussion.

For what little it is worth, I will try to see if I can come up with straightforward language that resolves the former.

George, the core of the reason why nearly everyone here says you are wrong is the sentence I quoted above. It simply isn't right. I'll use Lightroom as an example. For the moment, assume you haven't saved ("exported," in LR-speak) an image, but you are editing one. In that case, LR does have to rasterize the image, but only--as Manfred pointed out, I believe--to display it on screen. The editing itself is being applied to the raw data. There is no intermediate temporary rasterized, edited file to work on.

In contrast, take a look at the temporary files created by photoshop.

At least on a computer like mine (several years old), you can easily see the difference in the behavior of the software. For example, in photoshop, if you use the smart sharpen filter, you will see rapid changes in the screen preview. However, when you decide to accept the changes, you have to wait while the software modifies the scratch file you are working on. In contrast, a sharpening change in LR is nearly instant. It doesn't have to rewrite a working rasterized file, although it does need to rewrite the rasterized image on screen.

You have often insisted that there is no difference until one saves a file, but that simply isn't accurate. As many of us have pointed out, these two different approaches have functional consequences.

[Manfred M]

Very good analysis, Dan.

In a pixel based editor like Photoshop, the changes in an edit have to be applied to the entire image; so if you are shooting a 24MB or 36 MB camera, then there is a lot of data that has to be manipulated. Once the manipulation is done, the image has to be scaled down to a size that can be displayed on your computer screen. This all takes computer and graphics power.

In a parametric editor we face another issue. There is no pixel based data and every effect has to be recalculated from the raw file data every time we make a change. Lightroom cheats a bit here. If you look at the default size of the window that we see our edits in, it is only a fraction of the screen size, which means that the effect only has to be calculated for and displayed on an image that is less than one MB in size. This speeds things up considerably.

Being a parametric editor, every change is applied over and over again to the data from the raw file (or image file if we happen to be editing a JPEG or TIFF). In Photoshop, once the bit-level data has been calculated, the program just has to apply the edit to the data that is already in RAM. A parametric editor is handicapped by having to recalculate each edit every time.

When I use Lightroom, I use a dual screen approach and the refresh on the full size image on my secondary screen is far slower than in the Lightroom Develop Module window. Adobe Camera Raw uses a similar approach and has a separate edit window, which is larger than Lightroom's, but is still smaller than the whole screen.

One of the biggest criticisms that Adobe has promised to address with Lightroom is the speed issues when using stacked local effects, especially with the adjustment brush. These are complex localized adjustments that require far more computational power than a global change. As we stack these effects, the interpretive algorithms have to be applied in order and this really slows things down. The basic approach worked well when the types of edits were limited, but when we look at advances in Lightroom (and Adobe Camera Raw), the Develop Module is really where Adobe has done a lot to increase functionality. Unfortunately, the underlying architecture seems to be limiting how far they can push things.

[DanK]

Manfred,

Your post also explains why the order of edits in LR doesn't matter--something that has been the subject of several threads over the years. The software recomputes every time one adds an edit, and from what I have read, it doesn't apply the edits in the order carried out by the user.

As I understand it, this is why Adobe placed lens correction toward the bottom of the develop panel. Logically, it makes more sense to do it at the outset because it includes local tonality adjustments, but it is computationally complex, so putting it toward the bottom won't slow down the editing process much, if people follow the general advice to work from top to bottom.

I usually use local adjustments sparingly in Lightroom; if an image needs a lot of tinkering of that sort, I often move it to Photoshop. As a result, speed isn't usually a problem, even though I do a few local adjustments (lens profile correction and sometimes a bit of dodging or burning) early in the editing process.

Dan

[george013]

...
George, the core of the reason why nearly everyone here says you are wrong is the sentence I quoted above. It simply isn't right. I'll use Lightroom as an example. For the moment, assume you haven't saved ("exported," in LR-speak) an image, but you are editing one. In that case, LR does have to rasterize the image, but only--as Manfred pointed out, I believe--to display it on screen. The editing itself is being applied to the raw data. There is no intermediate temporary rasterized, edited file to work on.

Here we disagree. All editing is done on your raster image. You can see it changing, it's the same as what you see on your screen. I think, not sure, the only editing what is done with the help of the raw data is the white balance. In CaptureNx I can open a jpg. I believe that's possible in LR also and do several editings. Try to find out which editing are not possible. You can't change contrast, adjust sharpening, saturisation etc on the raw data. They are PIXELBASED. The software is examining the image based on what we see and investigates the pixels in relation to each other, well mostly.

When you export a file a diskfile has to be created. With compression or not, the normal headers, the metadata and what else? That rasterfile already exists. The same happens with Saving but the parametric editors have changed the meaning of it. Here only the sidecar files are written away.

In contrast, take a look at the temporary files created by photoshop.

At least on a computer like mine (several years old), you can easily see the difference in the behavior of the software. For example, in photoshop, if you use the smart sharpen filter, you will see rapid changes in the screen preview. However, when you decide to accept the changes, you have to wait while the software modifies the scratch file you are working on. In contrast, a sharpening change in LR is nearly instant. It doesn't have to rewrite a working rasterized file, although it does need to rewrite the rasterized image on screen.

You have often insisted that there is no difference until one saves a file, but that simply isn't accurate. As many of us have pointed out, these two different approaches have functional consequences.

There's no essential difference. And they've functional consequences.
The intention of that diagram was to show the flow of the files, how and in which sequence they are used. Not about how an editor works. It shows you can't "see" a raw file, you can't print a raw file.
Go to your favorite program/converter. Do some editing on a big file. Enlarge to 100% and travel with your window through the image. I don't think you'll see a delay.

I mentioned before, if it's important just add sidecar file to the list of diskfiles. And for Ted change the Color Filter Array in raw data or something else he is satisfied with. It doesn't change the intention of that diagram. Which I original made just for myself.

George

[DanK]

All editing is done on your raster image.

Open a raw file in Lightroom. make a few edits. On your computer, find the raster file your edits are working on (not the image for screen, which both Manfred and I explained is a raster image).

Until then, I am tuning this discussion out.

[xpatUSA]

Open a raw file in Lightroom. make a few edits. On your computer, find the raster file your edits are working on (not the image for screen, which both Manfred and I explained is a raster image).

Until then, I am tuning this discussion out.

Me too.

Just for interest here's how Sigma does it (pre-Merrill):

When a raw (X3F) file is opened, an embedded JPEG is shown in the review window initially, while the raw part of the file is converted to a temporary "working" file which is in linear Kodak RIMM (input-referred) space. Then that file is transformed into a review image according the X3F relevant meta-data (for example, ISO multiplier value). Any adjustment thereafter causes the whole review image to be re-done ab initio from the "working" file. When exporting, the output file is created with a choice of file type, file size and of course bit depth and color space, and "working" file simply disappears. Interestingly all the current adjustments can be saved back into the raw file, an embedded "side-car" so to speak - but the original settings are still retrievable .

Then Sigma, Consistency 'R' Us, changed everything for the Merrill and later models . . ho hum, won't bother you with all that . .