RAW/NEF versus TIFF - via Camera Raw

[george013]

The D700 has also a choice between lossy or lossless compression and no compression. But if you insert an empty memory card in it, the amount of possible pictures between no compression and lossless compression is the same. There is a significant difference between lossy and lossless.
I just keep thinking how it's possible: a lossless compression in photography.

George

[pnodrog]

The D700 has also a choice between lossy or lossless compression and no compression. But if you insert an empty memory card in it, the amount of possible pictures between no compression and lossless compression is the same. There is a significant difference between lossy and lossless.
I just keep thinking how it's possible: a lossless compression in photography.

George

The pictures available indicator for lossless compression is based on worst-case and no compression is possible. I mainly have my camera set to lossless compression. You can start the day with 1.1K showing as available then take say 300 photographs and still have over 900 of space still available.

On lossy the indication is still a conservative estimation. Same sort of thing when shooting Jpegs.

P.S. Zip files are one form of lossless compression.

[dje]

I just keep thinking how it's possible: a lossless compression in photography.

George

George this Wiki article gives an overview. Central to the concept is the use of variable bit lengths for data rather than a fixed bit length. This allows a reduction of total bits for a given amount of data.

Dave

PS: Please don't ask me for a detailed explanation!

[pnodrog]

George this Wiki article gives an overview. Central to the concept is the use of variable bit lengths for data rather than a fixed bit length. This allows a reduction of total bits for a given amount of data.

Dave

PS: Please don't ask me for a detailed explanation!

Must read the article. Pretty obvious that shadow area's require far fewer bits than highlights. What a conundrum - do I ETTR for lowest noise or ETTL for best compression.....

[dje]

Must read the article. Pretty obvious that shadow area's require far fewer bits than highlights. What a conundrum - do I ETTR for lowest noise or ETTL for best compression.....

Mate just stick the camera on auto and don't worry.

[george013]

The pictures available indicator for lossless compression is based on worst-case and no compression is possible. I mainly have my camera set to lossless compression. You can start the day with 1.1K showing as available then take say 300 photographs and still have over 900 of space still available.

On lossy the indication is still a conservative estimation. Same sort of thing when shooting Jpegs.

P.S. Zip files are one form of lossless compression.

I shoot uncompressed 12 bit raw. The file sizes are fairly constant. I didn't check it, but I think a lot of differences in the nef file size is due to the embedded jpg. I think by extracting the jpg and thumbnail using exiftools and subtracting that size from the nef file size the raw data file size is left. Plus the metadata, camera settings etc.

Zip files are based on repeating values. The change on repeating values in an digital image is very,very,very low. Except for the clipping areas. I remember playing with zipping images: they became larger, much larger.

George

[george013]

George this Wiki article gives an overview. Central to the concept is the use of variable bit lengths for data rather than a fixed bit length. This allows a reduction of total bits for a given amount of data.

Dave

PS: Please don't ask me for a detailed explanation!

I won't ask you.

George

[pnodrog]

Mate just stick the camera on auto and don't worry.

Thanks mate - you may have saved me from another sleepless night. Pretty unlikely really as I am bound to find something else to worry about...

[Pica]

Well, you're the first one
George

You should keep a link to that post.....

Thank you for these supplementary posts to my OP as well.

[Pica]

Unfortunately, Nikon D3xxx and D5xxx cameras actually lossy compress all raw data. You have no option for lossless compression at all. With D7xxx and above, you can choose either lossy or lossless data in your NEF files. FWIW

Hello Tom, Everyone,
Nice to meet you.

Background: I know plenty on photography, however this was new information to me (that D5xxx lossy compress raw). Thank you for bringing this to my attention! I appreciate it since I right now am contemplating, if I on a general level, should go for 14 or 12-bit when out photographing sunsets and more.


FOLLOW-UP QUESTIONS:
(i) After I learned this now, I researched the internet and found a thread on DxO, which touches on this rather briefly. I wonder as I did not find any say official Nikon information, could you (or anyone else) tell me where you have found the information so I can read more about it please?

(ii) You state D5xxx. I test flew Nikon D53000 last year. Just to confirm do I interpret you right: The same lossy compression is in force there too. Is that right? (I would like to compare my present images from the D5600 with my earlier D5300.)

---

(iii) I am an advanced user and I have asked advanced photographers before, however no one has been able to provide the answer to the following . Anyone here please?

If I receive a RAW file from someone where it is unknown if it is a 14-bit image or a 12-bit image - is there any way to learn this for sure (except for just looking at the file size)? If so how? (Is it also in such a case possible to identify if it has been lossy/lossless compressed?)

Thank you for your cooperation!

[dje]

If I receive a RAW file from someone where it is unknown if it is a 14-bit image or a 12-bit image - is there any way to learn this for sure (except for just looking at the file size)? If so how? (Is it also in such a case possible to identify if it has been lossy/lossless compressed?)

Hi Pica

The nef file contains EXIF information and Maker Notes which contain some information on this. You can read this data using a software tool such as EXIFToolGui (which is based on the well known ExifTool software). Here are a couple of screenshots for an image from my D610.

Dave

Extra info: i should also add that you can find this sort of info in Nixon's own raw converter ViewNX2 (and probably the other versions as well).







Added image : ViewNX2

[george013]

Hello Tom, Everyone,
Nice to meet you.

Background: I know plenty on photography, however this was new information to me (that D5xxx lossy compress raw). Thank you for bringing this to my attention! I appreciate it since I right now am contemplating, if I on a general level, should go for 14 or 12-bit when out photographing sunsets and more.


FOLLOW-UP QUESTIONS:
(i) After I learned this now, I researched the internet and found a thread on DxO, which touches on this rather briefly. I wonder as I did not find any say official Nikon information, could you (or anyone else) tell me where you have found the information so I can read more about it please?

(ii) You state D5xxx. I test flew Nikon D53000 last year. Just to confirm do I interpret you right: The same lossy compression is in force there too. Is that right? (I would like to compare my present images from the D5600 with my earlier D5300.)

---

(iii) I am an advanced user and I have asked advanced photographers before, however no one has been able to provide the answer to the following . Anyone here please?

If I receive a RAW file from someone where it is unknown if it is a 14-bit image or a 12-bit image - is there any way to learn this for sure (except for just looking at the file size)? If so how? (Is it also in such a case possible to identify if it has been lossy/lossless compressed?)

Thank you for your cooperation!

The NEF file structure, and probably all the others to, are based on the TIFF file format.
http://lclevy.free.fr/nef/

Going back to my diagram. The central point in this diagram is that RGB raster image. When that raw data is converted to that RGB raster image, it uses the tonal depth of what is used in the camera, 12 or 14 bits. But your screen and printer are nearly always 8 bit. You might wonder what is the use of that higher tonal depth. Editing. Only with editing you make sensitive use of that higher depth.
Saving to disk is just a way of storing that RGB raster image on disk for later use. It can be stored as a straight RGB raster image with the used tonal bitdepth, but you see what the file size becomes when that was 16 bit. TIFF is doing that. All the other file formats are mostly ways to reduce that file size. With each their own specific pro's and contra's. That idea has expanded. Not only the RGB raster image can be stored but also some layers that are the foundations of that image.

Have a look for Exiftools. https://www.sno.phy.queensu.ca/~phil/exiftool/

George

[tclune]

Background: I know plenty on photography, however this was new information to me (that D5xxx lossy compress raw). Thank you for bringing this to my attention! I appreciate it since I right now am contemplating, if I on a general level, should go for 14 or 12-bit when out photographing sunsets and more.


FOLLOW-UP QUESTIONS:
(i) After I learned this now, I researched the internet and found a thread on DxO, which touches on this rather briefly. I wonder as I did not find any say official Nikon information, could you (or anyone else) tell me where you have found the information so I can read more about it please?

I ran across a fascinating web site from a DPReview forum some time ago that showed the decimation tables for a wide variety of Nikon cameras. Stupidly, I did not bookmark the page. The upshot was this: 12-bit lossy compression decimates 12-bit data to approximately 9-bit and 14-bit data to approximately 11-bit. The decimation leaves the darkest data alone, and becomes increasingly draconian as the pixel gets brighter. By the time it hits its stride, the decimation combines four or five data levels at a time into a single level. What I took away from that is that ETTR is highly suspect as a strategy on lower-end Nikons, because you are moving the picture data into the sweet-spot of the decimation.

(ii) You state D5xxx. I test flew Nikon D53000 last year. Just to confirm do I interpret you right: The same lossy compression is in force there too. Is that right? (I would like to compare my present images from the D5600 with my earlier D5300.)

As far as I understand, (WARNING! That isn't very far.) Nikon uses the same decimation tables for a given bit-depth on all cameras. That is, if you lossy-compress 12-bit data on an 810, the routine is the same as would be used on a 3400 or a 5000. If you lossy-compress 14-bit data on the 810, the routine is the same as would be used on a 5100 or later. But I only know what I have read from others on the ever-reliable internet, so approach with caution...

[xpatUSA]

I ran across a fascinating web site from a DPReview forum some time ago that showed the decimation tables for a wide variety of Nikon cameras. Stupidly, I did not bookmark the page. The upshot was this: 12-bit lossy compression decimates 12-bit data to approximately 9-bit and 14-bit data to approximately 11-bit. The decimation leaves the darkest data alone, and becomes increasingly draconian as the pixel gets brighter. By the time it hits its stride, the decimation combines four or five data levels at a time into a single level. What I took away from that is that ETTR is highly suspect as a strategy on lower-end Nikons, because you are moving the picture data into the sweet-spot of the decimation.

As far as I understand, (WARNING! That isn't very far.) Nikon uses the same decimation tables for a given bit-depth on all cameras. That is, if you lossy-compress 12-bit data on an 810, the routine is the same as would be used on a 3400 or a 5000. If you lossy-compress 14-bit data on the 810, the routine is the same as would be used on a 5100 or later. But I only know what I have read from others on the ever-reliable internet, so approach with caution...

From when I had a D50 long ago, I remember a graph of something like that where 4095 levels in gave 683 levels out. The curve looked a bit like the gamma curve for a color space.

P.S. My memory served me well:

http://www.photonstophotos.net/Nikon...ompression.htm

[tclune]

P.S. My memory served me well:

http://www.photonstophotos.net/Nikon...ompression.htm

Thanks, Ted. Duly bookmarked.

ETA: To the OP, please notice that my recollection of NEF lossy compression was inaccurate. There does appear to be different versions of the compression -- although it seems mostly to change across models with time rather than being used to differentiate among models at a given point in time. FWIW

[Manfred M]

So the "bottom line" is:

1. For the "best" image data, shoot either raw uncompressed or lossless compressed; 14-bit, if these are options that your camera / data storage capabilities can provide.

2. Shoot one of the other options if you have space constraints on your storage media. 12-bit uncompressed or lossless compressed is going to provide more data than lossy compressed.

I've always had enough storage with either spare memory cards and computer storage capacity that I have either shot uncompressed raw (on cameras that did not offer lossless compressed) or lossless compressed raw on cameras that do. The tradeoff of lossless compressed versus uncompressed is processing time versus storage space on both the camera and the computer.

In terms of what my screen and printer can reproduce, I'd rather have extra data and throw it out when I create the final image for printing and posting as that gives me the most data while doing PP work.

[DanK]

So the "bottom line" is:
I've always had enough storage with either spare memory cards and computer storage capacity that I have either shot uncompressed raw (on cameras that did not offer lossless compressed) or lossless compressed raw on cameras that do. The tradeoff of lossless compressed versus uncompressed is processing time versus storage space on both the camera and the computer.

In terms of what my screen and printer can reproduce, I'd rather have extra data and throw it out when I create the final image for printing and posting as that gives me the most data while doing PP work.

I agree. Storage is cheap, and even modest-sized cards by today's standards hold so many files that I never come close to filling them up. I almost never use anything but Canon's largest, lossless raw format.

[John 2]

Lack of knowledge doesn't allow me to contribute to the debate but I found this today - so for interest:

https://photographylife.com/compress...compressed-raw

[xpatUSA]

Yes a raw file contains only one value per pixel (one colour)

Indeed, for example, the D700:



The blues do pick up some green, the greens pick up a bit blue and a bit of red and wow look at the IR response of the red!! I mean, IR isn't even a color!

More graphs here:

https://maxmax.com/spectral_response.htm
.

[Pica]

Hello,
This is to at this stage convey great thanks to everyone who has been contributing actively during this Think tank, OP of mine. I appreciate your input and time!

I have read all posts now with interest, will contemplate them and come back if needed.

Thank you for your cooperation forum members and photography peers!