image quality and file size

[Windytoo]

I've just started with digital photos. I'm trying ti get the whole pixel, resolution, image size thing into my head. I take a raw photo at 25mb, then save as a tif which auto saves at 42mb, I save as jpeg at 2.8mb. When comparing even at 1200 zoom I see no real difference. Properties shows no pixel, dic size, or resolution changes. Why is this?

[dan marchant]

RAW files are not images, they are all the data captured by the sensor, which is why they are large. When you view the RAW file on your camera or after importing into Lightroom a preview jpeg is created. This will only use some of the data but it is there and takes up space. This extra data is the reason why you can make such dramatic changes to a RAW file.

A TIFF file is an uncompressed image file. If you convert a RAW file to TIFF it will also contain a lot of data, allowing for greater flexibility in processing/editing but again a larger file size.

Jpeg is a compressed image format. When you convert the RAW file to jpeg it compresses the file by deleting any unused image info. This makes for a much smaller image but reduces your ability to alter the image later.

[dje]

For a start, the TIFF file is larger than the raw file because with the raw file, the data for each pixel is one colour only. When it is de-mosaiced in the raw processing software, info on the other two colours is estimated and added in. So there is 3 times as much data in the TIFF. The size of the TIFF will also be influenced by whether it is 8 bit or 16 bit.

Resolution is just the number of pixels in the image. If you convert the TIFF to a JPEG, without changing the resolution, then the file will get much smaller due to the compression process used in the encoding of the jpeg. When you compare the TIFF and the JPEG, even if you zoom in considerably, the only difference should be due to the degradation caused by the compression. If you use a high quality setting to save the jpeg, this degradation will be at a minimum.

The jpeg compression algorithm gives pretty good results, given the considerable reduction in file size it produces.

Dave

[John Morton]

For a start, the TIFF file is larger than the raw file because with the raw file, the data for each pixel is one colour only. When it is de-mosaiced in the raw processing software, info on the other two colours is estimated and added in. So there is 3 times as much data in the TIFF. The size of the TIFF will also be influenced by whether it is 8 bit or 16 bit.

Dave

Very good point! Also, TIFFs are a bitmapped format which means each and every pixel is defined by its specific location on an X/Y axis (like the squares on a piece of graph paper) and each of these co=ordinates is then given its specific color values.

JPEGs define images in terms of algorithms, with equations defining large areas of the same color (instead of a lot of discrete pixel locations on an X/Y axis). The basic unit for this is a block of pixels, with greater compression ratios using ever increasing sizes of pixel blocks.

If you edit a TIFF then the changes are saved on a pixel level, with each new value for each and every pixel being recorded for every pixel location.

If you edit a JPEG then the changes are necessarily applied to the blocks of pixels which define the compression ratio, because that is what is there when the JPEG is opened. If you keep editing and saving the JPEG, then these edits will cause the file compression to include ever increasing numbers of pixels in the compression blocks, as more and more pixels are judged by the compression algorithm as being "the same." Just opening and saving a JPEG won't make much if any difference, but, once you start editing then the image quality of the JPEG decreases when it is edited and saved repeatedly.

Different programs may use different compression algorithms to produce their JPEGs so, editing and saving a JPEG across several programs is a fairly quick route to image degradation.

Since TIFFs are bitmapped files with specific locations and color values defined for each pixel, they can be edited and saved repeatedly in a string of different programs with no degradation of image quality.

You might not see any difference between a TIFF and JEPG version of your favorite image, but, after a few years of edits and revisions the TIFF would still be of its original quality while the JPEG might exhibit visible compression artifacts such as banding in gradients, jagged edges along diagonal lines, visible blocks of colors in solid areas such as sky, and wavy outlines along the edges of objects.

[revi]

Very good point! Also, TIFFs are a bitmapped format which means each and every pixel is defined by its specific location on an X/Y axis (like the squares on a piece of graph paper) and each of these co=ordinates is then given its specific color values.

JPEGs define images in terms of algorithms, with equations defining large areas of the same color (instead of a lot of discrete pixel locations on an X/Y axis). The basic unit for this is a block of pixels, with greater compression ratios using ever increasing sizes of pixel blocks.

If you edit a TIFF then the changes are saved on a pixel level, with each new value for each and every pixel being recorded for every pixel location.

If you edit a JPEG then the changes are necessarily applied to the blocks of pixels which define the compression ratio, because that is what is there when the JPEG is opened. If you keep editing and saving the JPEG, then these edits will cause the file compression to include ever increasing numbers of pixels in the compression blocks, as more and more pixels are judged by the compression algorithm as being "the same." Just opening and saving a JPEG won't make much if any difference, but, once you start editing then the image quality of the JPEG decreases when it is edited and saved repeatedly.

Different programs may use different compression algorithms to produce their JPEGs so, editing and saving a JPEG across several programs is a fairly quick route to image degradation.

Since TIFFs are bitmapped files with specific locations and color values defined for each pixel, they can be edited and saved repeatedly in a string of different programs with no degradation of image quality.

You might not see any difference between a TIFF and JEPG version of your favorite image, but, after a few years of edits and revisions the TIFF would still be of its original quality while the JPEG might exhibit visible compression artifacts such as banding in gradients, jagged edges along diagonal lines, visible blocks of colors in solid areas such as sky, and wavy outlines along the edges of objects.

Well, no, or at least not quite.

I know you like/prefer TIFF, but you are going a bit overboard here.

For starters, in any bitmapped format "each and every pixel is defined by its specific location", and that includes JPEG. That is (afaik) the definition of a bitmapped image...TIFF is in no way superior to any other losslessly compressed format in existence in that respect. And note that TIFFs are very often compressed as well, so no more pixel-by-pixel description within the file.

And JPEG compression is a bit more complex than just saving variable sized blocks of pixels as single values. And compression ratio is not defined by a block size. Nor do repeated compressions increase the block size.

Although JPEG compression works on blocks of pixels (of a fixed size), it is not treating those blocks as homogeneous (luckily...). What happens is that the data per block is subjected to a mathematical treatment that allows the unimportant finest detail to be discarded (discrete cosine transform with rejection of part of the resulting coefficients, to keep it simple, there's a lot more to it). However, that's only for file storage.

The compression algorithm is prescribed by the standard, so different programs will use the same algorithms. If they would use different algorithms to do the compression, decompression by a program other than the one writing the file would be hazardous or impossible. So using different editors will not induce faster image degradation (multiple editing cycles is different, see below).

However, that's only for file storage.
For editing, there's no difference whatsoever between a TIFF and a JPG (or any other file format): the editor 'sees' an array of uncompressed pixel data in memory, which is not dependent on the storage file format (first thing your editor does on loading a file is decoding it). That means that any edit is still applied on a pixel by pixel basis, and not on a block. Good thing too, otherwise e.g. masking would be impossible.

Now, repeated loading, editing and saving in JPEG can cause image degradation, but the first few cycles won't be visible if you use a moderate compression (quality at 90%, or 10/12 in photoshop), see references below. What most certainly won't happen is that you'll get "ever increasing numbers of pixels in the compression blocks" as you state, the block size is fixed by the standard. And the time over which you do your edits most certainly doesn't play role...

That said, I wouldn't use JPEG as an intermediate format (but then I start from RAW), and I always leave my original file untouched, be it RAW or JPEG.
But neither will I use TIFF:
- TIFF files are much (2x) larger than PNG files of the same image (and PNG is also a lossless format that allows 16 bit/channel), or
- TIFF contains only part of the information that's contained in the native editor format, if I need to store intermediate multilayer images.
So I use either PNG (finished images or RAW converter output) or native editor format (.xcf or .psd depending on the editor).
Then, in a final step, the image is resized, sharpened as needed and converted to JPEG as a function of the planned use.

So, if you shoot in JPEG, and want to edit, optimum is to use a lossless format to store the intermediate and final result, and to store the original JPEG as archive (and and always work on a copy of the original, if only to avoid complete loss of the image). But if for whatever reason you cannot use e.g. PNG for intermediate results, try and do as much of the needed editing in one session, and limit the number of open/edit/save cycles as much as possible (NOTE: repeated saving without closing during an edit session will not cause extra degradation). What I mean is, if you are not happy with an edit, don't keep modifying the same copy, but start over from a fresh copy of the original JPEG.


Regards,

Remco

Some references (Google only gave 1.7 million results on 'jpeg compression image degradation'):
http://en.wikipedia.org/wiki/JPEG describes the JPEG compression algorithm;
http://www.prophotoshow.net/2008/03/...dits-compared/ compares several treatments of JEPG files (and see the 'resize' test!)
http://www.michaelfurtman.com/jpeg_myths.htm (note the number of editing/save cycles before he notices degradation) and
http://www.patricktaylor.com/1940 (don't forget to read the comments and note the number of 'save as..' cycles) discuss the (lack of) degradation caused by JPEG compression

P.S. if you save the JPEGs at too low a quality setting and do repeated open/edit/save cycles, you will soon end up with visible square blocks; those are compression artifacts, but will always be the same size. And they will not be blocks of solid colour either. But note that part of the problem is the user-selected quality setting!

PP.S. I'm not saying that TIFF is a bad format. But it's not the superior format to be used to the exclusion of all others, either...

[John Morton]

May I respectfully refer the discussion to Tom's excellent explanation of JPEG compression; Re: Image post processing and printing; RAW/TIFF Printing; message 14:

*Originally Posted by*John Morton*... so perhaps I am completely wrong about the degradation of image quality caused by successive 'saves' of a*JPEG*file!

Reply:

No, you just don't understand what "generational loss" is all about. If you open a*JPEG*and resave it using the same application at the same settings, the only change from one save to the next will be caused by the round-off error in the routine. That quickly approaches 0 for each successive save. Where generational loss is important is any of the following conditions:

1. You change the application that you use. In that case, the quantization tables may be different between the two applications (Quant tables in essence low-pass the luminance and chrominance data of the DCT-transformed 8x8 "GOPs" of the image. If the two applications use different low-pass filters for their quant tables, or use different "quality factors" applied to the commonly-employed sample quant tables presented in the original*JPEG*standard, there will be significant generational loss between compressions -- even if the image itself actually grows in size!) A good way to track these changes is by diff'ing the images and histogram-equalizing the resulting image.

2. If you apply any editing to the image before resaving, you will get additional loss on the edited areas when the quant tables low-pass the newly-modified regions. The rest of the image will be affected only by round-off error when saved, but the edited regions will have new frequency content, which will result in a whole new*compression*loss.

3. Changing the quality factor within your application can result in generational loss across the entire image.

FWIW

[ETA: I love this stuff. If you don't you may want to skip the following. I thought it might be reasonable to indicate why a file might grow in size under*JPEG*even though it has shrunk in information content.JPEG*does the following: First, you convert the file to YUV (typically, YUV 4:2:2), which shrinks the pixel data to 8-bits per channel from the original (with typical RAW or TIFF data, the format would be 16 bits per channel, but the actual data will normally be no more than 12 or 14 bits per channel). Further, the two color channels use only one value of chrominance for every two values of luminance ("luminance" == grayscale, "chrominance" == color). So the data has been massively reduced just by the conversion to YUV.

Next, the data is grouped into 8x8 clumps of pixels called "GOPs" (groups of pixels) and converted using the Discrete Cosine Transform (DCT) to frequency space. Then the luminance GOPs are subjected to a relatively gentle low-pass filter that throws away some of the frequency data in the GOP, and the two chrominance channels are subjected to what is typically a very agressive low-pass filter. If you have ever*JPEG'ed an image with sharp colored edges (like, say, colored text in an image), you will almost always find the edges to become fuzzy due to the standard over-compression of the chrominance channels.

Finally, the step that does the actual*compression*is applied. This is Huffman coding of the results of the previous step. The Hufman code represents a table entry indicating a funny kind of run-length encoding. Basically, the contents of the Huffman code are indexes into the Huffman table. The Huffman table indicates a particular run of 0s and 1s. Imagine that the first entry in the Huffman table is a run of 200 0s. Then, the value "1" represents 200 zeros. The*compression*is a matter of how well the Huffman table captures really long runs of repeated patterns. The reason that we apply the quant tables should now be clear -- we are trying to get a lot of 0s in our data so the Huffman tables will be more effective at compressing the data. But it should also be clear why data may actually grow when it loses some information -- the position of the 0s in the resulting GOPs may have shifted, leaving a shorter run of bits and requiring more entries in the Huffman coding. This is exacerbated by those applications that use a default Huffman table instead of generating the tables from the actual data in the particular image.]

Last edited by tclune; 24th August 2012 at*10:39 AM.

[revi]

I never denied that re-encoding a jpeg several times causes quality loss, the more so after an edit.
My problem with the way you stated the case was that it went too far towards "TIFF is the only good format, and JPEG is rubbish",
and some parts were phrased in a way that could very well be misunderstood, notably anything concerning block size and changes to same,
and in a lesser degree the changes that happen during editing.

To take your points in order:
1 - How many use several different editors for their JPEG files? Add to that that there are not that many JPEG libraries around, meaning that most
systems will have only one or two installed. And I do hope that any decent editor uses a library for this kind of stuff, and didn't re-invent the
wheel... So even when using several programs to manipulate JPEGs, there's a good chance that the same encoder is used by all.

2 - I never contested that editing and saving caused quality loss, I'm just not convinced that it's visible for the first few saves.

3 - With the same encoder, that shouldn't be too bad: lowering quality discards more coefficients, and might get a bit more rounding error in the
remaining ones, but as you are lowering the quality, you will get a much larger quality loss from that. And increasing the quality setting means
that you are trying to recover data that was already lost...

From the information I have, TIFF isn't always the best format to use, and JPEG is very useful when employed properly.
You might have noticed that I actually suggest not using JPEG as an intermediate format.
Nor is it suitable for any kind of image like line drawings, screen shots and such, with a lot of sharp transitions or large areas of solid colour.

So, although in theory any re-encoding of a JPEG is bad, in practice you can get away with a few edit/save cycles. I'd guess you'll get
more damage from banding due to excessive editing (curves, or WB correction), or noise when trying to recover shadows.

----------------------------------------
Now I'll have to get technical, as well.
First off (to get that out of the way), the part that really causes image degradation is the DCT (Discrete Cosine Transform) with quantization.
The YUV↔RGB conversions can also cause some degradation, but this will be (mostly) independent of quality settings, and hopefully negligable.

The entropy and Hufman compressions are both lossless, so can be ignored when looking at image quality.

Now, the DCT is a form of Fourier transform. One of the characteristics of any Fourier transform is that it doesn't like any kind of discontinuity in the data (a.ka. sharp edges),
and shows what's called 'ringing' around any sharp edge. (That's what you see when storing any kind of line image as JPEG, and no need for colour, B/W shows it quite well.)
But, to completely describe our data after a DCT we would need as many coefficients as we have data points.

Well, the whole point of using the DCT in the first place is that we want to get rid of some of the coefficients, and we discard the high-frequency ones
(the least important ones, and otherwise the next entropy encoding step won't work well). That means we will introduce edges when we restore the image. So any re-encoding
will add to the degradation, and this error will actually not tend to zero.

So I guess we agree that repeated JPEG encoding is bad, we just disagree on how bad exactly.

Now, as to why the quality setting of 100% increases file size. That's mostly due to the fact we actually have to add data to the file to allow decoding.
There's three steps that affect JPEG compression:
- DCT transform and quantization (i.e. deciding which coefficients we keep)
- entropy encoding: this replaces the stretches of zero values we got after quantization by a count of the zero values
- Huffman encoding: replaces repeated stretches of bytes with a shorter code. Huffman encoding of image data typically doesn't work well
(a few % reduction in size), here it works as we have structured data after the entropy encoding. But it produces a table needed for decoding that needs
to be stored with the data

As there are different programs (or rather libraries) to decode JPEGs, the quantisation tables will need to be stored with the data.

If we use a 100% quality setting, we do no discard any DCT coefficients. Therefore, we don't have any repeated stretches of zeros for our entropy encoding,
and no reduction in data size. We actually might have to add data to indicate the that the odd zero value we find is a single zero, and/or to indicate the end of a
data block. Also, the Huffman encoding has less repetitive structure to work on (and can't reduce the data as much), but will still produce its decoding tables. (this
is not strictly true, as the standard provides a basic Huffman code table that doesn't need to be included in the final file, but implementations can create their own,
and those must be stored with the compressed data).

So, at a 100% quality, we can't reduce the volume of the interesting data, but we still have to add our bookkeeping information => the file after compression is larger.

[Windytoo]

I started this thread and am indebted to you all for the efforts made to clarify the issue. I have several books that just confuse by over simplification. To summarise what you have said:

A raw "picture" is jpeg preview, hence no visual difference. The software recording the raw file (in the camera) mosaics the three colour elements into a single digital code representing the colour of the pixel. o

If saving as a TIFF the software (algorithm?) estimates the three original colour element of each pixels, in effect it is more raw than "raw". I have viewed the Jpeg and Tiff images side by side down to 1600 magnification in PS. While there are the same number of pixels (obviously) you can clearly see that some pixels are blocked together in the jpeg. These are not regular quadlateral shapes but rather a grouping together. Even at x 1600 you might miss this miss if asked to comment on what was on the screen. The reduction of file size from raw to Jpeg is not I think archived by this grouping alone, I suspect the actual digital code for each pixel is also shortened by the algorithms in jpeg. Zooming out to a practical A 3 size, even an A 2 there is little or no difference. I have a 22 inch high resolution screen and use my wife to compare the images. On every occasion viewing tiff and jpeg side by side she has marginally preferred the Jpeg. On the generational issue I have concluded that if you use photo shop to save a jpeg at quality setting 10 you get away with a couple of edits, but there after you will get generational degradation. I am now going to start layering and will adopt the following strategy. Shot in raw and high quality jpeg (my camera does this). Save most shots as jpeg only any discard the raw. For my best shots I will save as Tiff, Raw, do my editing in Tiff save my final edit in jpeg at high quality,discard the tiff, archive the raw and keep the jpeg as my file to view. one final question, I assume that you cannot edit in raw as what you are seeing is a jpeg, and therefore editing in Tiff is best.

I never denied that re-encoding a jpeg several times causes quality loss.
My problem with the way you stated the case was that it went too far towards "TIFF is the only good format, and JPEG is rubbish",
and some parts were phrased in a way that could very well be misunderstood, notably anything concerning block size and changes to same,
and in a lesser degree the changes that happen during editing.

From the information I have, TIFF isn't always the best format to use, and JPEG is very useful when employed properly.
You might have noticed that I actually suggest not using JPEG as an intermediate format.

Now I'll have to get technical, as well.
First off (to get that out of the way), the part that really causes image degradation is the DCT (Discrete Cosine Transform) with quantization.
The YUV↔RGB conversions can also cause some degradation, but this will be (mostly) independent of quality settings, and hopefully negligable.
The entropy and Hufman compressions are both lossless, so can be ignored when looking at image quality.

Now, the DCT is a form of Fourier transform. One of the characteristics of any Fourier transform is that it doesn't like any kind of discontinuity in the data (a.ka. sharp edges),
and shows what's called 'ringing' around any sharp edge. (That's what you see when storing any kind of line image as JPEG, and no need for colour, B/W shows it quite well.)
But, to completely describe our data after a DCT we would need as many coefficients as we have data points.

Well, the whole point of using the DCT in the first place is that we want to get rid of some of the coefficients, and we discard the high-frequency ones
(the least important ones, and otherwise the next entropy encoding step won't work well). That means we will introduce edges when we restore the image. So any re-encoding
will add to the degradation, and this error will actually not tend to zero.

So I guess we agree that repeated JPEG encoding is bad, we just disagree on how bad exactly.

Now, as to why the quality setting of 100% increases file size. That's mostly due to the fact we actually have to add data to the file to allow decoding.
There's three steps that affect JPEG compression:
- DCT transform and quantization (i.e. deciding which coefficients we keep)
- entropy encoding: this replaces the stretches of zero values we got after quantization by a count of the zero values
- Huffman encoding: replaces repeated stretches of bytes with a shorter code. Huffman encoding of image data typically doesn't work well
(a few % reduction in size), here it works as we have structured data after the entropy encoding. But it produces a table needed for decoding that needs
to be stored with the data

As there are different programs (or rather libraries) to decode JPEGs, the quantisation tables will also need to be stored with the data.
So, if we use a 100% quality setting, we do no discard any DCT coefficients. Therefore, we don't have any repeated stretches of zeros for our entropy encoding,
and no reduction in data size. We actually might have to add data to indicate the that the odd zero value we find is a single zero. Also, the Huffman encoding has
no repetitive structure to work on (and can't reduce the data), but will still produce its decoding tables.

So, at a 100% quality, we can't reduce the volume of the interesting data, but we still have to add our bookkeeping information => the file after compression is larger.

[revi]

I started this thread and am indebted to you all for the efforts made to clarify the issue. I have several books that just confuse by over simplification. To summarise what you have said:

A raw "picture" is jpeg preview, hence no visual difference. The software recording the raw file (in the camera) mosaics the three colour elements into a single digital code representing the colour of the pixel. o

If saving as a TIFF the software (algorithm?) estimates the three original colour element of each pixels, in effect it is more raw than "raw". I have viewed the Jpeg and Tiff images side by side down to 1600 magnification in PS. While there are the same number of pixels (obviously) you can clearly see that some pixels are blocked together in the jpeg. These are not regular quadlateral shapes but rather a grouping together. Even at x 1600 you might miss this miss if asked to comment on what was on the screen. The reduction of file size from raw to Jpeg is not I think archived by this grouping alone, I suspect the actual digital code for each pixel is also shortened by the algorithms in jpeg. Zooming out to a practical A 3 size, even an A 2 there is little or no difference. I have a 22 inch high resolution screen and use my wife to compare the images. On every occasion viewing tiff and jpeg side by side she has marginally preferred the Jpeg. On the generational issue I have concluded that if you use photo shop to save a jpeg at quality setting 10 you get away with a couple of edits, but there after you will get generational degradation.

I am now going to start layering and will adopt the following strategy. Shot in raw and high quality jpeg (my camera does this). Save most shots as jpeg only any discard the raw. For my best shots I will save as Tiff, Raw, do my editing in Tiff save my final edit in jpeg at high quality,discard the tiff, archive the raw and keep the jpeg as my file to view. one final question, I assume that you cannot edit in raw as what you are seeing is a jpeg, and therefore editing in Tiff is best.

(I slightly modified the layout of your text in the quote)

You'll probably get more answers regarding your workflow, but my take is this.
- For any shot that is worth keeping, I always keep the RAW file (If I shoot RAW+JPEG, I'd rather delete the JPEG than the RAW in borderline cases).
- After decoding the RAW and doing the basic editing, I save as a PNG (my reasons to prefer PNG mentioned elsewhere). Any further edits will be done starting from this PNG.
- If I have to use layers, masking and such, I'll save the work in the native editor format.
- The final image when editing is done, will be stored as a PNG file.
- For display or printing, I'll generate a JPEG version as needed, and at that point the image will be downsized and resharpened as needed.
So I'll end up with several files for each image: the original RAW, an edited PNG, and one or more JPEGs as needed. For some, there will be an editor file as well (.xcf, as I use the GIMP).

When I started using RAW files, I got results that weren't as good as the camera-produced JPEGs. After a while, that changed and I now get better results than with in-camera JPEGs. There's quite a lot to learn to get the best out of a RAW. But as I kept them, I can take older pictures and rework them. And then there's changes in software quality and computing power: better routines for image treatment are being developed all the time, but some are too complex for the computers available at the time. So please, don't discard the RAW files before the JPEGs

Do keep in mind that the RAW file is the one that contains the most information about your image. Any JPEG (even or especially camera-produced ones) has already had a lot of treatment and lost a lot of the information that is still present in the RAW file. Any further editing can only lose more information (and that is independent of the file format).

And I keep the final image as a PNG, as you don't want the same conversion to JPEG in all cases: a version to be shown on screen will be at most 1900 pixels wide or 1000 high (that's about half or less of what my camera produces) and will need only a little bit of sharpening after downsizing. A version to be printed can be the full sized image, but in any case needs to be sharpened more aggressively. Sharpening increases the contrast at the edges in your image. As the pixels in print are a lot smaller, you'll need more edge contrast than on screen to get the same visual effect. An image sharpened for printing can be unpleasant to look at on screen, and a version sharpened for screen tends to give prints that are softer than you want.

[ajohnw]

It's amazing how often this crops up and still jpg is too lossy. Most applications will allow saving jpg's at 99% some 100%. As pointed out 100% is a large file - around the same as the raw file I would say from the few times I have done this.

In real terms the only time the loss matters above a compression quality level of around 85% is when the image is re edited again from the jpg. In some instances this is irrelevant when working from raw as some software will save the editing settings along with the jpg.

The other aspect is size reduction. Now that does loose data. Actually if people worked from 85% full sized jpg's and reduced and saved them and also saved the large version for any further work It would take several edits before anyone could see any difference in the final results. Save at 95% and there could be many many more. Dslr's type cameras appear to produce jpg's of this level of quality. Those however are never have a linear tonal range. Other types may or may not depending on how they are produced.

The only real problem with jpg's is bit depth. They are 24bit colour so can not be manipulated by as much as 48bit colour images. When working on 48bit images we are altering the way that the 48bit is reduced to 24. There is a surprising amount of compressed detail in camera jpg's however so this area may not be as straight forwards as it seems. The same can be true of any jpg.

The most efficient image encoding format in many ways is PNG. Despite that I am only aware of one instance of the use of it on here. It's lossless. I understand there are also lossless jpg standards and that loss in some of the standards is some what dependent on image size. Some may remember when files started using .jpeg rather than just ,jpg The term jpg can many a number of things as outlined here http://en.wikipedia.org/wiki/Jpeg As can the term tiff - outlined here http://en.wikipedia.org/wiki/Portable_Network_Graphics along with why there is little use of all of it's capabilities.

Next time this crops up most will still say it's lossy without any direct evidence of the amount or idea of suitable compression qualities. Basically they have just read it some where and miss the main problem in relationship to photography - bit depth - but even that depends on the source of the jpg and the nature of the shot. They also omit the problem with raw - bayer mask interpolation leading to jpg type artefacts at some level of magnification. Final reproduction size and it's dpi have more to do with the outcome really what ever format the file is in within reason.
-

[ajohnw]

(I slightly modified the layout of your text in the quote)

When I started using RAW files, I got results that weren't as good as the camera-produced JPEGs. After a while, that changed and I now get better results than with in-camera JPEGs. There's quite a lot to learn to get the best out of a RAW. But as I kept them, I can take older pictures and rework them. And then there's changes in software quality and computing power: better routines for image treatment are being developed all the time, but some are too complex for the computers available at the time. So please, don't discard the RAW files before the JPEGs

Do keep in mind that the RAW file is the one that contains the most information about your image. Any JPEG (even or especially camera-produced ones) has already had a lot of treatment and lost a lot of the information that is still present in the RAW file. Any further editing can only lose more information (and that is independent of the file format).
.

There are some cases where a camera jpg is likely to achieve better results than can easily be obtained from raw. Usually when more extreme iso ratings are used or when there are large areas of an image that are producing very little light while other areas are producing lots. These 2 tend to inter react. Max usable relatively noise free iso ratings vary according to the extremity of the lighting conditions. The reason a jpg is likely to be better is that the camera manufacturers are fully aware of the capabilities of their sensors and process images accordingly.

I had an interesting example of that recently. A moon shot I posted. As I use a telescope from time to time I do know what it should look like. From that I can tell that the tonal range is compressed - probably down to the lens in part - raw or jpg and that the shot is marginally over exposed - raw or jpg. The camera compresses highlights into the jpg. The jpg is virtually noise free in the black areas around the moon. Not so the raw. The noise is very apparent and rather difficult to remove without spoiling the moon detail as there are similar light levels in it. Leaves me wondering how they do it? I suspect it's an area related noise reduction technique that selects automatically and at various levels going on high iso shots at tonal ranges that are easier to cope with. It preserves detail too so must be some form of complex adaptive colour smoothing. A little different to what is normally provided in image processing software for noise removal.

-

[revi]

May I ask what type of noise reduction you tried on that? Switching to Lab for instance, then doing the noise reduction and switching back to RGB can get a much better result than doing the noise reduction in an RGB colour space. And then there are a number of algorithms available...

Also, if you can do noise reduction before demosaicing, you can get a better result than doing it after (as that step can 'spread out' the noise over several pixels, making noise removal more difficult).
And if they get really nasty, the in-camera JPEG uses black frame subtraction, which is not saved in the RAW file... (but you can do that afterwards).

I guess part of the better result with in-camera JPEGs is related to that, as a camera normally doesn't have the calculating power to use very complex (adaptive) algorithms. Just think of the time your PC spends on a noise reduction, and then compare that to the time a camera has available in burst mode.

[ajohnw]

I have tried several noise removal methods Revi and will try as many as I can find. Or any suggestions. I haven't tried hsv yet but the general problem is that any filtering technique removes detail from the moon as well. The noise in this case is a blue red mix with more blue and very evident from raw. The camera has a facility called pixel mapping which might help. It seems to tune things up sensor wise. Need to try that when the sky is reasonably clear again.

Working from raw involves developing a suitable tone curve. That can take a long time. I found that raw noise removal didn't help. The noise is too splogey. Playing with the tone curve and then using eye dropper style black level adjustment killed the noise. It's still there if the image is blown up but not apparent at 1:1. This is the best result I have had so far that way. Black point about 30 from memory. It's had slight sharpening and very slight tone mapping. Fair enough really as there was high mist in the sky. This one is a png.



This one is a straight crop from the camera jpg. Same treatment but a little more tone mapping. The maria are darker. Nothing to do with the tone mapping which was still very moderate. Brightness also adjusted to more or less match the other.



The difference is probably largely down to the tone curve used but no tone curve I could produce removed a significant amount of noise. I need to try again and this time try and keep the tone level in the maria more like or better than the jpg.

One aspect that is better on the raw version is the white rim. It's still there but less pronounced. This is probably down to jpg compression. The rim in the raw file will be down to demosaicing - the edge of the moon isn't smooth, 100% resolution views are seldom used so that aspect makes this problem more pronounced.

Using black point correction has also led to some colour in the raw moon in a few places. Absent in the jpg version,

It isn't wise to compare in camera processing speed with a pc. Unfortunately PC's need to spend a considerable amount of their processing power watching out for things that it needs to do next. It's a draw back with multi tasking software. A very high proportion of the processing power that is available is spent doing just that. A jpg engine needn't be doing anything else.

I have also tried dark frame subtraction on another image. Different subject matter but while it removes I would guess 1/2 of the noise from 10sec exposures it doesn't entirely clear it up. This is why asto people average many shots. The idea is that if it's truly noise it will average to zero. The dark frame just takes care of variations that are inherent in a particular sensor - varying degrees of so called hot pixels. All pixels have this to some degree. Exposed to zero light the apparent light level will slowly increase. Noise is a random effect that sits on top of that.

Should mention that the camera was set on intelligent noise reduction and all else zero including a standard tone curve. Hope it's not more intelligent than me. The Pen's are sometimes reckoned to have the best noise reduction available though.
-

[xpatUSA]

First, you convert the file to YUV

First time I've seen that . . I thought it was YCbCr?

[dje]

I started this thread and am indebted to you all for the efforts made to clarify the issue. I have several books that just confuse by over simplification. To summarise what you have said:

A raw "picture" is jpeg preview, hence no visual difference. The software recording the raw file (in the camera) mosaics the three colour elements into a single digital code representing the colour of the pixel. o

If saving as a TIFF the software (algorithm?) estimates the three original colour element of each pixels, in effect it is more raw than "raw".

Hi Windytoo (sorry to be so formal - I'd use your real name if I knew it !)

A camera sensor has a colour mosaic filter in front of it which restricts the light falling on any one pixel to one of the three primary color components. The RAW output from the sensor for that pixel therefore only represents one colour component and before a proper colour image can be viewed, de-mosaicing must occur (either in camera or in external sofftware) so that the other two colour values for each pixel can be estimated.

If you shoot JPEG, the de-mosaicing and other processing is all done in camera and a JPEG file is saved in camera. If you shoot RAW, then a lower resolution JPEG is still produced in camera but for camera preview only. The main RAW data is saved in a RAW file. A copy of the preview JPEG is also embedded in the RAW file.

If you shoot RAW + JPEG you get a combination of both of the above.

The RAW file has to be processed externally in RAW processing software such as Adobe Camera RAW, Canon DPP, Nixon NX2 etc etc etc. With this process, when the RAW file is opened up in the RAW processing software, the first thing that happens is that de-mosaicing and other basic processing tasks are done. At this stage, the file has not been converted to any other format. When you are finished editing in this software, you have the choice of saving the file in a variety of formats depending on the software used and a 16 bit TIFF or PSD are popular choices here. If you are using ACR and Photoshop, you don't even need to save it here but simply transfer it from ACR to PS.

The next stage is to do further editing in PS if necessary and then you face the choice of saving that work. I prefer to save this as a PSD file as it preserves all the layer information and can be opened up again later for further work (including re-adjustment of layers) without any degradation due to jpeg conversions.

Just as an aside, there are some software apps around that "handle" RAW files but in reality all they do is extract the embedded jpeg preview file. This is not much use for serious editing, only a quick preview.

Hope this clarifies things a bit.

Dave

[John Morton]

I started this thread and am indebted to you all for the efforts made to clarify the issue. I have several books that just confuse by over simplification. To summarize what you have said:

A raw "picture" is jpeg preview, hence no visual difference. The software recording the raw file (in the camera) mosaics the three colour elements into a single digital code representing the colour of the pixel. o

If saving as a TIFF the software (algorithm?) estimates the three original colour element of each pixels, in effect it is more raw than "raw".

Save most shots as jpeg only any discard the raw. For my best shots I will save as Tiff, Raw, do my editing in Tiff save my final edit in jpeg at high quality,discard the tiff, archive the raw and keep the jpeg as my file to view. one final question, I assume that you cannot edit in raw as what you are seeing is a jpeg, and therefore editing in Tiff is best.

I think that the main issue you face here is one which has not been addressed yet: simply, as time goes on, you will become a much better photographer and much better at post processing your images. JPEGs do degrade in the course of edit/save cycles (this we all seem able to agree upon); so, as you become better at image editing, you don't want to be dealing with favorite images that are getting worse as you go along.

My strategy: keep RAW files as a archive; do any re-edits starting from the original RAW files. Use TIFFS for editing; save finalized versions as the source material for producing JPEGs that are required for an immediate use. I might need a full sized JPEG if I am taking a file to a commercial, consumer-grade printer; I might need a smaller one for display on the Internet; I might want to use a smaller one yet for e-mailing.

You can edit in RAW and indeed usually should to some extent; but the edits are saved as a separate file and do not change the RAW file itself. When the file is converted to another format, and taken out of the RAW converter into another program for editing, then the edits to the RAW file are finalized into the converted file (but not for the RAW file itself). You can always go back to your RAW file and open it again in the converter, then change the white balance or any other of a number of image parameters.

The RAW file is just what comes out of the camera's sensor. Each sensor well gives a greyscale value, but each sensor well also has a color filter over it (either Red, Green, or Blue) so it is assigned that color. Then, complex algorithms determine the RGB mixture of values that defines color of each pixel.

If you open a RAW file to edit it, then it must go through a RAW converter. If you open a JPEG, PNG, or TIFF file then it is ready to be edited right there and then; but those edits are not reversible per say once the edited file is saved (unless you edit non-destructively, using layers and leaving the base image intact and unchanged). I use "Save As" a lot when editing, and will number/letter my edits so that I don't have to go all the way back to square one if I decide at the end that I should have done something differently a few steps back in my editing process.

Memory for storage is pretty cheap, so I don't mind saving a number of my editing steps as TIFFs; which is one reason I really don't care if uncompressed TIFFs are larger than some other format.

In the end, I have ALL of my RAW files archived and various stages of edits for my current favorites saved as TIFFs. Of my best edits, a number will go on to become various sizes of JPEGs for one specific use or another.

I don't delete RAW files because after decades of film photography I eventually understood that I would always wish I had taken more photos at the time when I was here or there or anywhere; and that some of the images I didn't think were that great at first turned out to be better than I initially thought once I finally got around to editing them. In addition, other people have often preferred images I would have abandoned over the ones I preferred from any given photo shoot: there is no accounting for taste, so, I think that it doesn't hurt to make allowances for other people's preferences by keeping all the RAW images from any given photo session.

[ajohnw]

I've just started with digital photos. I'm trying ti get the whole pixel, resolution, image size thing into my head. I take a raw photo at 25mb, then save as a tif which auto saves at 42mb, I save as jpeg at 2.8mb. When comparing even at 1200 zoom I see no real difference. Properties shows no pixel, dic size, or resolution changes. Why is this?

Just elaborating something I have already mentioned but sticking to just that this time.

The usual sensor only records light levels not colour so there is a colour filter in front of each pixel. Usually 2 green and one each of red and blue. This is the information contained in a raw file. The black and white image directly from the sensor and information on how the pixels colours are laid out. The image is generally stored in a tiff like format but the colour isn't. The important aspects to realise about these is the depth of colour they record. Compacts may use 10bit and only output jpg's, dslr's 12 or 14bit colour for each of the colours. 12bit for instance can record 4096 intensity steps of any of the colours that the sensor records through it's colour filters. Sounds a lot but when viewed as we see light level changes it isn't as much as some may think and 14bit just adds one more stop.

When a raw file is developed it's converted into a bit mapped image and shown on a PC screen. An interpolation routine is used to do this as a block of 4 pixels are used to record the actual colour of the detail in the scene. This process is generally called demosaicing. A simple way of looking at this is that each "pixel" on the screen has 3 colour channels determining it's brightness and colour. In real terms the screen uses a similar idea to the sensor. There are 3 colours of pixel that are so close together we can't see them. The PC screen generally can only handle 8bits per colour - 256 light level changes in each of red green and blue so the software has to do something else as well. Get the bit depth recorded by the camera into the PC screens colour space. This may fit so there isn't a problem. If it doesn't fit, say the dark areas are too dark and lack detail the image can be brightened up in the dark areas drawing the detail in the larger colour space into the smaller one. That's a very simplistic way of explaining this. The same sort of thing can be done with bright areas as well. Things called tone curves are use as well. Part of that is due to the fact that we see to a power law but sensors are linear. In rough terms that means that we see light levels changes related by factors of 2 in terms of equal brightness changes. 1 2 4 8 etc will be roughly seen as equal brightness changes.

Once the picture is adjusted this way it's in 3x8bit colour space. It can be saved as a jpg with various quality levels right up to 100%. Lower quality levels tend to blur detail - very low quality levels. If that picture is edited again there is no out of range information to bring into the colour space. If your happy with the shot there really isn't much point in keeping it in any other format other than maybe png. If your not happy with the shot it's best to go back to the original raw file and process it again in a different fashion. No point in doing the same processing again as the results will be the same. If saved as an 8bit / colour png there is no loss and the file can be edited over and over again without any problem but with no extra colour space to bring into the reduced colour space - not a problem for lots of post processing options. If jpgs are saved at say an 85% quality level then 3 edits reduce this to 0.85x0.85x0.85=61%. On the other hand 95% would finish up with 85%. 99% at 97%. Png would finish up unchanged as would 100% jpgs. Generally more information is lost when shots are reduced to typical web sizes so saving the full sized jpg at a decent quality level is a viable option for web use.

The raw file can be converted and saved as a tiff. It will be a lot bigger as tiff is usually implemented because each pixel will have colour information rather than a black and white image with separate information on it's colour mask details as raw is stored. If your software supported tiff/ep that wouldn't be the case. It probably isn't the case when saved as adobe dng either. The ancient reason for saving in large tiff files was the worry that software that decoded specific raw files would disappear over time as manufacturers bought out different standards, eg Canon cr variations. This has never happened but might. If all manufacturers adopted something like tiff/ep this would be unlikely to happen but still might. The other option is to save in 16 bit png format but you may find that your software only supports 8 bit png. Png is compressed and is lossless. It too might disappear as might the usual tiff. There are numerous image storage standards. Probably more than people realise. Even jpg's can be fundamentally different. Some variations are available for conversion in a program called Converseen. More likely the format and media the information is stored in will be superseded. Ordinary people worry about this sort of thing as they want their descendants to be able to look at the shots. Museum people and those that collect evidence worry about this sort of thing a lot and realise that there really isn't a safe option. They have to hope that conversion software will be available if and when it's needed.

When you compare your jpg's and tiff files it's hardly surprising that you can't see any difference. They are both being shown in the same colour space and have been processed in the same way. One way of looking at that is that the larger colour space in the tiff has been distorted so that it looks good when truncated into the reduced colour space that the screen has. It's worse than that really in some ways. The software used to process the shot probably works in an entirely different colour space and many can even store the information in an entirely different manner.

Hope this helps.

As to raw handling programs that only deal with the embedded tiny jpg pass I don't know of any and also believe that not all raw formats actually include a jpg. The camera has the ability to decode raw and display as a jpg. Storing it is a separate option. Maybe some image browsers use an embedded jpg. Pass. They don't appear to. Some packages even generate their own thumbnails for quick browsing. That infuriates me as I use several different packages. It make more sense to a have a single comprehensive archiving and retrieving package.

Part through work and the software doesn't store the processing steps and you want to stop? - use the pc's resume feature if you software doesn't allow saving an exact copy of your current processing steps.

No time to check for typo's as strange as they may be.

Add one factor mentioned that is rather silly. No editing program changes the original file unless the user is silly enough to save to the same file other than when they intend to. Most even warn if an attempt is made to write to the original file.
-

[ajohnw]

Perhaps the answer to all of this is for people to put their foibles to one side for a moment, develop there raw files as usual and save a tiff and a jpg, several jpg's at different quality levels and see for themselves when there is a marked difference between the tiff and the jpg. As mentioned in the 1st post it will be rather hard to see any difference at some levels of quality and much depends on the fineness of detail in the shot anyway.

Then reduced both to typical large web size - maybe a 1/4 size or smaller depending on the pixel count in the camera. This may loose detail in some shots but again the tiff and jpg results can be compared. Reductions from different quality level jpg's can also be compared with reductions from the tiff.

Then try the same with reductions to email size.

Each reduction will need a degree of sharpening from tiff or jpg, lessening with increasing reductions. A full size camera jpg can take a lot more. From raw the shot will have generally been sharpened as part of the development process so tiff and jpg start from the same footing.

One of the aspects of all of this that in some ways amuses me is the fact that I still have my 6mp canon 300d/rebel. In terms of what is usually displayed in this forum that pixel count is more than adequate. Those 6mp are over sampling by at least a factor of 4 in those terms. A lot more in many cases. The only way to show more detail than it can capture is via a much bigger image.
-

[xpatUSA]

. . . Each reduction will need a degree of sharpening from tiff or jpg, lessening with increasing reductions. A full size camera jpg can take a lot more. From raw the shot will have generally been sharpened as part of the development process so tiff and jpg start from the same footing.

John, is that sharpening done after re-sizing? Bart van der Wolf has a good article about down-sizing here which makes a compelling case, with examples, for a slightGaussian smoothing before re-sampling with no sharpening during down-sizing, only after.

One of the aspects of all of this that in some ways amuses me is the fact that I still have my 6mp canon 300d/rebel. In terms of what is usually displayed in this forum that pixel count is more than adequate.

Fully agree with that. My watch shots for the web are usually taken at 1134x756px (0.8MP, SD10 in LO res which is binned in the Foveon sensor) and just cropped to about 800px wide for posting, or 512-640px for eBay.

[ajohnw]

John, is that sharpening done after re-sizing? Bart van der Wolf has a good article about down-sizing here which makes a compelling case, with examples, for a slightGaussian smoothing before re-sampling with no sharpening during down-sizing, only after.
.

I've been away in Wales again.

Interesting link Ted. I've mentioned minimal sharpening AFTER downsizing several time. Looks like that may need qualifying on the basis of what package is used to do the reduction. I noticed that images could be improved that way a long time ago using the Gimp. That defaults to cubic as per the imagemagik examples and offers none, linear or Sinc as alternatives. Sinc is also named as Lanczos3. I'm sure from the names that some of the imagemagik examples are using Fourier filtering techniques. They are loosely refered to as windowing techniques and improve the performance of Fourier filters. There is some info on that here and elsewhere on the web. http://www.fmwconcepts.com/imagemagi...s/fourier.html

I mostly use Fotoxx now and assume that as the results are the same as I have achieved with the Gimp it uses Cubic too. I don't set any sharpening in the camera and find that full sized images can usually take a lot of sharpening without problem. In Gimp I have usually used the default amount of 0.5 but have found that is often too much on browser sized images but I didn't notice until I improved my monitor. Fotoxx handles amounts from 0 to 200 and defaults to 100 which is often ok on full sized images. Reduced images range from 15 to 40 or so. Image size seems to be the limiting factor. I have to hope my monitor is ok now or at least good enough but clearly the look of a shot will depend on the monitor used to view it. I suspect that the actual shape of the gamma curve has a lot to do with that.
-