Not just a matrix of numbers. The control the output of your monitor, printer. They control the devices that make the image visible.
I did post this diagram several times and will do it again. Until now now response.
Everything you're doing is done on that raster image. The files you mentioned are disk files, containing the raster image in some way for some purpose. If you want to do pixel peeping you have to find a disk file that saves the raster image without modification. As far as I know that's the TIFF, either 8 or 16 bit.
George