Adobe RAW "sidecar" file format hacked

[xpatUSA]

Being a bit of a webmaster, I wondered if these files, of type .xmp, were by any chance like XML files which meant that they could be opened with a text editor. Then, indeed, I successfully opened one up in NotePad. The content was all there but hard to read, being just one long line. However if you open it up in a HTML editor, all is revealed in neat lines complete with line numbers.

There you see EXIF data for your shot and all the ACR adjustments made to your work of art - including adjustments that you may not even have in your current version of ACR.

A real programmer could then write something that lists all the pictures taken with a "D50" AND a camera profile of "red sunlight" AND an aperture of "28/10" . . you know, that kind of SQL stuff. That is, if they were so inclined and bored with taking the oft-mentioned "real photographs"

Ted

[Momo]

Alternatively, you could just use Lightroom.

[xpatUSA]

Alternatively, you could just use Lightroom.

Indeed. Or just select the "info" menu in PSE, since I don't have Lightroom :-)

One of the perils of posting or asking something technical, using long sentences and real paragraphs, is that there's always someone ready and waiting with a one-liner like "buy a new camera" or something like that.

The point was only that the mysterious "sidecar" file is readable without too much trouble.

Not "this is the easiest way to read EXIF stuff" LOL.

Ted

[benm]

If you are interested in finding sets of photos that meet certain criteria (such as your example of camera model AND profile AND aperture) you can already do this with a photo organizer. No programming needed. Photoshop Elements 6, which is what I am currently using and is now an elderly program, can sort on 30 different metadata tags. I am sure newer versions can do more.

[Momo]

One of the perils of posting or asking something technical, using long sentences and real paragraphs, is that there's always someone ready and waiting with a one-liner like "buy a new camera" or something like that.

Duly noted. Perhaps you can better enlighten me as to the nature of your "hack" while opening a plain text file inside of Notepad. What is the hack? What was hacked? When I read a header like, "Adobe RAW 'sidecar' file format hacked", I expect to read something along the lines of, "Anonymous partners with North Korea and China to exploit sidecar files and take over the world".

This is somewhat anti-climactic. I was so looking forward to scanning my HDD for rogue .xmp files.

There. That's more than one line.

[Colin Southern]

The point was only that the mysterious "sidecar" file is readable without too much trouble.

Hi Ted,

There never has been anything "mysterious" about sidecar files. As the filetype specifies, they're simply XMP (Extensible Metadata Platform).

The story behind them is that basically Adobe don't want to take the risk that a manufacturer might change the format of their RAW files and that any subsequent metadata writes to that file may result in corruption if the Adobe product wasn't aware of the format change ... so they write these changes to a separate file for safety. If you do as I do and convert all files to Adobe's open DNG format (using their free converter) then changes are written directly to the DNG file (because it's a standardised format that they understand).

Personally, I don't like sidecar files - (a) I find them messy, and (b) I find them dangerous in that if they EVER get disassociated from the RAW images then you've just lost ALL of your edits to those RAW images (and that's the point where 80% of my work gets done). So for me, it's DNG all the way.

[Photon Hacker]

Duly noted. Perhaps you can better enlighten me as to the nature of your "hack" while opening a plain text file inside of Notepad. What is the hack? What was hacked? When I read a header like, "Adobe RAW 'sidecar' file format hacked", I expect to read something along the lines of, "Anonymous partners with North Korea and China to exploit sidecar files and take over the world".

This is somewhat anti-climactic. I was so looking forward to scanning my HDD for rogue .xmp files.

There. That's more than one line.

You also misunderstood the meaning of a hack. A hack is something ingenious. Sometimes a workaround or unexpected solution but not necessarily. See http://stallman.org/articles/on-hacking.html.

[Colin Southern]

A hack is something ingenious.

So I'm deducing that a "Photon Hacker" is someone who does something ingenious with photons?

[Momo]

You also misunderstood the meaning of a hack.

Uh, in a word, no. I'm still waiting for the hack.

@Colin, I too went with DNG for a couple of years and loved the format. However, I recently bought a Nikon and found that converting to DNG, either through Lightroom or Adobe's DNG converter, stripped away some useful EXIF data. I thought it was supposed to be some sort of RAW wrapper and not muddle the original files? I'm still open to using DNG if it doesn't strip anything away.

[Colin Southern]

I too went with DNG for a couple of years and loved the format. However, I recently bought a Nikon and found that converting to DNG, either through Lightroom or Adobe's DNG converter, stripped away some useful EXIF data. I thought it was supposed to be some sort of RAW wrapper and not muddle the original files? I'm still open to using DNG if it doesn't strip anything away.

I wasn't aware of anything getting stripped out - any links you can point me to? The original v1.1.0.0 used to strip out black masked pixels, but that was fixed in v1.1.1.0. Certainly all current formats support the inclusion of encrypted fields for "secret sauce" type data.

[OldFrank]

Being a bit of a webmaster, I wondered if these files, of type .xmp, were by any chance like XML files which meant that they could be opened with a text editor. Then, indeed, I successfully opened one up in NotePad. The content was all there but hard to read, being just one long line. However if you open it up in a HTML editor, all is revealed in neat lines complete with line numbers.

There you see EXIF data for your shot and all the ACR adjustments made to your work of art - including adjustments that you may not even have in your current version of ACR.

A real programmer could then write something that lists all the pictures taken with a "D50" AND a camera profile of "red sunlight" AND an aperture of "28/10" . . you know, that kind of SQL stuff. That is, if they were so inclined and bored with taking the oft-mentioned "real photographs"

Ted

Being a "real programmer" I can see some use for this information whether it is a hack or not. I would go so far as to modifying and writing information back into the file. Like any such antics there are risks, but that's life.

So thanks!

[enaiman]

I can see an advantage for batch processing if there is a need to apply the same settings to a number of images.
Other than that I can't think of anything else.

You don't need to be a "real programmer" - a simple VB script would do the job.

[Colin Southern]

I can see an advantage for batch processing if there is a need to apply the same settings to a number of images.

ACR, Bridge, and Photoshop all do that now anyway.

[shreds]

Nikon written files are slightly different to Adobe ones and some of the Nikon data gets lost when opening in CS. Try it and see. The basic file remains unaltered, but the additional info is only readily readable in NX2.

(Imagine Nikon putting a Nikon only adjustment layer around alterations made in NX2).

[xpatUSA]

My apologies to All for posting something trivial and then getting tetchy about the subsequent leg-pulling!

I'll get my coat . . .

Ted

[Momo]

My apologies to All for posting something trivial and then getting tetchy about the subsequent leg-pulling!

I'll get my coat . . .

Ted

Oh, c'mon now. I have yet to start in on the other leg. I should apologize as well for my Inspector Clouseau investigation into what seemed at the time like a galactic takeover of all sidecar files.

[Colin Southern]

Nikon written files are slightly different to Adobe ones and some of the Nikon data gets lost when opening in CS. Try it and see. The basic file remains unaltered, but the additional info is only readily readable in NX2.

Opening in CS and having the data written into a DNG file are different things though; the DNG format supports all data, but that doesn't mean CS will use it (good examples being Canon picture styles and Nikons D-lighting where I wouldn't want that kind of thing applied to the image automatically).

[jasustar]

Since when is opening an XMP file with a text editor considered hacking? Until something detrimental is done to the XMP file, I don't considering that hacking.done,

[Inkanyezi]

Since when is opening an XMP file with a text editor considered hacking? Until something detrimental is done to the XMP file, I don't considering that hacking.done,

You might just have your pants in a knot about the term "hacking". A hack is not generally supposed to be detrimental, but I agree, that doing something useful might be considered a hack, while just opening a file to look at it is not much of hacking, although you might open it in order to hack it.

See: http://www.catb.org/jargon/html/meaning-of-hack.html

[Colin Southern]

Since when is opening an XMP file with a text editor considered hacking? Until something detrimental is done to the XMP file, I don't considering that hacking.done,

as noted by Momo 6 months ago ...