John, I can only give you a view under English Law. There are similarities between English law and US law though, so much so that when I was writing contracts with US companies, jurisdiction under the laws of NY, NY were an acceptable fall back.
What we are talking about is intellectual property rights of which copyright is just one. There is such a thing as a "User Right" as in if you take a photograph, you automatically own the copyright but you can grant a user right to a second party. That can be open ended or have conditions attached to it e.g. they can use it for personal use but not for commercial purposes. You can of course sign away your rights under contract for a consideration.
The same principle can be applied to the right to exploit the commercial potential of say a landmark building. You may well be able to photograph the building and you will the owner of the copyright associated with the image itself but you may not be free to exploit the image for commercial gain because that right belongs to the owners of the property. Hence the need for a property release. Equally however, they can't use your photograph because you own the copyright.
There are (at least) two other considerations though. In making your image, you might be breaching someone's privacy. Under English law they would have to sue you in a civil court for that if they wanted redress. In France though as an example, I think you might be breaking an actual privacy law. I'm not sure what the situation is in the US. The second consideration arises from the fact that most countries have statute laws preventing you from actually taking photographs (not just exploiting them) of sensitive subjects. These include the obvious like the military, it's equipment and premises but in the UK the Official Secrets Act also covers things like nuclear power stations and the premises of defence contractors.
It's a mine field as I said before but now you know why lawyers live in big houses. Hope this helps.