Limits to photoshop's generative fill

[DanK]

Recently, my brother tried using Photoshop's generative fill to expand an image of some barren mountains in CA into a panorama. It did a superb job. My brother's suspicion, however, is that because the area is frequently photographed and shares many characteristics with other frequently photographed places, it was good match to the training set.

So, I tried an image that is less likely to match things in the training set, and generative fill failed miserably.

Here is the original shot. I took this with a drone last summer near Williamstown MA.



I first tried generative fill on the right hand side, but forest is an easy subject, it didn't fail badly at first glance, despite a few weird sections. So instead I first cropped a bit from the right, yielding this:



Then I did generative fill, just once:



What it produced was blurry, which fits with one thing Tim Grey wrote: presently, it can handle only low resolutions, and I fed it a relatively high-resolution image. That’s hardly the biggest problem, however. Take a look at the following, which is a crop of just part of the filled area. It’s full size if you click on the link.



The biggest problem is that the buildings look like they were painted by someone on powerful psychedelics. Some are merely distorted, but some appear to be a hodge-podge of elements from different buildings. The road is garbled and ends in a weird way, and if you follow it to the bottom end, you will see a bunch of random odd shapes.

I tried one last thing. I created the image above using the standard technique: expand the canvas and then select the expansion and a very small slice of the extant image. This time, I selected more of the extant image, including the road and some of the buildings. The result was different but equally bad:



Dan

[LenR]

Dan, interesting experiment - thanks for sharing.

[Manfred M]

Thanks Dan - Some interesting results here.

Your testing lines up with what I was finding, so the last time I tried something using Adobe Firefly, I approached things a bit differently, more like I would if I were doing a photo-composite along the lines of a sky replacement and then a fair bit of work in Photoshop to clean up the newly introduced element.

I did not like the background in the original, as it had a lot of "stuff" showing through the background. This is the image I started with.




I then used Photoshop's sky "Select Sky" functionality and had Firefly produce a new overcast, soft sky. Unlike the regular sky replacement functionality, Firefly built some new grasses that extended into the sky, giving a more "natural" appearance to the scene. I tried running the algorithm a few different ways and sometimes the grasses looked all wrong, but this one here was not bad. The new section was set to 75% opacity so that the original background still influenced the overall scene and some of the "distracting elements" were at the edge of being noticeable.




It is still early days and the software is still in Beta, but I do see some interesting, if limited potential here, if one understands the strengths and weaknesses of the product. It should be interesting once they roll out the full production version.

Opening the image in Lightbox and toggling between the two images shows how this turned out and what the new material did for the image.

[Cantab]

I've realized that the generative fill might help with a crease in a damaged B&W image from 1941. I have the latest version 24.6 of PS and pressed the links to download the beta generative fill software. But I just end up on a creative cloud webpage with no access to downloading the beta software.
Any comments would be very helpful.

[Manfred M]

I've realized that the generative fill might help with a crease in a damaged B&W image from 1941. I have the latest version 24.6 of PS and pressed the links to download the beta generative fill software. But I just end up on a creative cloud webpage with no access to downloading the beta software.
Any comments would be very helpful.

I seem to remember needing to use the Creative Cloud desktop to download it (it's part of the Photoshop Beta).

[Cantab]

I've realized that the generative fill might help with a crease in a damaged B&W image from 1941. I have the latest version 24.6 of PS and pressed the links to download the beta generative fill software. But I just end up on a creative cloud webpage with no access to downloading the beta software.
Any comments would be very helpful.

Edit:
A bit later: I found the non-obvious link on the creative cloud webpage; it's now downloading.

[Cantab]

Success. I did a test and was pleased with the results. The problem was reflections in my digital photograph, caused by a crease in the original photograph. The problem areas were all in an area occupied by a leafless tree during winter. I selected the offending areas one by one, gave no written instructions and simply let the AI do whatever it thought best. It gave me three options to choose from for each of the four or five times I selected an area. In each case, there was an option which solved the problem.

This is one of a good number of old photographs which I'm including in a short book about my father's overseas service in the Canadian army during WW2. The text of the book is finished. Now I'm working through the photos, creating final versions which will then go to the book designer.

[DanK]

Bruce,

I did a few more tests. Some were garbage, but others yielded results that were OK, if not perfect. I think the issue is what the material is that it has to generate. I extended some trees and reflections that came out ok. That's material that is common (hence probably abundant in the training set), fairly simple, and forgiving, in that trees can take many shapes, etc. The failure I posted was complex and not at all forgiving--it's immediately obvious when a barn has the wrong shape.

I suspect my use of this will be limited and will probably be more for removing things than adding them.

Dan