Adobe and AI - where is it processed

[Codebreaker]

Having spent a lifetime in high tech, albeit in the Defence and Aerospace industry, I'm a little cautious about where my data goes.

My question then is to do with where all these 'wonderful' features that AI can bring to Photoshop are actually processed. I'm assuming its not on my machine which has limited brain power, so it's probably being passed over to Adobe to mess around with before giving it back to me.

If this is the case I'm pretty sure that it's not been made clear to the end user - unless they can employ a lawyer to read through the small print.

So, where is the processing done and can I be sure my images remain mine? Which leads to the next question - do these features work off-line? (I guess I could try this for myself)

[DanK]

I don't know for a fact, but I think it almost has to be that the software is not using AI in processing your photos. It's had to imagine harnessing that much power and producing results that quickly, even if the processing were done remotely. I think what software vendors are saying is that they used machine learning to develop the algorithms that they then packaged into the software. In a recent video, Jeff Shewe, whose history with Adobe goes back many years, said that Adobe used a large number of his images, among others, as the training set for machine learning in developing the "auto" tonality algorithm in Lightroom.

If I'm right about this--I have an appt this morning and can't try using one of the "AI" features while off the net until later--then at least some of the manufacturers who tout their use of AI are misleading people, either intentionally or not. For example, Luminar bills its software as "The first image editor fully powered by artificial intelligence." That gives the impression that the software is using AI to perform your edits, not that machine learning was used to develop its algorithms.

anyone know for certain? Colin, if you beat me to the punch and try one of the AI functions while off the net, please post your results.

[Manfred M]

I had a small experimental neural network (9 - 16 neurons) running on my home computer close to 30 years ago.

The way these systems work is that they try to emulate the way the human brain works by introducing the same data and outcome information into the software and running it. The second part is a test set where a similar input is used and the computer is required to produce an output based on the input data. Once the system is capable of creating an appropriate output from the test set, it is considered "trained" and can be used for real world input. In general, the larger the training set, the more accurate the final output will be.

If we get an expert to identify and score the quality of an image on a scale of 1 to 10 and put together a set of 5,000 image in total. We show each image to a test subject who goes through them, with their scores. We the same expert to score 100 images that we then show our test subject and based on the observations of the test set, we ask the subject to assign a score to each of these images. We then compare the score the trainee gives to an image and compares it to what the expert has done. If the trainee scores closely enough to the expert, then the training is complete and we let the person score other images with a level of confidence.

If the scores from the trainee are not good enough, we get the trainee to go through the training set / test set analysis again (and again) until their scoring meets the standard that has been set. If we do not see the anticipated level of improvement, we might enlarge the training set with more examples or may see if the trainee has trouble with a particular subset of the images and introduce more into the training set. As a example, if the trainee makes significant errors when looking at abstract images, we will bias the improved training set with more abstracts.

In machine learning, we replace the person with a computer algorithm that emulates the human brain using interconnected, artificial neurons. The training set will have to be digitized images that the computer can interpret.

Once the training data resides in the software, we no longer need the training front end; just the finalized network that can statistically generate appropriate results. I know Microsoft has written an API for windows computers that allow software developers to load the trained datasets onto local computers to ingest and output data.

I seem to remember hearing / reading (probably at the recent Adobe Max event) that in some cases image data is uploaded to their data processing hardware via the cloud to implement some of their Sensei AI system for analysis and output.

[DanK]

I find the adobe documentation unclear. They appear to sell access to their AI system, but I suspect that's quite different from what we mere mortals get packaged with photoshop. I'd be willing to wager that what we have is access to the finished product, that is, the model obtained by training the machine on many images. I noted that Adobe refers to the new features as "Sense [AI]-based". Whether applying that model is so complex that they do it on their machines rather than on our own is something we could test out. I currently can't get the new version of photoshop to download where I am now, so I'll have to wait to try.

Sky replacement isn't the only AI-based feature. Content-aware fill is as well, and the new "neural" filters.

[DanK]

I got home and could check. Photoshop sky replacement works fine when disconnected from the internet. So as I suspected, the processing is local, on your own computer. It's not doing machine learning on the images. the changes were nearly instantaneous.

[Codebreaker]

Thanks for your replies.

It would seem that there's a mixture of local and cloud based processing depending on what function you choose. When you select a filter it will tell you if you need to download it first - to run locally - or if its cloud based processed.

This is from the Adobe FAQs:-


Do I need to download the filters locally or do some filters work in the cloud?

Most of the filters work locally, so you need to download them before you can use them. Depth-Aware Haze is currently the only filter that works exclusively in the cloud.

Some filters, such as Smart Portrait, process some operations in the cloud. Internet connectivity is required to fully benefit from from the filters.


Is Adobe collecting my information? What type of information?

Adobe collects anonymous behavioral data (clicks, likes, slider drags)
Adobe collects images anonymously when you include them in your feedback, and will not associate these images with a person or user
Adobe collects data for the purpose of optimizing the machine learning models to generate better results

How is Adobe handling inclusivity and biases?

Since the gallery is mostly working with portraits and since most of the filters are based on very new technology, possible questionable results with respect to inclusivity and biases can occur. If you see any such results, please let us know using the feedback mechanism.

We are constantly working on improving the technology behind the filters to reduce the number of possible questionable results to zero, and have a process in place to handle these types of experiences in real time, should they occur.

[DanK]

Thanks. This is very helpful.

I'm curious what they are actually doing on their servers. I'd be willing to wager that it is just heavier-duty processing applying already-determined algorithms, not actual machine learning, and that the machine learning was done earlier, on training sets.

As I understand it--and I am certainly no expert--machine learning is fundamentally different from the rater training that Manfred mentioned. In the case of rater training, at least the rater training I have done, the trainer isn't trying to learn what underlies a rating. They are just trying to get raters to come close to matching a criterion. In fact, a standard part of rater training is telling the raters what they should attend to and how they should evaluate it. In the case of machine learning, this is reversed: the humans are using machines to find patterns that underlie positive ratings so that these patterns can be applied to new data. It's an inductive process based on the training sets.

Similar approaches have been used in statistics for many decades. For example, stepwise and other best subsets regressions use empirical fit to determine which predictors are useful and how they should be weighted. Like machine learning, this is subject to chance, so the results are cross-validated, as in Manfred's post. In one of the early efforts to get computers to rate essays, statistical analysis was used to find linguistic patterns that happened to predict human raters' scores. Those factors had nothing to do with actual meaning, but once the model was created, after years of work, the resulting ratings correlated with human ratings roughly as well as the ratings of two humans.

[Manfred M]

Thanks. This is very helpful.

I'm curious what they are actually doing on their servers. I'd be willing to wager that it is just heavier-duty processing applying already-determined algorithms, not actual machine learning, and that the machine learning was done earlier, on training sets.

As I understand it--and I am certainly no expert--machine learning is fundamentally different from the rater training that Manfred mentioned. In the case of rater training, at least the rater training I have done, the trainer isn't trying to learn what underlies a rating. They are just trying to get raters to come close to matching a criterion. In fact, a standard part of rater training is telling the raters what they should attend to and how they should evaluate it. In the case of machine learning, this is reversed: the humans are using machines to find patterns that underlie positive ratings so that these patterns can be applied to new data. It's an inductive process based on the training sets.

Similar approaches have been used in statistics for many decades. For example, stepwise and other best subsets regressions use empirical fit to determine which predictors are useful and how they should be weighted. Like machine learning, this is subject to chance, so the results are cross-validated, as in Manfred's post. In one of the early efforts to get computers to rate essays, statistical analysis was used to find linguistic patterns that happened to predict human raters' scores. Those factors had nothing to do with actual meaning, but once the model was created, after years of work, the resulting ratings correlated with human ratings roughly as well as the ratings of two humans.

Based on the speed of response when I use the Smart Portrait filter I suspect that we are seeing a combination of cloud and local processing. The "age" filter seems to do the most re-processing of the image. I have a 15 Mbps (1.88 MBps) internet upload connection and 150 Mbps download. It seems to take in the order of 10 - 20 seconds to make changes (which involve upload and download). Based on my nominal upload speeds, it would take in the order of 25 seconds to upload the file (assuming we are looking at some 36 - 40MB of data to upload).

I know this is not a conclusive test, but it suggests a combination of both cloud and local processing.

[Gallifings]

Is Adobe collecting my information? What type of information?
https://kalpapharmaceuticals.to/how-...ower-estrogen/

Adobe collects anonymous behavioral data (clicks, likes, slider drags)
Adobe collects images anonymously when you include them in your feedback, and will not associate these images with a person or user
Adobe collects data for the purpose of optimizing the machine learning models to generate better results

was wondering this for a while, huge thanks for response. may i ask some other questions if i would have any? thanks

[Codebreaker]

Ask away.