Use ChatGPT with On-Screen Content

Now that Siri has ChatGPT built in with Apple Intelligence, there are some really interesting and creative ways you can use it, such as allowing it to access a screenshot of what’s currently on your iPhone screen. What do I mean by that? Let’s take a look at an example - finding pizza in Copenhagen - and it’ll become a lot clearer.

On the map here, I could, of course, search for pizza in the search bar, and lots of little dots will light up across Copenhagen. All of these places probably serve a decent pizza, but how do I know where the best ones are? Which places are worth visiting while I’m in the city? Let’s use ChatGPT and its broader intelligence to find out.

I’m just going to long-press the lock button to bring up Siri, and then I’ll ask ChatGPT something like this: “Ask ChatGPT where would be a good place to grab a pizza based on this map.” You can see it’s working in the background, and it will ask if I’m happy to share a screenshot of what’s currently on my phone.

Tapping on this lets me choose between sharing a screenshot or the full content - so, for example, if I were browsing a webpage, I could share the entire page. In this case, the screenshot is enough. It can see the location I’m currently browsing on the map, and ChatGPT will then analyse the image and give me recommendations based on that specific context.

And just like that, you can see it’s recognised the district in Copenhagen that might be the best place to head to if I’m looking for great pizza. While this is really useful, I do wish it could interact directly with the Maps app on my phone and perhaps highlight those recommended locations. But as of now, that isn’t possible.

However, this information means I can easily copy and paste, search for exactly what I want, and use the power of ChatGPT to help make better travel decisions using Maps. And this functionality isn’t limited to Maps - you can use ChatGPT with screenshots in any app on your iPhone. Simply share the screenshot, and it’ll provide content and recommendations based on what you’re currently looking at. That’s pretty cool!