AI will soon run directly on your phone
Add Axios as your preferred source to
see more of our stories on Google.
Generative AI running a smartphone. Image: Qualcomm
ChatGPT and most other generative AI relies on the power of the cloud and a strong connection to the internet, but now some big names in tech are pushing hard to run generative AI directly on smartphones and laptops.
Why it matters: Running AI models on device could make them cheaper, faster and more private.
Cell phone chip specialist Qualcomm showed off a demo Tuesday of a reference smartphone running Stable Diffusion and producing an image in less than a second.
- In a demo earlier this year, a similar task took 15 seconds.
- At its Snapdragon Summit in Maui yesterday, Qualcomm also showed versions of Meta's Llama 2 model running on both smartphones and a prototype Snapdragon-powered laptop.
What's next: Apple was caught a bit flat footed as generative AI took off, but is said to be hard at work on models of its own and is aiming to incorporate broader uses of the technology into upcoming versions of iOS and Siri, per a weekend report from Bloomberg's Mark Gurman.
- Running its models on-device would be in line with the company's privacy efforts, which have focused on minimizing the amount of sensitive information collected by Apple servers.
- Google is making widespread use of on-device AI in its Pixel devices, especially with photos. It said Tuesday it is also working with Qualcomm to allow a number of its AI models to run on Android devices.
The big picture: Generative AI shows huge potential for a range of tasks, from drafting documents and presentations to sifting through large amounts of data. However, creating software that has to constantly ping servers over the internet adds significant costs.
- "Because cloud costs are going to skyrocket, these guys are going to say, 'Okay, I'm going to try to figure out how to do more and more on device,' " Qualcomm senior VP Alex Katouzian told Axios.
- Qualcomm focused on the power of on-device AI in touting both its latest Snapdragon 8 Gen 3 mobile chip and a long awaited PC chip it hopes to use to finally grab a significant chunk of the PC market.
- And even if some queries do require the power of the cloud, running others locally can save money and improve performance. Microsoft and others envision such a hybrid approach.
Hardware and chip makers are also hoping that generative AI could help spur sales of computers and smartphones, both of which have flagged after seeing a boom during the pandemic.
- Consumers who once replaced their smartphones every couple of years are now waiting as long as five or six years before purchasing a new one, Katouzian said.
Between the lines: Companies ranging from startups to giants like Microsoft and Meta also see the potential of AI to help unlock the hold that Google and Apple have had over smartphones, due in large part to their control over the main app stores for their respective platforms.
- There are new uses for generative AI that can run proactively, rather than requiring a person to actively query a chatbot. Qualcomm showed demos on Tuesday of an always-running AI agent that can offer suggested meeting places and times, or photos to share.
- "There is zero chance of that running in the cloud," Qualcomm senior VP Durga Malladi said in an interview. "It's simply not possible."
Yes, but: Today's generative AI consumes a tremendous amount of computing power. Running similar models on devices still requires a lot of horsepower, and its impact on battery life is yet to be seen.
- Plus, even if much of what is being done today can shift to the device, there will be further improvements and new uses that require computing resources beyond what a laptop or phone can deliver.
Disclosure: Reporting for this article took place at Qualcomm's Snapdragon Summit in Maui, where I am moderating an AI-related panel on Thursday. Qualcomm paid for my travel-related costs.
