AI's "bigger is better" faith begins to dim
Add Axios as your preferred source to
see more of our stories on Google.
Illustration: Aïda Amer/Axios
The generative AI revolution — built on the belief that models will keep getting wildly better as they grow crazily bigger — faces new fears that it might plateau out.
Why it matters: Two years after ChatGPT launched, the tech industry, led by OpenAI, has bet billions on a scaling strategy — assemble mountains of chips and data, make tomorrow's large language models even larger than today's, and watch the technology advance. Those bets, always risky, could go bad.
Driving the news: Some OpenAI employees are saying the company's next flagship model, called Orion, will not improve on its predecessor, GPT-4, as impressively as GPT-4 excelled over GPT-3, both The Information and Reuters reported over the weekend.
- Since GPT-4 was released in March 2023, industry observers have debated whether OpenAI can top it — and how long this next generation would take.
- Both Google and OpenAI competitor Anthropic are also encountering setbacks and delays in efforts to advance the next generation of their key foundation models, Bloomberg reported Wednesday.
OpenAI CEO Sam Altman has repeatedly affirmed his faith in the "just make it bigger" approach.
- In his "The Intelligence Age" manifesto earlier this year, Altman wrote that deep learning gets "predictably better with scale": "To a shocking degree of precision, the more compute and data available, the better it gets at helping people solve hard problems."
Yes, but: Computing power is not infinite, and it burns fast through dollars and electricity, while today's models have already been fed most of the quality data that's available (often used without a clear legal right to do so).
- Efforts to train models with synthetic data — data that's AI-generated — have yet to prove dependable.
The other side: When engineers find that one strategy stops working, they look for another.
- The industry has already begun researching and implementing alternative techniques to the "just make it bigger" approach that could continue to improve the performance of generative AI models.
- Researchers are trying to shrink models so they consume less computing energy but perform well on specialized tasks.
- OpenAI has rolled out a new "reasoning model," called o1 (formerly "Strawberry"), that improves its performance by using more computing resources, and taking more time, as it answers users' questions.
What they're saying: Skeptics have regularly warned of limits to the just-make-it-bigger approach to improving LLMs.
- A year ago, Bill Gates said he believed GPT-4's successor would disappoint.
- AI critic Gary Marcus, who has long predicted a plateauing of generative AI advances, took a victory lap over the new reports.
Between the lines: The prospect of LLMs hitting a wall touches on an even bigger debate about how the field might reach its coveted goal of human-like intelligence (also known as artificial general intelligence, or AGI).
- Some researchers believe the path to AGI lies through data-heavy and power-hungry generative AI.
- But others are working on different AI techniques, including combining neural networks that underpin generative AI and hard-wired knowledge. The approach was used by DeepMind to build an AI that can solve sophisticated math problems.
Our thought bubble: Moore's Law — the principle predicting regular doubling of chip performance every 18 months to two years — eventually hit a wall, too (as its namesake, Intel cofounder Gordon Moore, had predicted).
- But that took many decades, while generative AI's evolution has been speedier.
- The Moore's Law wall meant that it became a lot harder for semiconductor makers to boost performance by squeezing more transistors into the space on a chip.
- That just pushed the industry to find other ways to speed up computing, including the use of new materials and new kinds of lithography.
What we're watching: Wall Street is fretting about returns on the $200 billion Big Tech is spending on AI this year.
- The scaling approach is running up the technology's price tag to prohibitive levels.
- Applications outside of a few fields like software programming and customer service remain speculative. And consumer adoption may already be slowing down.
What they're saying: "[T]here's such an appetite and a yearning for something practical, real and not the pie-in-the-sky 'AI can do everything for you,' " says Karthik Dinakar, cofounder and CTO of Pienso, which helps people build custom AI models.
- "You can't GPT your way out of this," he says.
Editor's note: This story has been updated with a mention of reported delays at Google and Anthropic.
