Behind the Curtain: Intelligence explosion
Add Axios as your preferred source to
see more of our stories on Google.
Illustration: Lindsey Bailey/Axios
Anthropic, the AI lab whose identity is wrapped around warning the world about AI risk, is claiming "early signs" of AI not just coding its own products but building itself.
Why it matters: Anthropic co-founder Jack Clark predicted this week that there's a 60%+ chance of an AI model fully training its successor by the end of 2028. "What I'm looking at is a technological trend where, if anything, the speed will accelerate further," Clark told us.
In the new research agenda for The Anthropic Institute — first shared with us, and out Thursday — the company says it's seeing signs of "AI contributing to speeding up the research and development of AI itself," a process known as recursive self-improvement. And Anthropic researchers think the world should know.
"My prediction is by the end of 2028, it's more likely than not that we have an AI system where you would be able to say to it: 'Make a better version of yourself.' And it just goes off and does that completely autonomously," Clark, who heads the institute, told us from Anthropic headquarters in San Francisco.
- "It's always been the case that humans outside the technology need to come up with the ideas that they then put back into it. What happens if we have a technology that can generate ideas within itself for how to improve itself? That's a new concept."
The five-page document warns of a possible "intelligence explosion" — long a theoretical term confined to AI safety circles. Now it's in writing, in an official Anthropic document.
- Clark told us an intelligence explosion is when AI systems suddenly start improving at blinding speed. Lots of bad things can happen (cyber meltdowns and biological attacks). And lots of good:
- "What do you do with a tremendous amount of growth or a tremendous amount of abundance in many, many different fields of science all at once?" he asked. "Today's institutions have very, very narrow pipes through which you push new drug candidates. How do you massively broaden the size of those pipes in advance of this abundance?"
What's new: The Anthropic Institute is part research arm, part early-warning system, with an agenda built alongside Anthropic's Long-Term Benefit Trust. (Clark also has the title "head of public benefit.") The research agenda focuses on four buckets:
- Economic diffusion: jobs, productivity, who captures the gains.
- Threats and resilience: cyber, bio, surveillance.
- AI systems in the wild: agents, governance.
- AI-driven R&D: the recursive self-improvement question.
The promise: Anthropic is committing to publishing more "detailed information about how our work at Anthropic has sped up as a result of new AI tools, and ideas about the implications of potential recursive self-improvement of AI systems."
- Translation: A frontier lab is on the record promising to tell the public when the machine starts building itself.
- If AI is building itself, will we need AI companies? "We and the other companies are going to be taking this technology and trying to get it to do good in the world," Clark told us. "To help push forward things like biology or medicine or robotics … To steer that technology into domains where it's really, really, really hard to make progress, like cancer research."
Between the lines: The agenda asks how to run a "fire drill" for an intelligence explosion — a tabletop exercise that "actually tests the decision-making of lab leadership, boards, and governments."
- Labs don't draw up fire drills for problems they think are decades away.
- The institute notes that during the Cold War, the U.S. had a hotline to the Kremlin in case of a nuclear crisis. Similar geopolitical infrastructure could be needed for a crisis involving AI systems. "One of the lessons from the Cold War is that rival nations dealing with technology that has an existential impact on the human race found ways to talk to each other about it," Clark said. "And we are going to need to do the same here."
What it means for jobs: Anthropic will publish monthly reports on how AI is reshaping work, designed as "an early warning signal for significant change and disruption."
- The document asks whether AI companies, "in partnership with government," might turn industrywide "dials" to throttle AI diffusion sector by sector, the way central banks throttle inflation.
- An AI lab publicly entertaining coordinated industrial-policy levers on its own technology — that's new. "We are planning for success here," Clark said. "We're planning for a world where the technology gets as powerful as we think, and we deal with these issues of misuse or misalignment en route."
Reality check: This is also a positioning play. Anthropic has built a brand on being the responsible frontier lab. An institute anchored to its safety trust extends that brand ahead of whatever model upgrade lands next.
- "The motivation has always been: Tell the whole story," Clark told us. "Sometimes that means that we talk about risks that we're worried about. Sometimes that means that we're going to talk about amazing, hitherto uncontemplated amounts of abundance. … I'm just trying to get ahead of what I think of as the next big question and get Anthropic ahead of that."
The bottom line: That doesn't make the recursive self-improvement admission less real. It makes it more interesting that Anthropic chose to put it in writing.
📱 Watch a "Behind the Curtain" video of Mike interviewing Jack Clark. (Thanks to executive producer Jimmy Shelton for the lightning turn.)
