AI pioneer Bengio launches $30M nonprofit to rethink safety
Add Axios as your preferred source to
see more of our stories on Google.
Yoshua Bengio, testifying before a Senate subcommittee in 2023. Photo: Alex Wong/Getty Images
Machine learning pioneer Yoshua Bengio is launching a new nonprofit lab backed by roughly $30 million in funding to make AI systems act less like humans.
Why it matters: The move bucks a trend toward AI that acts independently, which Bengio and others fear might create systems that place their own interests over humanity's.
- "We've been getting inspiration from humans as the template for building intelligent machines, but that's crazy, right?" Bengio said in an interview.
- "If we continue on this path, that means we're going to be creating entities — like us — that don't want to die, and that may be smarter than us and that we're not sure if they're going to behave according to our norms and our instructions," he said.
Driving the news: Bengio, a Montreal-based researcher who has long warned about the risks of a technology he helped develop, has raised about $30 million for the nonprofit, dubbed LawZero.
- LawZero currently has about 15 staffers, Bengio said, "but the goal is to hire many more."
- Bengio is among those who have called for tougher regulation of AI development and even the breakup of big tech companies.
- Earlier this year he gave a TED Talk urging greater caution and collaboration.
The big picture: There's a growing sense of worry among critics — and even AI practitioners — that safety is taking a back seat as companies and countries race to be first with AI that can best humans in a wide variety of tasks, so-called artificial general intelligence (AGI).
- Bengio said there is also a high risk in concentrating control of advanced AI in a handful of companies.
- "You don't want AGI or superintelligence to be in the hands of one person or one company only deciding what to do, or even one government," Bengio said. "So you need very strong checks and balances."
Between the lines: Bengio says a large part of the problem is how current systems are trained. During initial training, the systems are taught to mimic humans, and then they're honed by seeing which responses people find most appealing.
- "Both of these give rise to uncontrolled agency," Bengio said.
- Some early glimmers of this are already appearing — as when Anthropic's latest model, in a test scenario, sought to blackmail its engineers to avoid shutdown.
- By contrast, Bengio says he wants to create AI systems that have intellectual distance from humans and act as more of a detached scientist than a personal companion or human agent.
- "The training principle is completely different, but it can exploit a lot of the recent advances that have happened in machine learning," he said.
Yes, but: Bengio told Axios that the $30 million should be enough to fund the basic research effort for about 18 months.
- But AI is expensive, and Bengio's new nonprofit may find it difficult to raise additional, larger funding rounds — as OpenAI and others have found.
- Bengio says he doesn't see future funding as a roadblock, as he's certain more investors now understand the plausible risks ahead.
- Governments could also be future backers of LawZero, Bengio said.
