Machine learning pioneer Yoshua Bengio is launching a new nonprofit lab backed by roughly $30 million in funding to make AI systems act less like humans.

Why it matters: The move bucks a trend toward AI that acts independently, which Bengio and others fear might create systems that place their own interests over humanity's.

• "We've been getting inspiration from humans as the template for building intelligent machines, but that's crazy, right?" Bengio said in an interview.
• "If we continue on this path, that means we're going to be creating entities — like us — that don't want to die, and that may be smarter than us and that we're not sure if they're going to behave according to our norms and our instructions," he said.

Driving the news: Bengio, a Montreal-based researcher who has long warned about the risks of a technology he helped develop, has raised about $30 million for the nonprofit, dubbed LawZero.

• LawZero currently has about 15 staffers, Bengio said, "but the goal is to hire many more."
• Bengio is among those who have called for tougher regulation of AI development and even the breakup of big tech companies.
• Earlier this year he gave a TED Talk urging greater caution and collaboration.

The big picture: There's a growing sense of worry among critics — and even AI practitioners — that safety is taking a back seat as companies and countries race to be first with AI that can best humans in a wide variety of tasks, so-called artificial general intelligence (AGI).

• Bengio said there is also a high risk in concentrating control of advanced AI in a handful of companies.
• "You don't want AGI or superintelligence to be in the hands of one person or one company only deciding what to do, or even one government," Bengio said. "So you need very strong checks and balances."

Between the lines: Bengio says a large part of the problem is how current systems are trained. During initial training, the systems are taught to mimic humans, and then they're honed by seeing which responses people find most appealing.

• "Both of these give rise to uncontrolled agency," Bengio said.
• Some early glimmers of this are already appearing — as when Anthropic's latest model, in a test scenario, sought to blackmail its engineers to avoid shutdown.

By contrast, Bengio says he wants to create AI systems that have intellectual distance from humans and act as more of a detached scientist than a personal companion or human agent.

• "The training principle is completely different, but it can exploit a lot of the recent advances that have happened in machine learning," he said.

Yes, but: Bengio told Axios that the $30 million should be enough to fund the basic research effort for about 18 months.

• But AI is expensive, and Bengio's new nonprofit may find it difficult to raise additional, larger funding rounds — as OpenAI and others have found.
• Bengio says he doesn't see future funding as a roadblock, as he's certain more investors now understand the plausible risks ahead.
• Governments could also be future backers of LawZero, Bengio said.

Mary Meeker, the famed internet analyst turned venture capitalist has published her first Trends report since 2019, focused on the AI revolution.

• Silicon Valley execs and investors are sure to pore over all 340 pages, but Axios chatted with Meeker to distill some top takeaways.

What follows is an edited version of our conversation:

Axios: What was the most important thing you learned?

Meeker: We've never seen anything like the user growth of ChatGPT, particularly outside the U.S., and it shows how the global dynamics of tech and distribution have changed.

• I wasn't around for the evolution of the mainframe or minicomputer, but have read up on it and was around for the PC, desktop internet, mobile internet, cloud, and now AI. This is a much faster pace of change.
• It's also fascinating to imagine what might have happened if Microsoft hadn't come in and given OpenAI capital and focused them on its cloud platform. Google was very early in AI, but it couldn't have launched a product that hallucinated like OpenAI did. Startups can do crazy things.

Does OpenAI have an insurmountable lead, at least in terms of consumer AI?

Early leaders fail when they fall behind on innovation or price themselves out of the market, but OpenAI shows no signs of doing either. I think it and ChatGPT will be around for a long time.

• That said, they also have intense competition the likes of which we've never seen before. Both startups and incumbents. Everyone is engaged. It's a period for lots of wealth creation and wealth destruction.

How should early-stage VCs try to suss out potential winners from losers?

I recently rewatched one of Steve Ballmer's speeches at an early Windows conference where he kept repeating the word "developers." And he's right, that the companies that get the best developers often win. It's got to be the focus.

If people are key, and the U.S. is in a race with China, what do you think about the Trump administration's recent moves to block foreigners from studying here?

America has to be a place where the brightest people in the world want to come. ... Our tech ecosystem would not be where it is today without a lot of first-generation immigrants and even more second-generation immigrants.

• I think this administration has shown an ability to make foundational calls, but then also to change. And I go back to the first Trump administration and its focus on immigrants who could really contribute to America in positive ways.

• The FDA is using AI to review clinical protocols and speed up scientific evaluations. (Axios)
• Elon Musk is selling $5 billion in debt for xAI Corp., the latest sign he's turning his attention away from politics and back to his sprawling business ventures. (Bloomberg)
• Meta signed a 20-year agreement to buy nuclear power from an Illinois plant to lock up energy for its AI needs, in what participants called a "multibillion-dollar" deal. (Axios)

Flying to New York midday allowed me to get some fun photos of the Bay Area, including pictures of Lake Merritt and of an all-too-empty Oracle Arena/Oakland Coliseum, but one image left me puzzled.

• ChatGPT (which is known for helping identify locations from photos) says it is likely Lehigh Permanente Quarry, located in Cupertino at the base of the Santa Cruz Mountains.