The AI system operating in maze environments with partial information. Credit: DeepMind

A machine learning system from Google's DeepMind can collect snapshots of a 3D scene taken from different angles and then predict what that environment will look like from a viewpoint it hasn't seen before, according to research published today in Science.

The big picture: Researchers want to create AIs that can build models of the world from data they've seen and then use those models to function in new environments. That capability could take an AI from the realm of learning about a space to understanding it — much the same way humans do — and is key to developing machines that can move autonomously through the world. (Think: driverless cars.)

The context: Computer vision — spurred by the availability of data and increased computing power — has rapidly advanced in the past six years. Many of the underlying algorithms largely learn via supervision: an algorithm is given a large dataset that is labeled with information (for example, about the object in a scene) and uses it to predict an output.

“Supervised learning has been super successful but it’s unsatisfying for two reasons. One, humans need to manually create the [training] datasets, which is expensive and they don’t capture everything. And two, it is not the way infants or higher mammals learn.”
— Ali Eslami, study author and researcher at DeepMind

Instead, researchers want to train machines to learn from unlabeled inputs that they process without any guidance from a human, and then to be able to apply or transfer what they learn to other new scenarios and tasks.

How it works: The system uses a pair of images of a virtual 3D scene taken from different angles to create a representation of the space. A separate “generation” network then predicts what the scene will look like from a different viewpoint it hasn’t seen before.

  • After training the generative query network (GQN) on millions of images, it could use one image to determine the identity, position and color of objects as well as shadows and other aspects of perspective, the authors wrote.
  • That ability to understand the scene's structure is the "most fascinating" part of the study, wrote the University of Maryland's Matthias Zwicker, who wasn't involved in the research.
  • The DeepMind researchers also tested the AI in a maze and reported the network can accurately predict a scene with only partial information.
  • A virtual robotic arm could also be controlled by the GQN to reach a colored object in a scene.

Yes, but: These are relatively simple virtual environments and "it remains unclear how close [the researchers'] approach could come to understanding complex, real-world environments," Zwicker writes.

Harvard's Sam Gershman told MIT Technology Review the GQN still solves only the narrow problem of predicting what a scene looks like from a different angle. According to the article:

"Gershman says it’s unclear whether DeepMind’s approach could be adapted to answer more complex questions or whether some fundamentally different approach might be required."

The challenges: Eslami says it took a couple of months to train the network. “We really were pushing the hardware available to us to its limits. We need a step up in hardware capabilities and the techniques to build these deep neural networks and train them.”

Go deeper: Read more about the various ways researchers are trying to design AI to work like the human brain.

Go deeper

Updated 2 hours ago - Politics & Policy

Coronavirus dashboard

Illustration: Sarah Grillo/Axios

  1. Global: Total confirmed cases as of 7 p.m. ET: 19,734,428— Total deaths: 728,612 — Total recoveries — 12,001,537Map.
  2. U.S.: Total confirmed cases as of 7 p.m. ET: 5,036,387 — Total deaths: 162,851 — Total recoveries: 1,656,864 — Total tests: 61,792,571Map.
  3. Politics: Pelosi says states don't have the funds to comply with Trump's executive order on unemployment — Mnuchin says Trump executive orders were cleared by Justice Department.
  4. Public health: Ex-FDA head: U.S. will "definitely" see 200,000 to 300,000 virus deaths by end of 2020 — Fauci says chances are "not great" that COVID-19 vaccine will be 98% effective.
  5. Schools: Nine test positive at Georgia school where photo showing packed hallway went viral — How back-to-school is playing out in the South as coronavirus rages on.
2 hours ago - World

Protests erupt in Belarus after "Europe's last dictator" claims election victory

A man lies on the ground in front of riot police in Minsk. Photo: Sergei Gapon/AFP via Getty Images

Demonstrations broke out across Belarus on Sunday after a government exit poll predicted that President Aleksander Lukashenko, an authoritarian who has ruled the Eastern European country since 1994, had swept to overwhelming victory over a pro-democracy opposition candidate.

Why it matters: It's a precarious moment for the former Soviet republic, where decades of repression and a complete disregard for the coronavirus pandemic now threaten to topple "Europe's last dictator."

Scoop: Inside Trump's debate prep

Trump and Christie. Photo: Jim Watson/AFP via Getty Images

Two weekends ago, President Trump met with a group of his closest aides in the conference room of his Bedminster golf club to discuss a subject that has been weighing heavily on his mind: the three scheduled debates with Joe Biden.

Behind the scenes: In the room with Trump were his son-in-law Jared Kushner, campaign manager Bill Stepien, senior adviser Jason Miller, and former New Jersey Gov. Chris Christie, who role-played Hillary Clinton in Trump's 2016 debate prep sessions.