Jan 23, 2019

Why AV sensors need simulations of their own

Illustration: Rebecca Zisser/Axios

Simulation has been critical to speeding up AV development, especially for motion planning and control algorithms. But in these simulations, vehicles are either not perceiving the simulated world at all or perceiving only scenarios that have been previously encountered on the road.

Why it matters: Using these approaches, vast amounts of on-road data will need to be gathered in order to reach the edges of a perception module's capabilities. An additional layer of simulation for AV sensors and perception systems could use synthetic data to accelerate the development of AVs.

Background: AV simulations test different modules of the self-driving software using map data and information about obstacles on the road.

  • In re-simulations, saved data from on-road drives can be fed back through the software. AV engineers can develop test scenarios to benchmark software performance and then compare the results after making updates.

What's needed: Simulating sensor models as well could help in several areas.

  • Optimizing sensor placement on vehicles. Being able to position and visualize the output from sensors is easier and faster using software than real hardware.
  • Creating data sets to train for rare situations. Simulations can model any imaginable scenario, however unlikely.
  • Feeding perception algorithms with synthetic sensor data. This allows engineers to continuously create new scenarios and test their algorithms, rather than waiting for new on-road data.

The catch: Sensor simulation for perception algorithms is difficult because each sensor type (lidar, radar, camera, infrared) has different levels of complexity.

  • In an ideal case, a real hardware sensor’s information could be reproduced using purely software. This requires modeling the underlying physics of each sensor and carefully constructing the 3D environment in which the synthetic sensors “operate.”
  • Synthetic data could then be passed to perception algorithms in a form that they can’t differentiate from real data.

These simulation platforms have to be capable of creating and rendering large photorealistic environments in real time.

  • Sensor tasks add to the load: Properly simulating radar or lidar can require casting millions of synthetic rays per second.

The bottom line: Sensor simulation for perception is still a new process, but could fill an important gap in the development cycle of AV software.

Chris Gundling is a senior software engineer at Applied Intuition, an AV simulation software company.

Go deeper

Updated 46 mins ago - Politics & Policy

U.S. enters 6th day of nationwide protests over George Floyd's killing

A protest in Philadelphia on May 31. Photo: Mark Makela/Getty Images

The D.C. National Guard is being called to assist police with protests, per AP, as protests continue past the city's 11 p.m. curfew.

What's happening: Police fired tear gas into a crowd of over 1,000 people in Washington, D.C.'s Lafayette Square across from the White House one hour before Sunday's 11 p.m. curfew, AP reports. Earlier in the night, protestors held a stand off in Lafayette Square, after previously breaking through a White House police barricade. A fire in the basement of the city's historic St. Johns Church was extinguished.

Updated 3 hours ago - Politics & Policy

Journalists get caught in the crosshairs as protests unfold

A man waves a Black Lives Matter flag atop the CNN logo outside the CNN Center during a protest in response to the police killing of George Floyd, Atlanta, Georgia, May 29. Photo: Elijah Nouvelage/Getty Images

Dozens of journalists across the country tweeted videos Saturday night of themselves and their crews getting arrested, being shot at by police with rubber bullets, targeted with tear gas by authorities or assaulted by protesters.

Driving the news: The violence got so bad over the weekend that on Sunday the Cleveland police said the media was not allowed downtown unless "they are inside their place of business" — drawing ire from news outlets around the country, who argued that such access is a critical part of adequately covering protests.

Updated 4 hours ago - Politics & Policy

Tanker truck plows into Minneapolis protesters

The tanker after plowing into protesters on the shut-down bridge in Minneapolis on Sunday evening. Authorities said it appeared protesters escaped injury. Photo: Jeff Wheeler/Star Tribune via Getty Images

Minnesota authorities said in a statement they're investigating as a criminal matter what happened with a truck that "drove into demonstrators" on a Minneapolis bridge Sunday evening while the eight-lane road was closed for a protest.

What they're saying: Minnesota Department of Public Safety tweeted, "Very disturbing actions by a truck driver on I-35W, inciting a crowd of peaceful demonstrators. The truck driver was injured & taken to a hospital with non-life threatening injuries. He is under arrest. It doesn't appear any protesters were hit by the truck."