Illustration: Sarah Grillo/Axios
Training a new driver is straightforward — make them practice until they can master basic skills well enough to pass a driver's license exam. But there are no such tests for automated vehicles, leaving it up to AV developers to decide when their technology is safe enough to deploy.
Why it matters: AVs could reduce the number of traffic deaths and increase mobility for those who can't drive, but only if the public trusts them. With no prescribed validation methods — and regulators largely on the sidelines as the technologies are advancing — it's difficult to know how safe is safe enough.
The big picture: "It's kind of the wild, wild west out there," says Consumer Reports' David Friedman, a former acting administrator of the National Highway Traffic Safety Administration.
- Even in California, which is a leader in AV policy, manufacturers merely need to certify that they have conducted their own validation tests and are satisfied their vehicles are safe for deployment on public roads.
What's happening: There are basically 3 ways to test automated vehicle technology...
- Run computer simulations of every possible scenario.
- Perform tests on closed-course loops.
- Drive around endlessly on pre-mapped routes.
Most developers use a combination of all three, but their approaches can vary. Some recent examples are...
- Aurora Technologies leans heavily on simulation rather than trying to rack up millions of real-world miles in a large test fleet, per a voluntary safety report filed with NHTSA on April 29.
- Argo AI, owned by Ford, uses a "continuous testing" process including how it trains its AV safety drivers, according to posted details of its testing process on its website.
Tesla is the outlier — its testing process is heavily dependent on cars already on the road, which the company sees as a huge competitive advantage.
- The company pulled back the curtain on its powerful new computer chip at an investor event on April 22, explaining how it collects data from 425,000 customer-owned vehicles to help train its algorithms.
- The system learns every time a Tesla driver intervenes with the car's Autopilot system.
- Even when Autopilot isn't engaged, the system keeps learning through what Tesla describes as "shadow mode" — computing what the car would have done if Autopilot had been active, and comparing it to what the driver actually did.
- Tesla says it can shadow test many millions of miles every day, compared to the much slower rate of companies using safety drivers.
Yes, but: It's not clear if the data captured from Tesla's consumer fleet is better at training its AV system than data collected through more deliberative testing processes by other companies.
What's needed: Several experts tell me there should be an independent, third-party assessment of each company's safety claims.
"Self-driving cars are anything but proven. All we're asking is to get us the data to prove these things deliver real safety benefits before we start using people as human guinea pigs."— David Friedman
What to watch: A new consortium is working to define safety testing practices to lay the groundwork for formal industry standards. Its first recommendations are expected this summer.
Go deeper: Regulating the humans behind the wheels of autonomous vehicles