Sign up for our daily briefing
Make your busy days simpler with Axios AM/PM. Catch up on what's new and why it matters in just 5 minutes.
Catch up on the day's biggest business stories
Subscribe to Axios Closer for insights into the day’s business news and trends and why they matter
Stay on top of the latest market trends
Subscribe to Axios Markets for the latest market trends and economic insights. Sign up for free.
Sports news worthy of your time
Binge on the stats and stories that drive the sports world with Axios Sports. Sign up for free.
Tech news worthy of your time
Get our smart take on technology from the Valley and D.C. with Axios Login. Sign up for free.
Get the inside stories
Get an insider's guide to the new White House with Axios Sneak Peek. Sign up for free.
Catch up on coronavirus stories and special reports, curated by Mike Allen everyday
Catch up on coronavirus stories and special reports, curated by Mike Allen everyday
Want a daily digest of the top Denver news?
Get a daily digest of the most important stories affecting your hometown with Axios Denver
Want a daily digest of the top Des Moines news?
Get a daily digest of the most important stories affecting your hometown with Axios Des Moines
Want a daily digest of the top Twin Cities news?
Get a daily digest of the most important stories affecting your hometown with Axios Twin Cities
Want a daily digest of the top Tampa Bay news?
Get a daily digest of the most important stories affecting your hometown with Axios Tampa Bay
Want a daily digest of the top Charlotte news?
Get a daily digest of the most important stories affecting your hometown with Axios Charlotte
Illustration: Rebecca Zisser/Axios
Autonomous vehicle companies track — and publish, at least in California — a variety of performance metrics, including how many times an operator has to take control of the autonomous system, accidents, and distance traveled autonomously. These convenient figures have become de facto measures of progress, yet the view they offer into the state of AV technology is fuzzy at best.
The bottom line: As AV technology approaches commercialization, companies will have to engage in a dialogue with government and the public — using statistical performance measures as well as qualitative analyses and processes — to demonstrate that autonomous systems are thoughtfully and safely engineered.
The metrics:
- Disengagements: Absent contextual details, the number of times an operator takes control of the vehicle is a poor indicator of performance. Too few disengagements indicate a lack of instructive situations from which to learn. Too many, especially in similar situations, suggest the system is not learning enough. Rather than minimizing disengagements (because they are publicly reported), teams should monitor them to ensure their technology is being sufficiently challenged and learning the right lessons along the way.
- Accidents: Egregious accidents can focus needed attention on companies with unsafe cultures or practices. But in aggregate, the lack of reliable baseline numbers for human driving makes it difficult to draw meaningful conclusions for AVs. (The majority of accidents involving human drivers go unreported.) Accident reports might provide limited insight into one aspect of a system’s capabilities, but they don't reflect a system’s overall performance — or the industry’s.
- Distance: The number of miles driven is probably the most useful metric because real-world performance matters. While simulated miles are hugely valuable, the right road miles present a system with the challenges of unanticipated situations. But all experience is not equal: Less complicated miles, like those on uncongested freeways, are easy to accumulate though less instructive per mile than those in a challenging urban environment.
Chris Urmson is a co-founder and the CEO of Aurora.
Read more stories like this in our new weekly Axios Autonomous Vehicles newsletter. Sign up here.