Illustration: Rebecca Zisser/Axios

Autonomous vehicle companies track — and publish, at least in California — a variety of performance metrics, including how many times an operator has to take control of the autonomous system, accidents, and distance traveled autonomously. These convenient figures have become de facto measures of progress, yet the view they offer into the state of AV technology is fuzzy at best.

The bottom line: As AV technology approaches commercialization, companies will have to engage in a dialogue with government and the public — using statistical performance measures as well as qualitative analyses and processes — to demonstrate that autonomous systems are thoughtfully and safely engineered.

The metrics:

  • Disengagements: Absent contextual details, the number of times an operator takes control of the vehicle is a poor indicator of performance. Too few disengagements indicate a lack of instructive situations from which to learn. Too many, especially in similar situations, suggest the system is not learning enough. Rather than minimizing disengagements (because they are publicly reported), teams should monitor them to ensure their technology is being sufficiently challenged and learning the right lessons along the way.
  • Accidents: Egregious accidents can focus needed attention on companies with unsafe cultures or practices. But in aggregate, the lack of reliable baseline numbers for human driving makes it difficult to draw meaningful conclusions for AVs. (The majority of accidents involving human drivers go unreported.) Accident reports might provide limited insight into one aspect of a system’s capabilities, but they don't reflect a system’s overall performance — or the industry’s.
  • Distance: The number of miles driven is probably the most useful metric because real-world performance matters. While simulated miles are hugely valuable, the right road miles present a system with the challenges of unanticipated situations. But all experience is not equal: Less complicated miles, like those on uncongested freeways, are easy to accumulate though less instructive per mile than those in a challenging urban environment.

Chris Urmson is a co-founder and the CEO of Aurora.

Read more stories like this in our new weekly Axios Autonomous Vehicles newsletter. Sign up here.

Go deeper

Cleanup on aisle Biden

Photo: Andrew Caballero-Reynolds/AFP via Getty Images

After two gaffes and a low blow from President Trump questioning his faith, Joe Biden spent Thursday evening off his own message — clarifying comments and responding to attacks.

Why it matters: Biden’s responses reflect what we could see a lot more of in the next few months — cringeworthy comments and Trump smears, smacking into each other and pulling the Democrat off course.

2020 election strategy: Hire all the lawyers

Illustration: Eniola Odetunde/Axios

The coronavirus has sent overall U.S. unemployment into the double digits — but it's a sort of full-employment act for election law attorneys.

The big picture: The prospect of extended court fights over COVID-19-related voting changes, an absentee ballot avalanche, foreign interference and contested presidential results has prompted a hire-all-the-lawyers binge by candidates and campaigns — not just in swing states but around the country.

Right-wing media defanged by dissolving anti-Biden storylines

Data: NewsWhip; Chart: Naema Ahmed/Axios

The three biggest anti-Joe Biden storylines in right-wing media over the last year have either fizzled or are getting less online traction than they used to, according to data from NewsWhip provided exclusively to Axios.

Why it matters: This dynamic has rendered a formidable media ecosystem less effective in boosting President Trump as we move into the heart of the 2020 campaign.