Illustration: Rebecca Zisser/Axios

Autonomous vehicle companies track — and publish, at least in California — a variety of performance metrics, including how many times an operator has to take control of the autonomous system, accidents, and distance traveled autonomously. These convenient figures have become de facto measures of progress, yet the view they offer into the state of AV technology is fuzzy at best.

The bottom line: As AV technology approaches commercialization, companies will have to engage in a dialogue with government and the public — using statistical performance measures as well as qualitative analyses and processes — to demonstrate that autonomous systems are thoughtfully and safely engineered.

The metrics:

  • Disengagements: Absent contextual details, the number of times an operator takes control of the vehicle is a poor indicator of performance. Too few disengagements indicate a lack of instructive situations from which to learn. Too many, especially in similar situations, suggest the system is not learning enough. Rather than minimizing disengagements (because they are publicly reported), teams should monitor them to ensure their technology is being sufficiently challenged and learning the right lessons along the way.
  • Accidents: Egregious accidents can focus needed attention on companies with unsafe cultures or practices. But in aggregate, the lack of reliable baseline numbers for human driving makes it difficult to draw meaningful conclusions for AVs. (The majority of accidents involving human drivers go unreported.) Accident reports might provide limited insight into one aspect of a system’s capabilities, but they don't reflect a system’s overall performance — or the industry’s.
  • Distance: The number of miles driven is probably the most useful metric because real-world performance matters. While simulated miles are hugely valuable, the right road miles present a system with the challenges of unanticipated situations. But all experience is not equal: Less complicated miles, like those on uncongested freeways, are easy to accumulate though less instructive per mile than those in a challenging urban environment.

Chris Urmson is a co-founder and the CEO of Aurora.

Read more stories like this in our new weekly Axios Autonomous Vehicles newsletter. Sign up here.

Go deeper

Updated 51 mins ago - Politics & Policy

Coronavirus dashboard

Illustration: Sarah Grillo/Axios

  1. Politics: The swing states where the pandemic is raging. Ex-FDA chief: Pence campaigning after COVID exposure puts others at risk — Mark Meadows: "We are not going to control the pandemic"
  2. Health: 13 states set single-day coronavirus case records last week — U.S. reports over 80,000 new cases for second consecutive day.
  3. Education: The dangerous instability of school re-openings.
  4. World: Australian city to exit one of world's longest lockdowns — In photos: Coronavirus restrictions grow across Europe
  5. Media: Fox News president and several hosts advised to quarantine after possible COVID-19 exposure
  6. Nonprofit: Rockefeller Foundation commits $1 billion for COVID-19 recovery
Dion Rabouin, author of Markets
2 hours ago - Economy & Business

Bond investors see brighter days

Illustration: Sarah Grillo/Axios

U.S. government bonds could breakout further after yields on the benchmark 10-year Treasury note ticked up to their highest since early June last week.

But, but, but: Strategists say this move is about an improving outlook for economic growth rather than just inflation.

Erica Pandey, author of @Work
4 hours ago - Economy & Business

The dangerous instability of school re-openings

Illustration: Eniola Odetunde/Axios

Schools across the country have flip-flopped between in-person and remote learning — and that instability is taking a toll on students' ability to learn and their mental health.

The big picture: While companies were able to set long timelines for their return, schools — under immense political and social strain — had to rush to figure out how to reopen. The cobbled-together approach has hurt students, parents and teachers alike.