Sep 21, 2018

The fuzzy numbers for tracking AV progress

Illustration: Rebecca Zisser/Axios

Autonomous vehicle companies track — and publish, at least in California — a variety of performance metrics, including how many times an operator has to take control of the autonomous system, accidents, and distance traveled autonomously. These convenient figures have become de facto measures of progress, yet the view they offer into the state of AV technology is fuzzy at best.

The bottom line: As AV technology approaches commercialization, companies will have to engage in a dialogue with government and the public — using statistical performance measures as well as qualitative analyses and processes — to demonstrate that autonomous systems are thoughtfully and safely engineered.

The metrics:

  • Disengagements: Absent contextual details, the number of times an operator takes control of the vehicle is a poor indicator of performance. Too few disengagements indicate a lack of instructive situations from which to learn. Too many, especially in similar situations, suggest the system is not learning enough. Rather than minimizing disengagements (because they are publicly reported), teams should monitor them to ensure their technology is being sufficiently challenged and learning the right lessons along the way.
  • Accidents: Egregious accidents can focus needed attention on companies with unsafe cultures or practices. But in aggregate, the lack of reliable baseline numbers for human driving makes it difficult to draw meaningful conclusions for AVs. (The majority of accidents involving human drivers go unreported.) Accident reports might provide limited insight into one aspect of a system’s capabilities, but they don't reflect a system’s overall performance — or the industry’s.
  • Distance: The number of miles driven is probably the most useful metric because real-world performance matters. While simulated miles are hugely valuable, the right road miles present a system with the challenges of unanticipated situations. But all experience is not equal: Less complicated miles, like those on uncongested freeways, are easy to accumulate though less instructive per mile than those in a challenging urban environment.

Chris Urmson is a co-founder and the CEO of Aurora.

Read more stories like this in our new weekly Axios Autonomous Vehicles newsletter. Sign up here.

Go deeper

Coronavirus dashboard

Illustration: Aïda Amer/Axios

  1. Global: Total confirmed cases as of 4 p.m. ET: 5,463,392 — Total deaths: 344,503 — Total recoveries — 2,195,325Map.
  2. U.S.: Total confirmed cases as of 4 p.m. ET: 1,653,904 — Total deaths: 97,948 — Total recoveries: 366,736 — Total tested: 14,163,915Map.
  3. World: Italy reports lowest number of new cases since February — Ireland reports no new coronavirus deaths on Monday for the first time since March 21 — WHO suspends trial of hydroxychloroquine over safety concerns.
  4. 2020: Trump threatens to move Republican convention from North Carolina — Joe Biden makes first public appearance in two months.
  5. Public health: Officials are urging Americans to wear masks over Memorial Day.
  6. Economy: New York stock exchange to reopen its floor on Tuesday — White House economic adviser Kevin Hassett says it's possible the unemployment rate could still be in double digits by November's election — Charities refocus their efforts to fill gaps left by government.
  7. What should I do? Hydroxychloroquine questions answeredTraveling, asthma, dishes, disinfectants and being contagiousMasks, lending books and self-isolatingExercise, laundry, what counts as soap — Pets, moving and personal healthAnswers about the virus from Axios expertsWhat to know about social distancingHow to minimize your risk.
  8. Other resources: CDC on how to avoid the virus, what to do if you get it, the right mask to wear.

Subscribe to Mike Allen's Axios AM to follow our coronavirus coverage each morning from your inbox.

Updated 2 hours ago - Politics & Policy

Italy reports lowest number of new coronavirus cases since February

Italy’s aerobatic team Frecce Tricolori fly over Milan in Duomo Square on May 25. Photo: Francesco Prandoni/Getty Images

The Italian government reported 300 new cases of coronavirus on Monday, the lowest daily increase since Feb. 29.

Why it matters: Italy, the first country in Europe to implement a nationwide lockdown after emerging as a hotspot in March, appears to have finally weathered its coronavirus outbreak. Italy has reported nearly 33,000 total deaths, the third-highest total behind the U.S. and U.K.

Joe Biden makes first public appearance in over two months

Photo: Oliver Douliery/AFP via Getty Images

Former Vice President Joe Biden made his first in-person appearance in over two months on Monday to honor Memorial Day by laying a wreath at a Delaware veterans park, AP reports.

Why it matters: Biden, the Democratic Party's presumptive nominee, has taken the unprecedented step of campaigning from his home during the coronavirus pandemic, ever since canceling a rally in Cleveland on March 10.