Illustration: Eniola Odetunde/Axios

In the most exclusive AI conferences and journals, AI systems are judged largely on their accuracy: How well do they stack up against human-level translation or vision or speech?

Yes, but: In the messy real world, even the most accurate programs can stumble and break. Considerations that matter little in the lab, like reliability or computing and environmental costs, are huge hurdles for businesses.

Why it matters: Some stumbles that have tarred high-profile AI systems — like facial recognition that fails more often on darker faces, or medical AI that gives potentially harmful advice — have resulted from unanticipated real-world scenarios.

The big picture: Research often guns for incremental advances, juicing an extra percentage point of accuracy out of a previously proposed model.

  • "We feel like this is the only thing that gets rewarded nowadays in the community," says Roy Schwartz, a researcher at the Allen Institute for Artificial Intelligence. "We want to argue that there are other things to consider."
  • In a paper posted this summer, Schwartz and several co-authors proposed that researchers present not just the accuracy of their models but also the computing cost it took to get there — and the resulting environmental toll.

That's one of several hidden considerations that can drive up the cost of creating an AI model that works in the real world. Among the other important factors accuracy doesn't capture:

  • The cost of labeling training data, which is still mostly done by humans.
  • The reliability of a system or, conversely, its tendency to make critical mistakes when it sees a new situation it hasn't been trained to deal with.
  • Vulnerability to adversarial examples, a special kind of attack that can cripple certain kinds of AI systems, potentially making an autonomous car blind to a fast-approaching stop sign.
  • Bias, or whether a model's accuracy varies depending on the kind of person or thing it is evaluating.
  • Interpretability of a model's results, or how easy it is to understand why an AI system made a particular prediction.

"The machine learning model is just a tiny piece of the machine learning product," says Andrew Ng, founder of, a startup that helps companies set up AI processes.

  • "One of the challenges is that you ship a system and then the world changes in some weird and unpredictable way," says Ng, who previously started Google's and Baidu's AI operations.
  • "Anyone can download code from GitHub, but that's not the point," Ng tells Axios. "You need all these other things."

One of the byproducts of the hidden costs associated with increasingly accurate AI systems is that they can make it hard for a new startup or cash-strapped university lab to compete.

  • "The AI models that are being designed and developed don't always contemplate the application of the model," says Josh Elliot, director of AI at Booz Allen Hamilton.
  • Big companies offer pre-packaged AI programs — like one for detecting street signs, say — but users often lack the computing firepower required to tweak them.
  • That means the reigning players get to decide the important problems worth trying to solve with AI, Schwartz says, and everyone else has to go along.

What's next: Schwartz, Ng and others propose putting more effort toward solving problems more efficiently rather than making a slightly more accurate model at an enormous cost.

Go deeper: How not to replace humans

Go deeper

Updated 7 hours ago - Politics & Policy

Coronavirus dashboard

Illustration: Sarah Grillo/Axios

  1. Global: Total confirmed cases as of 7 p.m. ET: 12,859,834 — Total deaths: 567,123 — Total recoveries — 7,062,085Map.
  2. U.S.: Total confirmed cases as of 7 p.m. ET: 3,297,501— Total deaths: 135,155 — Total recoveries: 1,006,326 — Total tested: 40,282,176Map.
  3. States: Florida smashes single-day record for new coronavirus cases with over 15,000 — NYC reports zero coronavirus deaths for first time since pandemic hit.
  4. Public health: Ex-FDA chief projects "apex" of South's coronavirus curve in 2-3 weeks — Coronavirus testing czar: Lockdowns in hotspots "should be on the table"
  5. Education: Betsy DeVos says schools that don't reopen shouldn't get federal funds — Pelosi accuses Trump of "messing with the health of our children."

Scoop: How the White House is trying to trap leakers

Illustration: Sarah Grillo/Axios

President Trump's chief of staff, Mark Meadows, has told several White House staffers he's fed specific nuggets of information to suspected leakers to see if they pass them on to reporters — a trap that would confirm his suspicions. "Meadows told me he was doing that," said one former White House official. "I don't know if it ever worked."

Why it matters: This hunt for leakers has put some White House staffers on edge, with multiple officials telling Axios that Meadows has been unusually vocal about his tactics. So far, he's caught only one person, for a minor leak.

11 GOP congressional nominees support QAnon conspiracy

Lauren Boebert posing in her restaurant in Rifle, Colorado, on April 24. Photo: Emily Kask/AFP

At least 11 Republican congressional nominees have publicly supported or defended the QAnon conspiracy theory movement or some of its tenets — and more aligned with the movement may still find a way onto ballots this year.

Why it matters: Their progress shows how a fringe online forum built on unsubstantiated claims and flagged as a threat by the FBI is seeking a foothold in the U.S. political mainstream.