Creative Commons

Leading artificial intelligence researchers are debating whether machines — once they achieve super-human smarts — should be expected to explain themselves in robust discussion, or if spitting out an exceptional, if mysterious, answer is good enough.

The debate is part of the raging global discussion over a perceived challenge posed by machine intelligence to jobs, the human role in society, geopolitical power — to us.

Battle lines: Peter Norvig, head of research at AI powerhouse Google, has attracted attention with a June 22 speech in Sydney arguing that future intelligent computers shouldn't be expected to explain why they think what they think, since, after all, humans themselves don't do a very good job at explaining the true reasons for their decisions. Others argue that neural networks are simply a different way of thinking, and that even if we should be able to plumb the machine mind, there is no way for us to get at it.

But David Ferrucci, who ran IBM's Watson project for the 2011 Jeopardy challenge and now runs an AI company called Elemental Cognition, asserts that machines have to defend themselves. "I need an explanation. I wanted to be able to critique it," he tells Axios.

Why it matters: Machines that don't explain themselves scare people who worry about godlike computer overlords. But researchers like Ferrucci are posing a more fundamental question: Are we ourselves willing to be held accountable for decisions we base on those of a computer? Unless he knows the computer's rationale, Ferrucci doesn't want to gamble. And others agree: Darpa, the Pentagon's radical research lab, is studying what it calls "Explainable Artificial Intelligence," giving it the acronym XAI.

Let's talk about it: The debate played out on stage at the O'Reilly Artificial Intelligence Conference this week in New York. Ferrucci argued that the best way to get to the kind of AI he is thinking of — the kind you can have a normal exchange of opinions with — is to study and develop a computer that thinks like a human. "[But] there is a level to those dialogues that we are nowhere close to having," he said.

The future: Josh Tennenbaum, a professor at MIT, said the answer is to study young children and develop machine intelligence that learns and thinks like them. He played a video of an 18-month-old child who, confronted with an adult having trouble putting books into a closed cabinet, spontaneously opens its doors and watches to make sure the troubled man succeeds. "That's the heart of intelligence right there. And I think that's the grand challenge for AI," he said. "...If we could reverse engineer what's going on in that kid's mind, just think what we can do with robots and other machines that could really help us out."

The Amazon effect: At this stage commercially, many of us are using Alexa or technology like it at home, but Kris Hammond, chief scientist at Narrative Science, said, "Alexa understands words. It doesn't understand language." Alexa can hold only one idea at a time. If you ask a followup, it does not remember what was just said. "You need to be able to challenge," Hammond said. "Otherwise, we would only be listening."

Go deeper

McMaster: Trump's peaceful transition comments are a "gift to our adversaries"

President Trump's refusal to commit to a peaceful transfer of power if he loses November's presidential election is a "gift to our adversaries," Trump's former national security adviser H.R. McMaster said Sunday.

The big picture: McMaster, a retired three-star general, said that the American people must understand that the military will have "no role" in a presidential transition, and that it's "irresponsible" to even talk about it as a possibility.

Trump says Supreme Court ending Obamacare would be "a big WIN"

Photo: Spencer Platt/Getty Images

President Trump on Sunday tweeted that the Supreme Court invalidating the Affordable Care Act would be "a big WIN for the USA!"

Why it matters: Democrats have argued that confirming a Trump-appointed justice to the Supreme Court would put the Affordable Care Act, which protects pre-existing conditions, in jeopardy. Trump's Supreme Court pick, Amy Coney Barrett, has written that she disagreed with Chief Justice John Roberts when he ruled to uphold the law.

Trump sees court fight as virus respite

Spotted at Trump's rally last night at Harrisburg International Airport in Middletown, Pa. Photo: Joshua Roberts/Reuters

At a rally in Pennsylvania last night, President Trump basked in adulation for Judge Amy Coney Barrett and said, "She should be running for president!"

Why it matters: She might as well be. The Trump campaign is thrilled to be talking about something besides the president's handling of COVID, and it's going all-in to amp up the court conversation.