Too much computer
Add Axios as your preferred source to
see more of our stories on Google.
Illustration: Rebecca Zisser/Axios
Somewhere in the human-computer nexus, two brand-new Boeing aircraft have crashed shortly after takeoff, with the loss of 346 lives. One conclusion: We can't assume that adding more technology to our lives will make us safer.
- We saw something similar when a driverless car killed a pedestrian in Tempe, Arizona.
Why it matters: The way that Boeing and the FAA approach the latest crisis should provide a blueprint not only for the aviation industry, but also for autonomous vehicles and many other technologies taking over human decision-making — even the algorithm determining what you watch next on YouTube.
Our thought bubble, from Axios editor-in-chief Nicholas Johnston: Too much computer can be anodyne, like Waze directing me off a highway when it erroneously thinks there's a jam up ahead. But it can also be a Boeing flying into the ground because it's confused about a stall.
What we don’t know: The big unknowns in aviation right now:
- What caused Ethiopian Airlines flight 302 to crash on Sunday, killing all 157 people on board?
- Is the 737 MAX safe?
- Why didn't Boeing immediately ground the 737 MAX until it found out the answer to the first two questions?
The answers to the first two questions are known unknowns. We know that we don't know the answer; we can also be reasonably certain that an answer is forthcoming. The black box will be found, the aircraft's data will be retrieved (somewhere), and we will find out what went wrong.
The answer to the third question is harder, given that a safety-first reputation is the most valuable asset that Boeing possesses.
- What we do know: Pilots have registered complaints about the 737 MAX in a federal database, and they've criticized the training too. The Dallas Morning News reports that one captain called the flight manual "inadequate and almost criminally insufficient."
- By the numbers: Until we find out what went wrong, we have two simple fatality statistics to guide us. Air travel in general is incredibly safe, thanks to technology, but air travel on the 737 MAX is not. The plane has been flying for less than two years, and it has already crashed twice. That's unprecedented in modern aviation history.
Be smart: The proof of the pudding — the reason why we know that technology has made air travel safer — is right there in the statistics. You don't need to know anything about ground proximity warning systems or wind shear alerts to know that planes have been getting safer, you can just look at the numbers.
It was reasonable to believe that the 737 MAX was safe before it crashed twice. All modern aircraft are safe. But then that assumption was tested in real-world conditions, and now it must be re-examined in the light of new evidence.
- Boeing said it has "full confidence in the safety of the 737 MAX" — before it has seen the data from the Ethiopian Airways black box, but after 346 people have died. That statement is going to make it very hard for the public and regulators to trust the company's safety statements in future.
- The New York Times reports that the FAA has outsourced safety designations on Boeing aircraft to Boeing's own employees, in a textbook case of regulatory capture.
This is bigger than just aviation. Technology has made cars, too, much safer over time. But it's dangerous to mindlessly extrapolate that tendency into the future.
Like aircraft manufacturers, carmakers design redundant sensors and fail-safe technology into their vehicles to maximize safety, Axios' Joann Muller reports.
- Automated vehicles need human monitoring, just as planes do. But without proper training and education, drivers — or pilots— are unprepared to deal with situations where something goes wrong.
- Mark Rosekind, chief safety innovation officer at Zoox, an autonomous vehicle startup, told Joann the crash sounds like another example of “mode confusion” – where the pilot (or the driver, in the case of an AV) doesn’t understand what the machine is doing and is “literally battling the computer in the cockpit.”
- "We’re adding complexity and technology to make it do its job better, but if you don’t design around what the human is capable of doing, and their limitations, it ends up creating more risk,” said Rosekind, who's also a former administrator of the National Highway Traffic Safety Administration.
The bottom line: When evidence emerges that a technology is not safe, that's the time to re-examine a lot of priors about technology making us safer. That goes for planes, it goes for cars, and it even goes for social networks.
Go deeper:
