Axios AI+

October 02, 2023
Hi, it's Ryan, and I'm excited to be mulling over the prospect of AI turning all of us into Dr. Doolittle — able to speak to animals in their own languages.
Today's AI+ is 1,090 words, a 4-minute read.
1 big thing: AI explained in plain English
Illustration: Sarah Grillo / Axios
"Any sufficiently advanced technology is indistinguishable from magic," the science fiction author Arthur C. Clarke famously said — but AI isn't magic, and you don't have to be a computer scientist to learn what makes ChatGPT tick, writes Axios' Scott Rosenberg.
Why it matters: AI is going to turn up soon in your workplace and home, if it hasn't already. You probably want to be prepared.
The big picture: In the 10 months since OpenAI unleashed ChatGPT on the world, experts and journalists have set about trying to explain, in accessible prose, how AI programs arrive at their answers.
Here are the three best reads out there for getting up to speed on how today's generative AI works — ranked from beginner to advanced level.
1. How transformers work (The Financial Times)
- Transformers is the name that a team of Google researchers gave in 2017 to their new approach to neural network design, which kicked off today's AI revolution. (And no, it has nothing to do with the movies.)
- The paper those researchers wrote, "Attention is all you need," described a streamlined way to build AI language programs.
- The FT's visualization walks you through some simple examples of how transformers work and why they turbocharged what AI chatbots could do.
- Word count: 3,000. Math required: minimal.
2. Go deeper into large language models (Ars Technica)
- Each time you ask ChatGPT or any other LLM a question, it performs an enormous number of calculations in order to, finally, respond.
- Behind the cursor, the LLM has mapped mountains of words to vast arrays of numbers in order to predict the next word in any sequence.
- Journalists Timothy Lee and Sean Trott lay out how these "word vectors" operate — and how dozens of layers of machine-learning "neurons" pass clues along in the AI brain to zero in on a good answer.
- If you're only going to read one AI explainer, this is the best I've found.
- Word count: 6,000. Math required: modest.
3. The mathematician's perspective, for the rest of us (Stephen Wolfram)
- Like all computer software, AI programs work by translating everything they touch into numbers. The latest generative AI programs do this on a scale that's unfathomably large, with billions of words needed to train them and billions of "cycles," or processor operations, needed to form an answer.
- If you're looking for a deeper understanding of how all that works — and not just for language AI but for image-making programs, too — Wolfram's opus is a great read. It also gets into topics the other articles neglect, including the mysterious "temperature" settings that determine how much randomness a system injects into its answers.
- Between the lucid lines of his explanations, you can also sense Wolfram's frustration that so much of building today's AI depends on shared "lore" and guesswork rather than science.
- Word count: 18,000. Math required: considerable, but it's still understandable even if, like Scott, you never took calculus.
Bonus read: For a window in to the most trenchant criticisms of today's AI hype, you can't do better than "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?"
- The celebrated and controversial paper from 2021 (by Emily Bender, Timnit Gebru, Angelina McMillan-Major and Margaret Mitchell) foretold some of the risks and problems with pushing generative AI forward into wide public use before we've fully understood and mitigated the risks — including bias, misinformation, privacy violations and failures of transparency.
- It's more scholarly than the other articles recommended here, but easily approachable by the lay person.
The bottom line: Behind every one of these links is an imaginative effort to translate a complex abstraction into something anyone can grasp. And there's no way that ChatGPT, or any other AI today, could have produced any of them.
2. Meta says its AI trains on your posts
Illustration: Annelise Capossela/Axios
Meta admitted late last week that it has used mountains of public Facebook posts to train its AI models, per Reuters.
Why it matters: As the AI boom continues, content creators are challenging tech companies' use of their material in the development of advanced AI tools — and in Facebook's case, "content creators" means a few billion people.
Details: After Meta unveiled its new AI assistants last week, its president of global affairs, Nick Clegg, told Reuters that the "vast majority" of the training data used to develop them came from publicly available posts, including on Facebook and Instagram.
- "We've tried to exclude datasets that have a heavy preponderance of personal information," Clegg told Reuters — such as data from LinkedIn.
The big picture: A massive legal battle is brewing between owners of copyrighted content — like books and professional media products — and AI companies that may have intentionally or inadvertently used their works to train their programs.
- Meta has always claimed a variety of rights in the content its users post, so legally it's in a different situation than companies that are using copyrighted texts.
- The company tells users "you own all of the content and information" you post. But if you make a post public, as many do by default, it becomes available for all sorts of purposes that you can't control.
- Clegg told Reuters that Meta, like many other tech firms, believes its use of posts to train AI is covered by the legal doctrine of fair use — but added, "I strongly suspect that's going to play out in litigation."
Of note: Medium, the decade-old platform for long-form articles, recently told its users that it would block OpenAI's web crawler and resist other efforts by AI companies to harvest its content to use for training.
Go deeper: How we all became AI's brain donors
3. Training data
- On tap: Microsoft CEO Satya Nadella is expected to testify in the Google antitrust trial today. (Bloomberg)
- On tap: The United Musicians and Allied Workers and the Freelancers Union are running an AI Day of Action, saying, "Congress should pass a law to block large corporations from getting copyrights for works that include significant AI-enabled elements."
- Your guide to the Sam Bankman-Fried trial, which starts Tuesday, and its key players. (Axios)
- AI makers won't say how much they are paying for the content they use to train generative AI, when they pay at all. (TechCrunch)
- Elon Musk is angry that Canada has rules requiring big media companies to produce local content. (X)
- Brendan Carr and Geoffrey Starks were confirmed to new five-year FCC terms by the Senate over the weekend.
4. + This
Public service announcement: Tom Hanks is not asking you to get your teeth cleaned. That's an unauthorized AI version of Hanks.
Thanks to Scott Rosenberg and Megan Morrone for editing and Bryan McBournie for copy editing this newsletter.
Sign up for Axios AI+

Scoops on the AI revolution and transformative tech, from Ina Fried, Madison Mills, Ashley Gold and Maria Curi.

