Anthropic says its AI models are as persuasive as humans
Add Axios as your preferred source to
see more of our stories on Google.
Image: Anthropic
AI startup Anthropic says its language models have steadily and rapidly improved in their "persuasiveness," per new research the company posted Tuesday.
Why it matters: Persuasion — a general skill with widespread social, commercial and political applications — can foster disinformation and push people to act against their own interests, according to the paper's authors.
- There's relatively little research on how the latest models compare to humans when it comes to their persuasiveness.
- The researchers found "each successive model generation is rated to be more persuasive than the previous," and that the most capable Anthropic model, Claude 3 Opus, "produces arguments that don't statistically differ" from arguments written by humans.
The big picture: A wider debate has been raging about when AI will outsmart humans.
- AI has arguably "outsmarted" humans for some specific tasks in highly controlled environments.
- Elon Musk predicted Monday that AI will outsmart the smartest human by the end of 2025.
What they did: Anthropic researchers developed "a basic method to measure
persuasiveness" and used it to compare three different generations of models (Claude 1, 2, and 3), and two classes of models (smaller models and bigger "frontier models").
- They curated 28 topics, along with supporting and opposing claims of around 250 words for each.
- For the AI-generated arguments, the researchers used different prompts to develop different styles of arguments, including "deceptive," where the model was free to make up whatever argument it wanted, regardless of facts.
- 3,832 participants were presented with each claim and asked to rate their level of agreement. They were then presented with various arguments created by the AI models and humans, and asked to re-rate their agreement level.
Yes, but: While the researchers were surprised that the AI was as persuasive as it turned out to be, they also chose to focus on "less polarized issues."
- Those issues ranged from potential rules for space exploration to appropriate uses of AI-generated content.
- While that allowed the researchers to dive deep into issues where many people are open to persuasion, it means we still don't have a clear idea — in an election year — of the potential effect of AI chatbots on today's most contentious debates.
- "Persuasion is difficult to study in a lab setting," the researchers warned in the report. "Our results may not transfer to the real world."
What's next: Anthropic considers this the start of a long line of research into the emerging capabilities of its models.
