Claude AI aces Penn Law exams, outperforming top students
Add Axios as your preferred source to
see more of our stories on Google.

Illustration: Sarah Grillo/Axios. Stock: Getty Images
Penn Law students hoping to graduate summa cum laude have new competition: Claude.
Why it matters: Law schools may need to radically rethink how they test students after an experiment at the Ivy League school found that AI outperformed many aspiring lawyers on final exams.
What's more shocking: Penn already admits some of the country's brightest law students, with median LSAT scores in the mid-170s for its latest incoming class.
- Its alumni include the late Sen. Arlen Specter and former Philly Mayor Ed Rendell.
State of play: It's a tough balancing act. Some schools are banning AI to preserve traditional methods of legal reasoning, while others, like Penn, are leaving it up to professors to decide how students can use the technology in their classrooms.
Threat level: The experiment also underscored a potential problem with outright AI bans: Shadow use, or using AI in unauthorized ways, can be hard to detect.
- Case in point: Professors in the Penn Carey Law study who graded AI-generated exams sometimes had trouble detecting the bot's handiwork.
- When they did spot it, it was through tells such as punctuation quirks or formatting glitches.
Zoom in: One professor called an AI-generated submission the "best of the lot" in a constitutional law class of 92 students.
How it worked: Claude sat for 11 exams administered to students during the spring 2026 semester, ranging from constitutional law to intellectual property.
- Penn used Anthropic's Claude Opus 4.7, the latest version available at the time of the experiment.
- Tests were administered twice: once with AI that had been explicitly trained on course materials and then again with AI that hadn't been trained, testing the model's baseline knowledge.
- AI-generated submissions were then slipped into the stack with regular students' work, and professors were told to grade according to their normal rubrics.
What they found: The results were mixed. On multiple-choice sections, the AI model performed much better when given course materials, in one case improving its score by 40 percentage points, per results reported by Penn Carey Law professor Polk Wagner.
- The additional materials "barely moved the needle" on essays, Wagner wrote, and in one case made AI's performance worse.
- The benefits were a wash on tests with multiple-choice and written sections.
AI passed all 10 final exams that were graded, completing each in just eight minutes.
- It scored in the top 10% of the class on seven and aced a legislation exam, outscoring all 84 real-life students who took the same test.
The intrigue: Humans still have one big advantage over the machine: creativity.
- One test-taker wove a mock judicial opinion into an essay in a "witty and structurally daring" way that the professor preferred over the AI.
- Wagner writes: "The machine reaches the ceiling of the competent comprehensive answer; the heights above it, where originality and judgment live, it did not reach."
Yes, but: That could change as AI rapidly improves, Wagner tells Axios.
What they're saying: Take-home tests were still the norm in his class a few semesters ago because Wagner says he didn't believe AI was sophisticated enough to give students an edge.
- Nowadays, he says more professors are banning laptops in the classroom and giving more closed-book, in-person exams to limit AI's reach.
The bottom line: Wagner's advice to future law students: Get really good at thinking for yourself and developing the muscle memory to know which arguments are most persuasive to humans.
- "So much of law is really about making choices," he says, referring to judgment calls prosecutors make every day that impact people's lives.
What we're watching: How the latest AI models perform when Wagner runs the next round of his experiment.
