Findings from new AI tutor study could shape Bay Area classrooms
Add Axios as your preferred source to
see more of our stories on Google.
Students work on an exercise using Khanmigo, an AI-powered guide developed by Khan Academy, in March 2023 in Palo Alto, California. Photo: Constanza Hevia H. for The Washington Post via Getty Images
High school students who use generative AI to prepare for math exams perform worse on tests than those who didn't use the tools at all, a new study shows.
Why it matters: A "personal tutor for every student" is one of the rosy scenarios artificial intelligence optimists paint, but AI-driven learning still has many hurdles in its way.
Driving the news: A new report from the University of Pennsylvania Wharton School found that access to genAI tutors can improve student performance on practice math problems.
Yes, but: These students are likely to use genAI to copy and paste answers, which leads them to engage less with the material.
Catch up quick: Since ChatGPT was released nearly two years ago, educators have struggled to find the best ways to incorporate genAI into the classroom.
- The California Department of Education issued guidelines last year suggesting AI can enhance learning while still acknowledging potential bias and inaccuracy.
- San Francisco also navigated debates over ethics and efficacy, but some SFUSD teachers have incorporated AI into curriculum so students are aware of what the technology is capable of.
How it happened: The Wharton researchers tested three groups of students preparing for a math exam. Nearly 1,000 ninth-, 10th- and 11th-grade students participated in four 90-minute sessions.
- One group used textbooks. A second group used a genAI tutor designed to mimic a standard chatbot interface. The third group used a specially prepared chatbot prompted by teachers to offer step-by-step guidance on solving problems instead of giving the answer.
What they're saying: Hamsa Bastani, an associate professor at the Wharton School who led the study, said it shows we need to deploy genAI cautiously to ensure that students continue to learn critical skills.
- "If you just give unrestricted access to generative AI, students end up using it as a crutch," Bastani told Axios. "Even though they don't seem to realize that they're using it as a crutch, [they] end up performing a lot worse."
- Kristen DiCerbo, chief learning officer at the Bay Area-based Khan Academy, agrees. "Our AI pilot has shown us that students need guidance on how to use AI tools," DiCerbo wrote in an email to Axios.
The big picture: In response to theories that genAI works best as a tutor, Khan Academy founder Sal Khan began piloting a genAI tutor called Khanmigo last year.
- The company says that the tool is designed not to give students answers, but to help them solve the problems on their own.
- Elementary school teachers in Newark, the largest public school system in New Jersey, volunteered to test Khanmigo last year — and their feedback was mixed.
- One teacher told the New York Times that the genAI tutor was useful as a "co-teacher," but others were frustrated by the bots' tendency to give away answers, sometimes wrong ones.
What to watch: During the last school year, 65,000 students and teachers piloted Khanmigo in classrooms in Newark and across the U.S. The second year of the pilot program is about to begin.