Finish Line: Learn to coach your chatbot
Add Axios as your preferred source to
see more of our stories on Google.
Illustration: Brendan Lynch/Axios
If you're still using your chatbot like it's Google, stop. Stop it right now.
Why it matters: Generative AI is fundamentally different — and far more useful — when you treat it like a collaborator and check its work.
- Think of yourself like a coach and the AI like the best player on your team. Then run the play.
- You're not alone if you're leaving value on the table. Anthropic put out an AI Fluency Index on Monday that showed people aren't using its Claude tools to their full advantage.
- The company assessed "AI fluency" using a framework developed with professors Rick Dakan and Joseph Feller.
Stunning stat: "In only 30% of conversations do users tell Claude how they'd like it to interact with them," per the report.
- Train the bot with the old adage: "Help me help you."
What they did: Researchers defined 24 specific behaviors they believe exemplify safe, effective human-AI collaboration.
- Some of the behaviors include questioning an AI's response, identifying when the AI is missing context, and defining the audience for the output.
The big picture: Generative AI keeps getting smarter and producing magical-seeming outputs, from code to legal briefs to possible medical diagnoses.
Reality check: The more polished the output, the less users question it, the index shows.
Zoom in: Hallucinations haven't gone away. And while the top-tier models have improved on many of AI's early flaws, free versions often lag in reliability and performance.
- Wharton professor Ethan Mollick puts it bluntly: The free models aren't built for accuracy. They're fast and fun, but often less capable.
- People posting examples of "AI doing something stupid" are usually using free models, or they "have not selected a smarter model to work with," he writes in his blog.
Some best practices based on the report:
1. Keep prompting: Treat the first response as a draft. Iterate, ask follow-ups, push back and refine. Ongoing back-and-forth is the strongest marker of real fluency.
- Example: "You got that wrong. Try again" or "Why did you give that answer?"
2. Be skeptical of polished answers: When output looks great, pause. Check the facts, probe the reasoning and ask what might be missing. Don't lower your guard and drop your critical thinking.
- Example: Try your prompt with different chatbots and see if you get the same answer. Or ask: "Fact-check yourself and show me your sources."
3. Set the terms: Don't just prompt — instruct. Tell the AI to flag uncertainty or challenge your assumptions. Being explicit up front changes the dynamic.
- Example: Use global instructions and custom settings that most tools offer, so you don't need to rewrite the same rules every time.
The bottom line: AI fluency is core to your experience with the chatbot, no matter which one you pick. It's not always them; it's you.
