Study: ChatGPT's creativity gap
Add Axios as your preferred source to
see more of our stories on Google.
Illustration: Lindsey Bailey/Axios
AI can generate a larger volume of creative ideas than any human, but those ideas are too much alike, according to research newly published in Nature Human Behaviour.
Why it matters: AI makers say their tools are "great for brainstorming," but experts find that chatbots produce a more limited range of ideas than a group of humans.
How it works: Study participants were asked to brainstorm product ideas for a toy involving a brick and a fan, using either ChatGPT, their own ideas, or their ideas combined with web searches.
- Ninety-four percent of ideas from those who used ChatGPT "shared overlapping concepts."
- Participants who used their own ideas with the help of web searches produced the most "unique concepts," meaning a group of one or more ideas that did not overlap with any other ideas in the set.
- Researchers used GPT-3.5 and GPT-4 and reported that while GPT-4 is creating more diverse ideas than 3.5, it still falls short ("by a lot") relative to humans.
Case in point: Nine participants using ChatGPT independently named their toy "Build-a-Breeze Castle."
The big picture: Wharton professors Gideon Nave and Christian Terwiesch and Wharton researcher Lennart Meincke found that subjects came up with a broader range of creative ideas when they used their own thoughts and web searches, compared with when they used ChatGPT.
- Groups that used ChatGPT tended to converge on similar concepts, reducing overall idea diversity.
- "We're not talking about diversity as a DEI type of diversity," Terwiesch told Axios. "We're talking about diversity in terms of the ideas being different from each other ... like in biology, we need a diverse ecosystem."
Zoom in: A 2024 study found similar results.
- Participants were asked to write short fiction with and without ChatGPT.
- Generative AI–enabled stories were found to be more similar to each other than stories by humans.
Yes, but: ChatGPT can be used as part of the brainstorming process. Terwiesch says idea variance comes from using ChatGPT to generate ideas, while also coming up with your own ideas and collecting original ideas from others.
- Terwiesch also recommends "chain of thought prompting," which means asking your chatbot to generate several ideas, but also specifically asking the bot to make those ideas different from each other.
- "If I just sit back and let ChatGPT do the work, I'm not taking the full advantage of what this tool has to offer. I can do better than that," Terwiesch told Axios.
- A spokesperson from OpenAI shared best practices for prompting ChatGPT, advice from writers on how to use the tool and a student's guide to writing with ChatGPT.
Our thought bubble: Teaching the next generation to use genAI to boost their creativity could help schools balance AI literacy with concerns about using ChatGPT to "cheat."
The bottom line: AI makers say their tools will eventually become as intelligent and creative as humans, but genAI outputs will always be more narrow than humans brainstorming together.
