Stories

Revenge of the deepfake detectives

Illustration of a police line-up with binary code over the people's faces
Illustration: Sarah Grillo/Axios

Tech giants, startups and academic labs are pumping out datasets and detectors in hopes of jump-starting the effort to create an automated system that can separate real videos, images and voice recordings from AI forgeries.

Why it matters: Algorithms that try to detect deepfakes lag behind the technology that creates them — a worrying imbalance given the technology's potential to stir chaos in an election or an IPO.

Driving the news: Dessa, the AI company behind the hyper-convincing fake Joe Rogan voice from earlier this summer, published a tool today for detecting deepfake audio — the kind that recently scammed a CEO out of $240,000.

  • The new detector, which Axios is reporting first, is open source, so anybody can go through the code for free to understand and potentially improve it.
  • But the company gets something out of it: The detector is built on a Dessa platform, which you have to download (without paying) to set it up.

The big picture: There's an all-hands scramble for better detectors, which generally require a lot of really good examples of deepfakes. Researchers use them to train algorithms that can tell if media was created by AI.

  • Yesterday, SUNY Albany deepfake expert Siwei Lyu released a dataset filed with celebrity deepfakes.
  • Earlier in the week, Google and Jigsaw — both owned by parent company Alphabet — released a large set of video deepfakes.
  • And earlier this month, Facebook, Microsoft and the Partnership on AI teamed up with academic researchers to release more deepfake videos — and offer a prize to the team that uses them to make the best detector.

Unlike these datasets, which allow researchers to cook up their own detectors, Dessa is releasing a pre-baked system — which has advantages and risks.

  • The company felt a responsibility to release an antidote after it made the realistic Rogan voice, says Ragavan Thurairatnam, Dessa's co-founder.
  • "I think it's inevitable that malicious actors are going to move much faster than those who want to stop it," he tells Axios. The free detector is a "starting point" for people to push detection forward.

But, but, but: Thurairatnam acknowledged that an open-source detector could help a particularly determined troll create new audio fakes that fool it. That's because generative AI systems can be trained to trick a specific detector.

  • He argues that the potential for creating better detectors outweighs the probability that someone will misuse Dessa's code.
  • But Lyu of SUNY Albany says there's some reason to worry. "In principle, such code will help both but probably more for making better generators."

Go deeper: Researchers struggle with containing potentially harmful AI

Deepfakes