New tool will scan AI models for malware
Add Axios as your preferred source to
see more of our stories on Google.
Illustration: Maura Losch/Axios
An AI security startup is launching a tool today designed to keep hackers out of increasingly popular open source AI models.
Why it matters: Hackers have gotten better at hiding malware inside the open source, foundational AI models available on platforms like Hugging Face — leaving companies that lack the resources to build their own proprietary models vulnerable to cyberattacks.
Driving the news: Protect AI, an AI security startup founded in 2022, is rolling out its new Guardian scanning tool for companies today.
- The product builds on the company's existing, open-source tool, ModelScan, to scan models before they hit a companies' networks and detect any hidden trojan malware.
How it works: Similar to the way virus scanning works, Guardian acts as an intermediary. Before a user downloads the AI model, Guardian will scan it for signs of tampering — including certain file formats and functions that can show signs of malware.
- Guardian also scans to see if the model meets a companies' specific internal AI policies, including those on data collection and use cases.
- If it is flagged, Guardian will halt the download and share details about the problems found.
Of note: The company plans to leverage Huntr, the AI-focused bug bounty program it acquired in August, to help inform what vulnerabilities Guardian will scan for.
By the numbers: Protect AI researchers found 3,354 models on Hugging Face since August that had malicious code, according to a blog post published alongside today's product announcement.
- Of those, 1,347 models were not marked as "unsafe" by Hugging Face's security scans, per the blog post.
What they're saying: "Companies are freely and blindly downloading, using these models, but they can contain things that can execute some pretty bad functions to steal data or take over systems," Ian Swanson, CEO and co-founder of Protect AI, told Axios.
The big picture: Creating an AI model requires terabytes of data and millions of dollars that not every company has — leaving many organizations to rely on open source, foundational models if they want to use AI.
- Many of these widely available tools are available on repositories like Hugging Face, but these repositories don't conduct deep enough security scans to catch all the ways hackers may have messed with a model, Swanson said.
- And even if a repository detects a security issue, they don't typically remove a model, leaving it available to download, he added.
Go deeper: Hackers explore ways to misuse AI in major security test
