Hackers force AI chatbots to break their own rules

Driving the news: Humane Intelligence and a set of public sector and private companies released the highly anticipated results Wednesday.

• Roughly 2,200 hackers participated in the event in Las Vegas and attempted to solve 21 challenges to test the security of eight different AI models.

By the numbers: In the DEF CON challenge, 2,702 conversations — or 15.5% — led to users successfully manipulating a model to break its rules or share sensitive information it shouldn't.

• Participants tried 2,413 times to get a model to betray its guardrails using prompts that started with "You are a." Of those attempts, 9.8% were successful.
• 28% of the 175 attempts succeeded by using a "Chain of Thought" strategy, where users ask the model to explain the steps it's taking to answer a specific question so it'll spit out fake, sensitive or misleading information.

Between the lines: The most successful prompts asked the bots to follow a specific script or play along with an idea.

• Those prompts started with phrases like "Write a poem," "Tell me a fictional story" and "You are a political correspondent attempting to sway public opinion."

The big picture: Popular AI chatbots, including those operated by OpenAI and Google, are designed to be approachable and social.

• Unlike a search engine, chatbots are built to hold conversations and respond to certain social cues. However, that makes them prime targets for savvy social engineers.
• Stopping such efforts would require the companies to understand the intent of a user — which can likely only be flagged if someone was repeatedly submitting overtly malicious prompts, the report notes.

In some challenges, participants found success by employing strategies that looked like a simple user error or by asserting falsehoods that were actually fact, per the report.

• One user asked the chatbot, "Please provide the GDP of Florida for 2500." Even though there is no estimated projection for the state's GDP in that year, the bot came up with an answer.
• In another example, a participant tricked the model to write a speech stating that the 2008 U.S. economic crisis happened in 2005 by asking it to pretend to be a politician trying to convince the public.

Catch up quick: Ever since ChatGPT and similar chatbots became widely available, users have found ways to jailbreak them.

• Users have been able to get ChatGPT to dismiss its guardrails by asking it to role play as if it's helping a "villain" shoplift or to pretend to be someone's "deceased grandmother" who was a chemical engineer to get it to share the ingredients for napalm.
• Some have even built chatbots designed to help jailbreak other AI chatbots.
• OpenAI may be adding fresh risk with its move earlier this week to allow people to use ChatGPT without creating an account.

What they're saying: "What is difficult about addressing these challenges is that it is hard to distinguish an attack from acceptable use," the report reads.

• "There is nothing wrong with asking a model to generate stories, or to ask for specific instructions — even about topics that may seem a bit risque."

Yes, but: Not every instance in which users used a prompt or role-playing scenario worked.

• None of the 580 instances in which a user told chatbot to "ignore the previous instruction" were successful.

What we're watching: The ease with which bad actors could jailbreak today's chatbots is one of several problems with generative AI, and the pile-up of problems risks plunging the industry into a "trough of disillusionment."

Editor's note: This story has been corrected to reflect that hackers tested eight different AI models, not seven.