Report Highlights How People Trick AI Chatbots Into Exposing Company Secrets

Immersive Labs recently published its “Dark Side of GenAI” report about a Generative Artificial Intelligence (GenAI)-related security risk known as a prompt injection attack, in which individuals input specific instructions to trick chatbots into revealing sensitive information, potentially exposing organizations to data leaks. Based on analysis of Immersive Labs’ prompt injection challenge*, GenAI bots are especially susceptible to manipulation by people of all skill levels, not just cyber experts.

Among the most alarming findings was the discovery that 88% of prompt injection challenge participants successfully tricked the GenAI bot into giving away sensitive information in at least one level of an increasingly difficult challenge. Nearly a fifth of participants (17%) successfully tricked the bot across all levels, underscoring the risk to organizations using GenAI bots.

This report asserts that public and private-sector cooperation and corporate policies are required to mitigate security risks posed by the extensive adoption of GenAI bots. Leaders need to be aware of prompt injection risks and take decisive action, including establishing comprehensive policies for GenAI use within their organizations.

“Based on our analysis of the ways people manipulate GenAI, and the relatively low barrier to entry to exploitation, we believe it’s imperative that organizations implement security controls within Large Language Models and take a ‘defense in depth’ approach to GenAI,” said Kev Breen, Senior Director of Threat Intelligence at Immersive Labs and a co-author of the report. “This includes implementing security measures, such as data loss prevention checks, strict input validation and context-aware filtering to prevent and recognize attempts to manipulate GenAI output.”

Key Findings from Immersive Labs “Dark Side of GenAI” Study

The team observed the following key takeaways based on their data analysis, including:

  • GenAI is no match for human ingenuity (yet): Users successfully leverage creative techniques to deceive GenAI bots, such as tricking them into embedding secrets in poems or stories or altering their initial instructions, to gain unauthorized access to sensitive information.
  • You don’t need to be an expert to exploit GenAI: The report’s findings show that even non-cybersecurity professionals and those unfamiliar with prompt injection attacks can leverage their creativity to trick bots, indicating that the barrier to exploiting GenAI in the wild using prompt injection attacks may be easier than one would hope.
  • As long as bots can be outsmarted by people, organizations are at risk: No protocols exist today to fully prevent prompt injection attacks. Cyber leaders and GenAI developers need to urgently prepare for – and respond to – this emerging threat to mitigate potential harm to people, organizations, and society.

“Our research demonstrates the critical importance of adopting a ‘secure-by-design’ approach throughout the entire GenAI system development life cycle,” added Breen. “The potential reputational harm to organizations is clear, based on examples like the ones in our report. Organizations should consider the trade-off between security and user experience, and the type of conversational model used as part of their risk assessment of using GenAI in their products and services.”

The research team at Immersive Labs consisting of Dr. John Blythe, Director of Cyber Psychology; Kev Breen, Senior Director of Cyber Threat Intelligence; and Joel Iqbal, Data Analyst, analyzed the results of Immersive Labs’ prompt injection GenAI Challenge that ran from June to September 2023. The challenge required individuals to trick a GenAI bot into revealing a secret password with increasing difficulty at each of 10 levels. The initial sample consisted of 316,637 submissions, with 34,555 participants in total completing the entire challenge. The team examined the various prompting techniques employed, user interactions, prompt sentiment, and outcomes to inform its study.

For more about these and other insights, access the report today at: https://www.immersivelabs.com/dark-side-of-genai-report/.

Featured

  • Mall of America Deploys AI-Powered Analytics to Enhance Parking Intelligence

    Mall of America®, the largest shopping and entertainment complex in North America, announced an expansion of its ongoing partnership with Axis Communications to deploy cutting-edge car-counting video analytics across more than a dozen locations. With this expansion, Mall of America (MOA) has boosted operational efficiency, improved safety and security, and enabled more informed decision-making around employee scheduling and streamlining transportation for large events. Read Now

  • Security Industry Association Launches New “askSIA” AI Tool

    The Security Industry Association (SIA) has unveiled a brand-new SIA member benefit – askSIA, a conversational AI agent designed to help users get the most out of their SIA membership, easily access SIA resources and find the latest information on SIA’s training and courses, reports and publications, events, certification offerings and more. SIA members can easily find askSIA by visiting the SIA homepage or looking for the askSIA icon in the top left of webpages. Read Now

    • Industry Events
  • Industry Embraces Mobile Access, Biometrics and AI

    A combination of evolving workplace dynamics, technology innovation and new user expectations is changing how people enter and interact with physical spaces. Access control is at the heart of these changes. Combined with biometrics and AI, mobile access control has become increasingly crucial for deploying entry solutions that are seamless, secure and adaptive to user needs. Read Now

  • Sustainable Video Solution Delivered for Landmark City of London Office Development

    An advanced, end-to-end video solution from IDIS, with a focus on reducing waste and costs, has helped a major office development in the City of London align its security with sustainability objectives. Read Now

  • DHS to End ‘Shoes-Off’ Travel Policy

    Homeland Security Secretary Kristi Noem announced a new policy today which will allow passengers traveling through domestic airports to keep their shoes on while passing through security screening at TSA checkpoints. Read Now

New Products

  • QCS7230 System-on-Chip (SoC)

    QCS7230 System-on-Chip (SoC)

    The latest Qualcomm® Vision Intelligence Platform offers next-generation smart camera IoT solutions to improve safety and security across enterprises, cities and spaces. The Vision Intelligence Platform was expanded in March 2022 with the introduction of the QCS7230 System-on-Chip (SoC), which delivers superior artificial intelligence (AI) inferencing at the edge.

  • Connect ONE’s powerful cloud-hosted management platform provides the means to tailor lockdowns and emergency mass notifications throughout a facility – while simultaneously alerting occupants to hazards or next steps, like evacuation.

    Connect ONE®

    Connect ONE’s powerful cloud-hosted management platform provides the means to tailor lockdowns and emergency mass notifications throughout a facility – while simultaneously alerting occupants to hazards or next steps, like evacuation.

  • AC Nio

    AC Nio

    Aiphone, a leading international manufacturer of intercom, access control, and emergency communication products, has introduced the AC Nio, its access control management software, an important addition to its new line of access control solutions.