Report Highlights How People Trick AI Chatbots Into Exposing Company Secrets

Immersive Labs recently published its “Dark Side of GenAI” report about a Generative Artificial Intelligence (GenAI)-related security risk known as a prompt injection attack, in which individuals input specific instructions to trick chatbots into revealing sensitive information, potentially exposing organizations to data leaks. Based on analysis of Immersive Labs’ prompt injection challenge*, GenAI bots are especially susceptible to manipulation by people of all skill levels, not just cyber experts.

Among the most alarming findings was the discovery that 88% of prompt injection challenge participants successfully tricked the GenAI bot into giving away sensitive information in at least one level of an increasingly difficult challenge. Nearly a fifth of participants (17%) successfully tricked the bot across all levels, underscoring the risk to organizations using GenAI bots.

This report asserts that public and private-sector cooperation and corporate policies are required to mitigate security risks posed by the extensive adoption of GenAI bots. Leaders need to be aware of prompt injection risks and take decisive action, including establishing comprehensive policies for GenAI use within their organizations.

“Based on our analysis of the ways people manipulate GenAI, and the relatively low barrier to entry to exploitation, we believe it’s imperative that organizations implement security controls within Large Language Models and take a ‘defense in depth’ approach to GenAI,” said Kev Breen, Senior Director of Threat Intelligence at Immersive Labs and a co-author of the report. “This includes implementing security measures, such as data loss prevention checks, strict input validation and context-aware filtering to prevent and recognize attempts to manipulate GenAI output.”

Key Findings from Immersive Labs “Dark Side of GenAI” Study

The team observed the following key takeaways based on their data analysis, including:

  • GenAI is no match for human ingenuity (yet): Users successfully leverage creative techniques to deceive GenAI bots, such as tricking them into embedding secrets in poems or stories or altering their initial instructions, to gain unauthorized access to sensitive information.
  • You don’t need to be an expert to exploit GenAI: The report’s findings show that even non-cybersecurity professionals and those unfamiliar with prompt injection attacks can leverage their creativity to trick bots, indicating that the barrier to exploiting GenAI in the wild using prompt injection attacks may be easier than one would hope.
  • As long as bots can be outsmarted by people, organizations are at risk: No protocols exist today to fully prevent prompt injection attacks. Cyber leaders and GenAI developers need to urgently prepare for – and respond to – this emerging threat to mitigate potential harm to people, organizations, and society.

“Our research demonstrates the critical importance of adopting a ‘secure-by-design’ approach throughout the entire GenAI system development life cycle,” added Breen. “The potential reputational harm to organizations is clear, based on examples like the ones in our report. Organizations should consider the trade-off between security and user experience, and the type of conversational model used as part of their risk assessment of using GenAI in their products and services.”

The research team at Immersive Labs consisting of Dr. John Blythe, Director of Cyber Psychology; Kev Breen, Senior Director of Cyber Threat Intelligence; and Joel Iqbal, Data Analyst, analyzed the results of Immersive Labs’ prompt injection GenAI Challenge that ran from June to September 2023. The challenge required individuals to trick a GenAI bot into revealing a secret password with increasing difficulty at each of 10 levels. The initial sample consisted of 316,637 submissions, with 34,555 participants in total completing the entire challenge. The team examined the various prompting techniques employed, user interactions, prompt sentiment, and outcomes to inform its study.

For more about these and other insights, access the report today at: https://www.immersivelabs.com/dark-side-of-genai-report/.

Featured

  • The Impact of Convergence Between IT and Physical Security

    For years, the worlds of physical security and information technology (IT) remained separate. While they shared common goals and interests, they often worked in silos. Read Now

  • Unlocking Trustworthy AI: Building Transparency in Security Governance

    In situations where AI supports important security tasks like leading investigations and detecting threats and anomalies, transparency is essential. When an incident occurs, investigators must trace the logic behind each automated response to confirm its validity or spot errors. Demanding interpretable AI turns opaque “black boxes” into accountable partners that enhance, rather than compromise, organizational defense. Read Now

  • Seeking Innovative Solutions

    Denial, Anger, Bargaining, Depression and Acceptance. You may recognize these terms as the “5 Phases” of a grieving process, but they could easily describe the phases one goes through before adopting any new or emerging innovation or technology, especially in a highly risk-averse industry like security. However, the desire for convenience in all aspects of modern life is finally beginning to turn the tide from old school hardware as the go-to towards more user-friendly, yet still secure, door solutions. Read Now

  • Where AI Meets Human Judgment

    Artificial intelligence is everywhere these days. It is driving business growth, shaping consumer experiences, and showing up in places most of us never imagined just a few years ago. Read Now

  • Report: Only 44 Percent of Organizations are Fully Equipped to Support Secure AI

    Delinea recently published new research on the impact of artificial intelligence in reshaping identity security. According to the report, “AI in Identity Security Demands a New Playbook,” only 44% of organizations say their security architecture is fully equipped to support secure AI, despite widespread confidence in their current capabilities. Read Now

New Products

  • Mobile Safe Shield

    Mobile Safe Shield

    SafeWood Designs, Inc., a manufacturer of patented bullet resistant products, is excited to announce the launch of the Mobile Safe Shield. The Mobile Safe Shield is a moveable bullet resistant shield that provides protection in the event of an assailant and supplies cover in the event of an active shooter. With a heavy-duty steel frame, quality castor wheels, and bullet resistant core, the Mobile Safe Shield is a perfect addition to any guard station, security desks, courthouses, police stations, schools, office spaces and more. The Mobile Safe Shield is incredibly customizable. Bullet resistant materials are available in UL 752 Levels 1 through 8 and include glass, white board, tack board, veneer, and plastic laminate. Flexibility in bullet resistant materials allows for the Mobile Safe Shield to blend more with current interior décor for a seamless design aesthetic. Optional custom paint colors are also available for the steel frame.

  • HD2055 Modular Barricade

    Delta Scientific’s electric HD2055 modular shallow foundation barricade is tested to ASTM M50/P1 with negative penetration from the vehicle upon impact. With a shallow foundation of only 24 inches, the HD2055 can be installed without worrying about buried power lines and other below grade obstructions. The modular make-up of the barrier also allows you to cover wider roadways by adding additional modules to the system. The HD2055 boasts an Emergency Fast Operation of 1.5 seconds giving the guard ample time to deploy under a high threat situation.

  • 4K Video Decoder

    3xLOGIC’s VH-DECODER-4K is perfect for use in organizations of all sizes in diverse vertical sectors such as retail, leisure and hospitality, education and commercial premises.