First LLM Benchmark Provides Vendors and SOC Teams Needed Guidance to Select the Best LLM

Recently, Simbian introduced the first benchmark to comprehensively measure LLM performance in SOCs, measuring LLMs against a diverse range of real alerts and fundamental SOC tools over all phases of alert investigation, from alert ingestion to disposition and reporting.

Existing benchmarks compare LLMs over broad criteria such as language understanding, math, and reasoning. Some benchmarks exist for broad security tasks or very basic SOC tasks like alert summarization.

For the first time ever, the industry has a comprehensive benchmarking framework specifically designed for AI SOC. What makes this different from existing cybersecurity benchmarks like  CyberSecEval  or  CTIBench? The key lies in its realism and depth.

Forget generic scenarios. Simbian’s benchmark is built on the autonomous investigation of 100 full-kill chain scenarios that realistically mirror what human SOC analysts face every day. To achieve this, the company created diverse, real-world-based attack scenarios with known ground truth of malicious activity, allowing AI agents to investigate and be assessed against a clear baseline.

These scenarios are even based on historical behavior of well-known APT groups (like APT32, APT38, APT43) and cybercriminal organizations (Cobalt Group, Lapsus$), covering a wide range of MITRE ATT&CK™ Tactics and Techniques, with a focus on prevalent threats like ransomware and phishing.

To facilitate this rigorous LLM testing, Simbian leveraged their AI SOC agent. The evaluation process is evidence-grounded and data-driven, always kicking off with a triggered alert realistically representing the way SOC analysts operate. The AI agent then needed to determine if the alert was a True or False Positive, find evidence of malicious activity (think “CTF flags” in a red teaming exercise), correctly interpret that evidence by answering security-contextual questions, and provide high-level overview of the attacker’s activity and respond to the threat in an appropriate manner – all autonomously. This evidence-based approach is critical for managing hallucinations to ensure the LLM isn’t just guessing but validating its reasoning and actions.

Surprising Findings and Key Takeaways Benchmarking some of the most well-known and high-performing models available as of May 2025, the study including models from Anthropic, OpenAI, Google, and DeepSeek.

The Results:

All high-end models were able to complete over half of the investigation tasks, with performance ranging from 61% to 67%. For reference, during the first AI SOC Championship the best human analysts powered by AI SOC scored in the range of 73% to 85%, while Simbian’s AI Agent at extra effort settings reached the value of 72%. This suggests that LLMs are capable of much more than just summarizing and retrieving data; LLM capabilities extend to robust alert triage and tool use via API interactions.

Another key finding: AI SOC applications heavily lean on the software engineering capabilities of LLMs. This highlights the importance of thorough prompt engineering and agentic flow engineering, which involves feedback loops and continuous monitoring. In initial runs, some models struggled, requiring improved prompts and fallback mechanisms for coding agents involved in analyzing retrieved data.

It should be noted that Sonnet 3.5 sometimes outperformed newer versions like Sonnet 3.7 and 4.0. This could be due to “catastrophic forgetting,” where further domain specialization for software engineering or instruction following might detrimentally affect cybersecurity knowledge and investigation planning. This underscores the critical need for benchmarking to evaluate the fine-tuning of LLMs for specific domains.

The discovery was also made that “thinking models” (those that use post-training techniques and often involve into internal self-talk) didn’t show a considerable advantage in AI SOC applications, with all tested models demonstrating comparable performance. This resembles the findings of the studies on software bug fixing and red team CTF applications, which suggest that once LLMs hit a certain capability ceiling, additional inference leads to only marginal improvements, often at a higher cost. This points to the necessity of human-validated LLM applications in AI SOC and the continued development of fine-grained, specialized benchmarks for improving cybersecurity domain-focused reasoning.

For the full details and results of the benchmarking, click here. Simbian looks forward to leveraging this benchmark to evaluate new foundation models on a regular basis, with plans to share findings with the public.

Featured

  • Freedom of Choice

    In today's security landscape, we are witnessing a fundamental transformation in how organizations manage digital evidence. Law enforcement agencies, campus security teams, and large facility operators face increasingly complex challenges with expanding video data, tightening budget constraints and inflexible systems that limit innovation. Read Now

  • Accelerating a Pathway

    There is a new trend touting the transformational qualities of AI’s ability to deliver actionable data and predictive analysis that in many instances, seems to be a bit of an overpromise. The reality is that very few solutions in the cyber-physical security (CPS) space live up to this high expectation with the one exception being the new generation of Physical Identity and Access Management (PIAM) software – herein recategorized as PIAM+. Read Now

  • Protecting Your Zones

    It is game day. You can feel the crowd’s energy. In the parking lot. At the gate. In the stadium. On the concourse. Fans are eager to party. Food and merchandise vendors ready themselves for the rush. Read Now

  • Street Smarts

    The ongoing acceptance of AI and advanced data analytics has allowed surveillance camera technology to shift from being a tactical tool to a strategic business solution. Combining traditional surveillance technology with AI-based data-driven insights can streamline transportation systems, enhance traffic management, improve situational awareness, optimize resource allocation and streamline emergency response procedures. Read Now

  • The Progress of Biometrics

New Products

  • Luma x20

    Luma x20

    Snap One has announced its popular Luma x20 family of surveillance products now offers even greater security and privacy for home and business owners across the globe by giving them full control over integrators’ system access to view live and recorded video. According to Snap One Product Manager Derek Webb, the new “customer handoff” feature provides enhanced user control after initial installation, allowing the owners to have total privacy while also making it easy to reinstate integrator access when maintenance or assistance is required. This new feature is now available to all Luma x20 users globally. “The Luma x20 family of surveillance solutions provides excellent image and audio capture, and with the new customer handoff feature, it now offers absolute privacy for camera feeds and recordings,” Webb said. “With notifications and integrator access controlled through the powerful OvrC remote system management platform, it’s easy for integrators to give their clients full control of their footage and then to get temporary access from the client for any troubleshooting needs.”

  • QCS7230 System-on-Chip (SoC)

    QCS7230 System-on-Chip (SoC)

    The latest Qualcomm® Vision Intelligence Platform offers next-generation smart camera IoT solutions to improve safety and security across enterprises, cities and spaces. The Vision Intelligence Platform was expanded in March 2022 with the introduction of the QCS7230 System-on-Chip (SoC), which delivers superior artificial intelligence (AI) inferencing at the edge.

  • ResponderLink

    ResponderLink

    Shooter Detection Systems (SDS), an Alarm.com company and a global leader in gunshot detection solutions, has introduced ResponderLink, a groundbreaking new 911 notification service for gunshot events. ResponderLink completes the circle from detection to 911 notification to first responder awareness, giving law enforcement enhanced situational intelligence they urgently need to save lives. Integrating SDS’s proven gunshot detection system with Noonlight’s SendPolice platform, ResponderLink is the first solution to automatically deliver real-time gunshot detection data to 911 call centers and first responders. When shots are detected, the 911 dispatching center, also known as the Public Safety Answering Point or PSAP, is contacted based on the gunfire location, enabling faster initiation of life-saving emergency protocols.