Study: Nearly 80 Percent Of IT Security Products Do Not Perform As Intended

Nearly 80 percent of security products fail to perform as intended when first tested and generally require two or more cycles of testing before achieving certification, according to a new ICSA Labs report.  The “ICSA Labs Product Assurance Report” -- a first-of-its-kind study co-authored by the Verizon Business Data Breach Investigations Report research team -- details lessons gleaned from testing thousands of security products over 20 years. 

The report found that the No. 1 reason why a product fails during initial testing is that it doesn’t adequately perform as intended. Across seven product categories, core product functionality accounted for 78 percent of initial test failures -- for example, an anti-virus product failing to prevent infection or an IPS (intrusion prevention system) product failing to filter malicious traffic.

The failure of a product to completely and accurately log data was the second most common reason security products do not perform as intended. Incomplete or inaccurate logging of who did what and when accounted for 58 percent of initial failures.

The report findings suggest that some vendors and enterprise users consider logging a nuisance and merely a “box to check.”  According to the report, logging is a particular challenge for firewalls.  Almost every network firewall (97 percent) or Web application firewall (80 percent) tested experienced at least one logging problem.

The third most significant reason for product failure is the finding that 44 percent of security products had inherent security problems, including vulnerabilities that compromise the confidentiality or integrity of the system and random behavior that affects product availability.

Even though it can be a demanding process, certification with a trusted, established third party is critical to verifying product quality, states the report.  Product categories studied were: anti-virus, network firewall, Web application firewall, network IPS, IPSec VPN, SSL VPN and custom testing.

“Our goal is to help vendors develop more secure products,” said George Japak, managing director, ICSA Labs and a co-author of the report.  “When a product fails, we encourage vendors to view that as an opportunity to improve the product before it goes to market.  In addition to benefiting the security industry, this open exchange of information can greatly benefit enterprises by providing them more reliable and available information to make educated product purchasing and use decisions.”

The ICSA Labs testing and certification process is rigorous; only 4 percent of products tested attained certification during the first testing cycle.  However, 82 percent of products resubmitted for testing eventually earn ICSA Labs certification.  Once a vendor earns certification, products are required to undergo ongoing testing to maintain certification.

Japak said, “The question I ask vendors is: ‘Who would you rather have find an issue in your product -- ICSA Labs in a safe testing environment or a criminal in the real world?’”

The study also identified several other issues including poor product documentation and problems involving patching -- a product’s ability to accept updates correctly.

The complete report is available at http://www.icsalabs.com/whitepaper/report.

Featured

  • AI to Help Resolve Non-Emergency Calls Across Utah and Decrease 911 Caller Wait Times

    The Utah Communications Authority (UCA), which oversees the state’s next generation 911 technology services, recently announced that public safety answering points (PSAPs) throughout the state plan to implement Motorola Solutions’ Virtual Response technology to automate the receipt and resolution of 10-digit non-emergency line calls in Utah with the help of AI. Read Now

  • Report: 2025 Video Surveillance Market Set to Grow After Small Decline in 2024

    Novaira Insights has unveiled its latest report, “World Market for Video Surveillance Hardware and Software – 2025 Edition.” The research indicates that the global market for video surveillance hardware and software experienced a slight decline of 0.3% in 2024. This performance fell short of previous forecasts, primarily due to a significant decrease of 7.8% in the Chinese market. Conversely, the rest of the world saw a growth of 4.9%. The global market for video surveillance equipment was estimated to be worth $25.0 billion in 2024. Read Now

  • Report Reveals Local Governments Face Surge in Ransomware Attacks with Minimal Resources

    KnowBe4, the cybersecurity platform that comprehensively addresses human risk management, recently released new research highlighting the critical cybersecurity challenges facing state, local, tribal, and territorial (SLTT) governments. The report details how government organizations have become prime targets for cybercriminals while simultaneously facing severe resource constraints. Read Now

  • Video Surveillance Trends to Watch

    With more organizations adding newer capabilities to their surveillance systems, it’s always important to remember the “basics” of system configuration and deployment, as well as the topline benefits of continually emerging technologies like AI and the cloud. Read Now

  • New Report Reveals Top Trends Transforming Access Controller Technology

    Mercury Security, a provider in access control hardware and open platform solutions, has published its Trends in Access Controllers Report, based on a survey of over 450 security professionals across North America and Europe. The findings highlight the controller’s vital role in a physical access control system (PACS), where the device not only enforces access policies but also connects with readers to verify user credentials—ranging from ID badges to biometrics and mobile identities. With 72% of respondents identifying the controller as a critical or important factor in PACS design, the report underscores how the choice of controller platform has become a strategic decision for today’s security leaders. Read Now

New Products

  • Compact IP Video Intercom

    Viking’s X-205 Series of intercoms provide HD IP video and two-way voice communication - all wrapped up in an attractive compact chassis.

  • QCS7230 System-on-Chip (SoC)

    QCS7230 System-on-Chip (SoC)

    The latest Qualcomm® Vision Intelligence Platform offers next-generation smart camera IoT solutions to improve safety and security across enterprises, cities and spaces. The Vision Intelligence Platform was expanded in March 2022 with the introduction of the QCS7230 System-on-Chip (SoC), which delivers superior artificial intelligence (AI) inferencing at the edge.

  • 4K Video Decoder

    3xLOGIC’s VH-DECODER-4K is perfect for use in organizations of all sizes in diverse vertical sectors such as retail, leisure and hospitality, education and commercial premises.