Online Exclusive: Calculating the Relative Performance of a PIDS System

Calculating a realistic Probability of Detection (POD) or determining a measurement of actual detection performance for a PIDS is not quite as simple as many people believe. You need to understand the highly interactive and closely coupled relationship that exists between the detection of intrusions versus unwanted nuisance alarms.

You’ve done your research and collected all of the Perimeter Intrusion Detection Systems (PIDS) brochures and technical information available, and read through all of the specifications from various manufacturers.  Now you are completely bamboozled by claimed detection rates, nuisance alarm rates and false alarm rates.   So, how do you compare and evaluate a range of products to select the best performing system?

Calculating a realistic Probability of Detection (POD) or determining a measurement of actual detection performance for a PIDS is not quite as simple as many people believe.  You need to understand the highly interactive and closely coupled relationship that exists between the detection of intrusions versus unwanted nuisance alarms.   As end users demand ever increasing sensitivity and detection performance from their PIDS systems, controlling nuisance alarms will continue to be the greatest challenge facing suppliers.

This article offers one method to calculate a systems comparative performance taking this relationship and trade-off between detection and nuisance alarms into account.   Although we are only covering perimeter intrusion detection systems in this article, these same techniques could potentially be applied to other detection areas and technologies such as Ground Based Radar, Microwave, VMD, etc.

Detecting each and every intrusion on your perimeter is the primary expectation of any PIDS system, but equally important is the confidence that your security staff have in the system not only capturing and reporting all legitimate intrusions, but in eliminating or not reporting nuisance alarms.   Too many false or nuisance alarms will seriously erode confidence in the system, often to the stage where all alarms – real or not – are simply ignored by the security staff.

When alarms are ignored, the actual quality of performance in detecting an intruder in this scenario suddenly drops to 0% - regardless of what figures your vendor quotes for the equipment. So, the question is how do you measure and quantify this Quality of Performance?

Almost any vendor can claim 100% detection of intrusions, but often because of the high sensitivity settings typically used to achieve this figure during testing, a corresponding increase in nuisance alarms may render the system performance unacceptable from the customer’s perspective. What we need to do is measure both detection and nuisance alarm figures simultaneously in order to get a realistic Quality of Performance (QOP) or a measurement of relative system performance.  This methodology can be applied to the comparison of PIDS systems from different manufacturers as well as different technologies.

With a relatively small number of tests (typically in the order of 100’s) being carried out on a site over a relatively  short period of time as part of an onsite evaluation or system acceptance procedure to determine the QOP and level of Confidence, one method to use is as follows:

Example1:  As part of the commissioning of a PIDS system if for example we do a series of 30 climb tests on the fence and detect 29 out of the 30 climbs, giving us a Detection Rate of 29/30 or 0.966, and we also received one nuisance alarm during the tests also giving us 29 hits but with a total of 30 (true and false) actual alarms received = 96.6% confidence

The Quality or Performance (QOP) would be:

 0.966 x 96.6 = 93.3% QOP with a 96.6% confidence.  

 If we recorded the same detection rate but with no nuisance alarms during this test, then the result would be:

  .966 x 100 = 96.6% QOP with 100% confidence.

Example 2:   If we then increased the sensitivity during the test in order to improve the detection rate, we may see an increase in nuisance alarms.   If we record a detection rate of 30/30 or 1.0, but a confidence of say 30/35 = 85.7% (5 nuisance alarms) then the QOP would be 85.7% with an 85.7% confidence.   Actually worse system performance overall even though we made it more sensitive.

Example 3:  Conversely, if we reduced the sensitivity to have fewer nuisance alarms, we may miss actual intrusion events.   So for example a detection rate of 27/30 or .9, but a confidence of 27/27 or 100% would yield a QOP of 90% with 100% confidence.

So as we go either side of these optimal detection / nuisance alarm settings, we find that the QOP or detection performance falls off significantly. 

Of course, you should always adjust the PIDS system to achieve the optimal QOP figure with any installation – ideally it would be 100% - but in the real-world there are many external and site specific factors that can influence this figure, such as fence type, fence quality, environmental conditions, etc.   No two sites are ever the same and in addition to site specific and environmental factors, the detection rate may also be affected by the skill and knowledge of the intruder and therefore their ability to defeat the system.

Please note that this proposal of system QOP measurement is only intended for use at commissioning or comparison time, with a relatively small number of measurements (100’s) over a relatively short period of time (hours).  For measuring an individual site QOP over an extended period of time, a different calculation method based on binomial distribution would probably be more appropriate.

Often vendors quote just the raw detection rates for their technologies without any sort of confidence factor – so it’s unlikely to be the QOP you can realistically expect to achieve on your site.   Vendor testing is usually carried out in a controlled or laboratory environment where they increase sensitivity to do the detection tests, record this figure, then reduce the sensitivity down to a level to eliminate nuisance alarms and record this figure – both in isolation of each other.  The reality is that due to the highly interdependent nature of these results, both of need to be evaluated together in order to come up with a meaningful measurement. 

Testing also needs to be carried out at the actual installed site so that the site specific environmental conditions are taken into account when calculating the QOP figure.   Any POD figure or raw detection rate quoted will be conditional and unique to a site, despite the claims made by some sensor manufacturers.   For example, a sensor may have quite a high detection rate or POD for a low-level threat such as a teenage vandal who has little knowledge of the system versus a more sophisticated threat from a professional thief or special operations person for whom the detection rate or POD will almost certainly be substantially lower.  

However, the quality of performance (QOP) figure provides a relative measure of a systems ability to detect intrusions within the protected area. The QOP depends not only on the characteristics of a particular PIDS system, but also takes into account the environment, the method of installation and adjustment, and the assumed behavior of an intruder.   

Other than measuring a systems relative performance for comparison reasons, a QOP figure is also useful for establishing the base line performance of your installation for future reference.    All of the installation, system settings/sensitivity levels, maintenance and testing activities should be clearly documented and kept on site so that they can be used to provide a known basis for system support or fault troubleshooting and to ensure proper performance of the PIDS and identify any performance degradation over time.

Regular performance and operability testing should be carried out every 6 months or whenever any system component has been adjusted, replaced or repaired to confirm correct and effective operation of the PIDS system.  This performance testing should include the use of and be referenced back to the documented QOP figures from the original acceptance testing.

You can see from the above that how a PIDS system will actually perform on your site is often markedly different to the expectation set by the raw detection rate figures (often quoted as the POD).  No two sites are ever the same with numerous external and site specific factors impacting on this figure to reduce the quality of the overall system performance.  

Featured

  • Survey: 54% of Organizations Cite Technical Debt as Top Hurdle to Identity System Modernization

    Modernizing identity systems is proving difficult for organizations due to two key challenges: decades of accumulated Identity and Access Management (IAM) technical debt and the complexity of managing access across multiple identity providers (IDPs). These findings come from the new Strata Identity-commissioned report, State of Multi-Cloud Identity: Insights and Trends for 2025. The report, based on survey data from the Cloud Security Alliance (CSA), highlights trends and challenges in securing cloud environments. The CSA is the world’s leading organization dedicated to defining standards, certifications, and best practices to help ensure a secure cloud computing environment. Read Now

  • Study: Only 35 Percent of Companies Include Cybersecurity Teams When Implementing AI

    Only 35 percent of cybersecurity professionals or teams are involved in the development of policy governing the use of AI technology in their enterprise, and nearly half (45 percent) report no involvement in the development, onboarding, or implementation of AI solutions, according to the recently released 2024 State of Cybersecurity survey report from ISACA, a global professional association advancing trust in technology. Read Now

  • New Report Series Highlights E-Commerce Threats, Fraud Against Retailers

    Trustwave, a cybersecurity and managed security services provider, recently released a series of reports detailing the threats facing the retail sector, marking the second year of its ongoing research into these critical security issues. Read Now

  • Stay Secure in 2024: Updated Cybersecurity Tips for the Office and at Home

    Cyber criminals get more inventive every year. Cybersecurity threats continue to evolve and are a moving target for business owners in 2024. Companies large and small need to employ cybersecurity best practices throughout their organization. That includes security integrators, manufacturers, and end users. Read Now

Featured Cybersecurity

Webinars

New Products

  • FEP GameChanger

    FEP GameChanger

    Paige Datacom Solutions Introduces Important and Innovative Cabling Products GameChanger Cable, a proven and patented solution that significantly exceeds the reach of traditional category cable will now have a FEP/FEP construction. 3

  • HD2055 Modular Barricade

    Delta Scientific’s electric HD2055 modular shallow foundation barricade is tested to ASTM M50/P1 with negative penetration from the vehicle upon impact. With a shallow foundation of only 24 inches, the HD2055 can be installed without worrying about buried power lines and other below grade obstructions. The modular make-up of the barrier also allows you to cover wider roadways by adding additional modules to the system. The HD2055 boasts an Emergency Fast Operation of 1.5 seconds giving the guard ample time to deploy under a high threat situation. 3

  • Compact IP Video Intercom

    Viking’s X-205 Series of intercoms provide HD IP video and two-way voice communication - all wrapped up in an attractive compact chassis. 3