Online Exclusive: Calculating the Relative Performance of a PIDS System

Calculating a realistic Probability of Detection (POD) or determining a measurement of actual detection performance for a PIDS is not quite as simple as many people believe. You need to understand the highly interactive and closely coupled relationship that exists between the detection of intrusions versus unwanted nuisance alarms.

You’ve done your research and collected all of the Perimeter Intrusion Detection Systems (PIDS) brochures and technical information available, and read through all of the specifications from various manufacturers.  Now you are completely bamboozled by claimed detection rates, nuisance alarm rates and false alarm rates.   So, how do you compare and evaluate a range of products to select the best performing system?

Calculating a realistic Probability of Detection (POD) or determining a measurement of actual detection performance for a PIDS is not quite as simple as many people believe.  You need to understand the highly interactive and closely coupled relationship that exists between the detection of intrusions versus unwanted nuisance alarms.   As end users demand ever increasing sensitivity and detection performance from their PIDS systems, controlling nuisance alarms will continue to be the greatest challenge facing suppliers.

This article offers one method to calculate a systems comparative performance taking this relationship and trade-off between detection and nuisance alarms into account.   Although we are only covering perimeter intrusion detection systems in this article, these same techniques could potentially be applied to other detection areas and technologies such as Ground Based Radar, Microwave, VMD, etc.

Detecting each and every intrusion on your perimeter is the primary expectation of any PIDS system, but equally important is the confidence that your security staff have in the system not only capturing and reporting all legitimate intrusions, but in eliminating or not reporting nuisance alarms.   Too many false or nuisance alarms will seriously erode confidence in the system, often to the stage where all alarms – real or not – are simply ignored by the security staff.

When alarms are ignored, the actual quality of performance in detecting an intruder in this scenario suddenly drops to 0% - regardless of what figures your vendor quotes for the equipment. So, the question is how do you measure and quantify this Quality of Performance?

Almost any vendor can claim 100% detection of intrusions, but often because of the high sensitivity settings typically used to achieve this figure during testing, a corresponding increase in nuisance alarms may render the system performance unacceptable from the customer’s perspective. What we need to do is measure both detection and nuisance alarm figures simultaneously in order to get a realistic Quality of Performance (QOP) or a measurement of relative system performance.  This methodology can be applied to the comparison of PIDS systems from different manufacturers as well as different technologies.

With a relatively small number of tests (typically in the order of 100’s) being carried out on a site over a relatively  short period of time as part of an onsite evaluation or system acceptance procedure to determine the QOP and level of Confidence, one method to use is as follows:

Example1:  As part of the commissioning of a PIDS system if for example we do a series of 30 climb tests on the fence and detect 29 out of the 30 climbs, giving us a Detection Rate of 29/30 or 0.966, and we also received one nuisance alarm during the tests also giving us 29 hits but with a total of 30 (true and false) actual alarms received = 96.6% confidence

The Quality or Performance (QOP) would be:

 0.966 x 96.6 = 93.3% QOP with a 96.6% confidence.  

 If we recorded the same detection rate but with no nuisance alarms during this test, then the result would be:

  .966 x 100 = 96.6% QOP with 100% confidence.

Example 2:   If we then increased the sensitivity during the test in order to improve the detection rate, we may see an increase in nuisance alarms.   If we record a detection rate of 30/30 or 1.0, but a confidence of say 30/35 = 85.7% (5 nuisance alarms) then the QOP would be 85.7% with an 85.7% confidence.   Actually worse system performance overall even though we made it more sensitive.

Example 3:  Conversely, if we reduced the sensitivity to have fewer nuisance alarms, we may miss actual intrusion events.   So for example a detection rate of 27/30 or .9, but a confidence of 27/27 or 100% would yield a QOP of 90% with 100% confidence.

So as we go either side of these optimal detection / nuisance alarm settings, we find that the QOP or detection performance falls off significantly. 

Of course, you should always adjust the PIDS system to achieve the optimal QOP figure with any installation – ideally it would be 100% - but in the real-world there are many external and site specific factors that can influence this figure, such as fence type, fence quality, environmental conditions, etc.   No two sites are ever the same and in addition to site specific and environmental factors, the detection rate may also be affected by the skill and knowledge of the intruder and therefore their ability to defeat the system.

Please note that this proposal of system QOP measurement is only intended for use at commissioning or comparison time, with a relatively small number of measurements (100’s) over a relatively short period of time (hours).  For measuring an individual site QOP over an extended period of time, a different calculation method based on binomial distribution would probably be more appropriate.

Often vendors quote just the raw detection rates for their technologies without any sort of confidence factor – so it’s unlikely to be the QOP you can realistically expect to achieve on your site.   Vendor testing is usually carried out in a controlled or laboratory environment where they increase sensitivity to do the detection tests, record this figure, then reduce the sensitivity down to a level to eliminate nuisance alarms and record this figure – both in isolation of each other.  The reality is that due to the highly interdependent nature of these results, both of need to be evaluated together in order to come up with a meaningful measurement. 

Testing also needs to be carried out at the actual installed site so that the site specific environmental conditions are taken into account when calculating the QOP figure.   Any POD figure or raw detection rate quoted will be conditional and unique to a site, despite the claims made by some sensor manufacturers.   For example, a sensor may have quite a high detection rate or POD for a low-level threat such as a teenage vandal who has little knowledge of the system versus a more sophisticated threat from a professional thief or special operations person for whom the detection rate or POD will almost certainly be substantially lower.  

However, the quality of performance (QOP) figure provides a relative measure of a systems ability to detect intrusions within the protected area. The QOP depends not only on the characteristics of a particular PIDS system, but also takes into account the environment, the method of installation and adjustment, and the assumed behavior of an intruder.   

Other than measuring a systems relative performance for comparison reasons, a QOP figure is also useful for establishing the base line performance of your installation for future reference.    All of the installation, system settings/sensitivity levels, maintenance and testing activities should be clearly documented and kept on site so that they can be used to provide a known basis for system support or fault troubleshooting and to ensure proper performance of the PIDS and identify any performance degradation over time.

Regular performance and operability testing should be carried out every 6 months or whenever any system component has been adjusted, replaced or repaired to confirm correct and effective operation of the PIDS system.  This performance testing should include the use of and be referenced back to the documented QOP figures from the original acceptance testing.

You can see from the above that how a PIDS system will actually perform on your site is often markedly different to the expectation set by the raw detection rate figures (often quoted as the POD).  No two sites are ever the same with numerous external and site specific factors impacting on this figure to reduce the quality of the overall system performance.  

Featured

  • 12 Commercial Crime Sites to Do Your Research

    12 Commercial Crime Sites to Do Your Research

    Understanding crime statistics in your industry and area is crucial for making important decisions about your security budget. With so much information out there, how can you know which statistics to trust? Read Now

  • Boosting Safety and Efficiency

    Boosting Safety and Efficiency

    In alignment with the state of Mississippi’s mission of “Empowering Mississippi citizens to stay connected and engaged with their government,” Salient's CompleteView VMS is being installed throughout more than 150 state boards, commissions and agencies in order to ensure safety for thousands of constituents who access state services daily. Read Now

  • Live From GSX: Post-Show Review

    Live From GSX: Post-Show Review

    This year’s Live From GSX program was a rousing success! Again, we’d like to thank our partners, and IPVideo, for working with us and letting us broadcast their solutions to the industry. You can follow our Live From GSX 2023 page to keep up with post-show developments and announcements. And if you’re interested in working with us in 2024, please don’t hesitate to ask about our Live From programs for ISC West in March or next year’s GSX. Read Now

    • Industry Events
    • GSX
  • People Say the Funniest Things

    People Say the Funniest Things

    By all accounts, GSX version 2023 was completely successful. Apparently, there were plenty of mix-ups with the airlines and getting aircraft from the East Coast into Big D. I am all ears when I am in a gathering of people. You never know when a nugget of information might flip out. Read Now

    • Industry Events
    • GSX

Featured Cybersecurity

Webinars

New Products

  • FEP GameChanger

    FEP GameChanger

    Paige Datacom Solutions Introduces Important and Innovative Cabling Products GameChanger Cable, a proven and patented solution that significantly exceeds the reach of traditional category cable will now have a FEP/FEP construction. 3

  • Connect ONE’s powerful cloud-hosted management platform provides the means to tailor lockdowns and emergency mass notifications throughout a facility – while simultaneously alerting occupants to hazards or next steps, like evacuation.

    Connect ONE®

    Connect ONE’s powerful cloud-hosted management platform provides the means to tailor lockdowns and emergency mass notifications throughout a facility – while simultaneously alerting occupants to hazards or next steps, like evacuation. 3

  • Camden CV-7600 High Security Card Readers

    Camden CV-7600 High Security Card Readers

    Camden Door Controls has relaunched its CV-7600 card readers in response to growing market demand for a more secure alternative to standard proximity credentials that can be easily cloned. CV-7600 readers support MIFARE DESFire EV1 & EV2 encryption technology credentials, making them virtually clone-proof and highly secure. 3