Online Exclusive: Calculating the Relative Performance of a PIDS System

Calculating a realistic Probability of Detection (POD) or determining a measurement of actual detection performance for a PIDS is not quite as simple as many people believe. You need to understand the highly interactive and closely coupled relationship that exists between the detection of intrusions versus unwanted nuisance alarms.

You’ve done your research and collected all of the Perimeter Intrusion Detection Systems (PIDS) brochures and technical information available, and read through all of the specifications from various manufacturers.  Now you are completely bamboozled by claimed detection rates, nuisance alarm rates and false alarm rates.   So, how do you compare and evaluate a range of products to select the best performing system?

Calculating a realistic Probability of Detection (POD) or determining a measurement of actual detection performance for a PIDS is not quite as simple as many people believe.  You need to understand the highly interactive and closely coupled relationship that exists between the detection of intrusions versus unwanted nuisance alarms.   As end users demand ever increasing sensitivity and detection performance from their PIDS systems, controlling nuisance alarms will continue to be the greatest challenge facing suppliers.

This article offers one method to calculate a systems comparative performance taking this relationship and trade-off between detection and nuisance alarms into account.   Although we are only covering perimeter intrusion detection systems in this article, these same techniques could potentially be applied to other detection areas and technologies such as Ground Based Radar, Microwave, VMD, etc.

Detecting each and every intrusion on your perimeter is the primary expectation of any PIDS system, but equally important is the confidence that your security staff have in the system not only capturing and reporting all legitimate intrusions, but in eliminating or not reporting nuisance alarms.   Too many false or nuisance alarms will seriously erode confidence in the system, often to the stage where all alarms – real or not – are simply ignored by the security staff.

When alarms are ignored, the actual quality of performance in detecting an intruder in this scenario suddenly drops to 0% - regardless of what figures your vendor quotes for the equipment. So, the question is how do you measure and quantify this Quality of Performance?

Almost any vendor can claim 100% detection of intrusions, but often because of the high sensitivity settings typically used to achieve this figure during testing, a corresponding increase in nuisance alarms may render the system performance unacceptable from the customer’s perspective. What we need to do is measure both detection and nuisance alarm figures simultaneously in order to get a realistic Quality of Performance (QOP) or a measurement of relative system performance.  This methodology can be applied to the comparison of PIDS systems from different manufacturers as well as different technologies.

With a relatively small number of tests (typically in the order of 100’s) being carried out on a site over a relatively  short period of time as part of an onsite evaluation or system acceptance procedure to determine the QOP and level of Confidence, one method to use is as follows:

Example1:  As part of the commissioning of a PIDS system if for example we do a series of 30 climb tests on the fence and detect 29 out of the 30 climbs, giving us a Detection Rate of 29/30 or 0.966, and we also received one nuisance alarm during the tests also giving us 29 hits but with a total of 30 (true and false) actual alarms received = 96.6% confidence

The Quality or Performance (QOP) would be:

 0.966 x 96.6 = 93.3% QOP with a 96.6% confidence.  

 If we recorded the same detection rate but with no nuisance alarms during this test, then the result would be:

  .966 x 100 = 96.6% QOP with 100% confidence.

Example 2:   If we then increased the sensitivity during the test in order to improve the detection rate, we may see an increase in nuisance alarms.   If we record a detection rate of 30/30 or 1.0, but a confidence of say 30/35 = 85.7% (5 nuisance alarms) then the QOP would be 85.7% with an 85.7% confidence.   Actually worse system performance overall even though we made it more sensitive.

Example 3:  Conversely, if we reduced the sensitivity to have fewer nuisance alarms, we may miss actual intrusion events.   So for example a detection rate of 27/30 or .9, but a confidence of 27/27 or 100% would yield a QOP of 90% with 100% confidence.

So as we go either side of these optimal detection / nuisance alarm settings, we find that the QOP or detection performance falls off significantly. 

Of course, you should always adjust the PIDS system to achieve the optimal QOP figure with any installation – ideally it would be 100% - but in the real-world there are many external and site specific factors that can influence this figure, such as fence type, fence quality, environmental conditions, etc.   No two sites are ever the same and in addition to site specific and environmental factors, the detection rate may also be affected by the skill and knowledge of the intruder and therefore their ability to defeat the system.

Please note that this proposal of system QOP measurement is only intended for use at commissioning or comparison time, with a relatively small number of measurements (100’s) over a relatively short period of time (hours).  For measuring an individual site QOP over an extended period of time, a different calculation method based on binomial distribution would probably be more appropriate.

Often vendors quote just the raw detection rates for their technologies without any sort of confidence factor – so it’s unlikely to be the QOP you can realistically expect to achieve on your site.   Vendor testing is usually carried out in a controlled or laboratory environment where they increase sensitivity to do the detection tests, record this figure, then reduce the sensitivity down to a level to eliminate nuisance alarms and record this figure – both in isolation of each other.  The reality is that due to the highly interdependent nature of these results, both of need to be evaluated together in order to come up with a meaningful measurement. 

Testing also needs to be carried out at the actual installed site so that the site specific environmental conditions are taken into account when calculating the QOP figure.   Any POD figure or raw detection rate quoted will be conditional and unique to a site, despite the claims made by some sensor manufacturers.   For example, a sensor may have quite a high detection rate or POD for a low-level threat such as a teenage vandal who has little knowledge of the system versus a more sophisticated threat from a professional thief or special operations person for whom the detection rate or POD will almost certainly be substantially lower.  

However, the quality of performance (QOP) figure provides a relative measure of a systems ability to detect intrusions within the protected area. The QOP depends not only on the characteristics of a particular PIDS system, but also takes into account the environment, the method of installation and adjustment, and the assumed behavior of an intruder.   

Other than measuring a systems relative performance for comparison reasons, a QOP figure is also useful for establishing the base line performance of your installation for future reference.    All of the installation, system settings/sensitivity levels, maintenance and testing activities should be clearly documented and kept on site so that they can be used to provide a known basis for system support or fault troubleshooting and to ensure proper performance of the PIDS and identify any performance degradation over time.

Regular performance and operability testing should be carried out every 6 months or whenever any system component has been adjusted, replaced or repaired to confirm correct and effective operation of the PIDS system.  This performance testing should include the use of and be referenced back to the documented QOP figures from the original acceptance testing.

You can see from the above that how a PIDS system will actually perform on your site is often markedly different to the expectation set by the raw detection rate figures (often quoted as the POD).  No two sites are ever the same with numerous external and site specific factors impacting on this figure to reduce the quality of the overall system performance.  

Featured

  • Making the Grade with Locks and Door Hardware

    Managing and maintaining locks and door hardware across a school district or university campus is a big responsibility. A building’s security needs to change over time as occupancy and use demands evolve, which can make it even more challenging. Knowing the basics of common door hardware, including locks, panic devices and door closers, can make a difference in daily operations and emergency situations. Read Now

  • Choosing the Right Solution

    Today, there is a strong shift from on-prem installations to cloud or hybrid-cloud deployments. As reported in the 2024 Genetec State of Physical Security report, 66% of end users said they will move to managing or storing more physical security in the cloud over the next two years. Read Now

  • New Report Reveals Top Security Risks for U.S. Retail Chains

    Interface Systems, a provider of security, actionable insights, and purpose-built networks for multi-location businesses, has released its 2024 State of Remote Video Monitoring in Retail Chains report. The detailed study analyzed over 2 million monitoring requests across 4,156 retail locations in the United States from September 2023 to August 2024. Read Now

  • Gaining a Competitive Edge

    Ask most companies about their future technology plans and the answers will most likely include AI. Then ask how they plan to deploy it, and that is where the responses may start to vary. Every company has unique surveillance requirements that are based on market focus, scale, scope, risk tolerance, geographic area and, of course, budget. Those factors all play a role in deciding how to configure a surveillance system, and how to effectively implement technologies like AI. Read Now

Featured Cybersecurity

Webinars

New Products

  • Automatic Systems V07

    Automatic Systems V07

    Automatic Systems, an industry-leading manufacturer of pedestrian and vehicle secure entrance control access systems, is pleased to announce the release of its groundbreaking V07 software. The V07 software update is designed specifically to address cybersecurity concerns and will ensure the integrity and confidentiality of Automatic Systems applications. With the new V07 software, updates will be delivered by means of an encrypted file. 3

  • Camden CM-221 Series Switches

    Camden CM-221 Series Switches

    Camden Door Controls is pleased to announce that, in response to soaring customer demand, it has expanded its range of ValueWave™ no-touch switches to include a narrow (slimline) version with manual override. This override button is designed to provide additional assurance that the request to exit switch will open a door, even if the no-touch sensor fails to operate. This new slimline switch also features a heavy gauge stainless steel faceplate, a red/green illuminated light ring, and is IP65 rated, making it ideal for indoor or outdoor use as part of an automatic door or access control system. ValueWave™ no-touch switches are designed for easy installation and trouble-free service in high traffic applications. In addition to this narrow version, the CM-221 & CM-222 Series switches are available in a range of other models with single and double gang heavy-gauge stainless steel faceplates and include illuminated light rings. 3

  • Mobile Safe Shield

    Mobile Safe Shield

    SafeWood Designs, Inc., a manufacturer of patented bullet resistant products, is excited to announce the launch of the Mobile Safe Shield. The Mobile Safe Shield is a moveable bullet resistant shield that provides protection in the event of an assailant and supplies cover in the event of an active shooter. With a heavy-duty steel frame, quality castor wheels, and bullet resistant core, the Mobile Safe Shield is a perfect addition to any guard station, security desks, courthouses, police stations, schools, office spaces and more. The Mobile Safe Shield is incredibly customizable. Bullet resistant materials are available in UL 752 Levels 1 through 8 and include glass, white board, tack board, veneer, and plastic laminate. Flexibility in bullet resistant materials allows for the Mobile Safe Shield to blend more with current interior décor for a seamless design aesthetic. Optional custom paint colors are also available for the steel frame. 3