NIST Software Uses Combination Testing To Catch Bugs Quickly

  • Nov 16, 2010

Researchers at the National Institute of Standards and Technology (NIST) have released an updated version of a computer system testing tool that can cut costs by more efficiently finding flaws. A tutorial on using the tool accompanies the new release.

Catching software "bugs" before a program is released enhances computer security because hackers often exploit these flaws to introduce malware, including viruses, to disrupt or take control of computer systems. But it's difficult. A widely cited 2002 study prepared for NIST (Research Triangle Institute, The Economic Impacts of Inadequate Infrastructure for Software Testing, NIST Planning Report 02-3, May 2002) reported that even though 50 percent of software development budgets go to testing, flaws in software still cost the U.S. economy $59.5 billion annually.

Exhaustive checking of all possible combinations of input actions that could cause software failure is not practical, explained NIST's Raghu Kacker, because of the huge number of possibilities, but it's also not necessary. Based on studies of software crashes in applications, including medical devices and Web browsers, NIST's Rick Kuhn and other researchers determined that between 70 and 95 percent of software failures are triggered by only two variables interacting and practically 100 percent of software failures are triggered by no more than six.

"Testing every combination up to six variables can be as good as exhaustive testing," said Kacker.

Working with researcher Jeff Yu Lei and his students from the University of Texas at Arlington, NIST designed Advanced Combinatorial Testing System (ACTS), a freely distributed software tool to generate plans for efficiently testing combinations of two to six interacting variables. The method goes beyond the commonly used "pairwise" approach to software testing, which tests combinations of two variables, so it can detect more obscure flaws.

Kuhn describes the process "as packing as many combinations into a set of tests as efficiently as we know how." For example, testing all possible interactions for a product with 34 on and off switches would require 17 billion tests. Using ACTS, all three-way interactions can be evaluated using only 33 tests and all six-way combinations with just 522 tests, instead of 17 billion.

The first version of ACTS was released in 2008. Since then, it has been distributed at no cost to 465 organizations and individuals in industry, academia and government. "About half of our users are in IT, but other heavy users are in the financial, defense and telecommunications sectors," said Kuhn. In August, NIST and Lockheed Martin initiated a Cooperative Research and Development Agreement to study the application of ACTS in the company's large and complex software applications. The two groups will jointly publish the results.

NIST released the latest update of ACTS in October. The new version includes an improved user interface and a better method of specifying relationships between parameters for testing. This can eliminate the problem, for example, of spending time on tests for invalid combinations, such as using Internet Explorer on a Linux system. Information for requesting ACTS is available at http://csrc.nist.gov/groups/SNS/acts/index.html.

Featured

  • Survey: 48 Percent of Worshippers Feel Less Safe Attending In-Person Services

    Almost half (48%) of those who attend religious services say they feel less safe attending in-person due to rising acts of violence at places of worship. In fact, 39% report these safety concerns have led them to change how often they attend in-person services, according to new research from Verkada conducted online by The Harris Poll among 1,123 U.S. adults who attend a religious service or event at least once a month. Read Now

  • AI Used as Part of Sophisticated Espionage Campaign

    A cybersecurity inflection point has been reached in which AI models has become genuinely useful in cybersecurity operation. But to no surprise, they can used for both good works and ill will. Systemic evaluations show cyber capabilities double in six months, and they have been tracking real-world cyberattacks showing how malicious actors were using AI capabilities. These capabilities were predicted and are expected to evolve, but what stood out for researchers was how quickly they have done so, at scale. Read Now

  • Why the Future of Video Security Is Happening Outside the Cloud

    For years, the cloud has captivated the physical security industry. And for good reasons. Remote access, elastic scalability and simplified maintenance reshaped how we think about deploying and managing systems. Read Now

  • UL Solutions Launches Artificial Intelligence Safety Certification Services

    UL Solutions Inc., a global leader in safety science, today announced the launch of artificial intelligence (AI) safety certification services, enabling comprehensive assessments for evaluating the safety of AI-powered products. Read Now

  • ESA Announces Initiative to Introduce the SECURE Act in State Legislatures

    The Electronic Security Association (ESA), the national voice for the electronic security and life safety industry, has announced plans to introduce the SECURE Act in state legislatures across the country beginning in 2025. The proposal, known as Safeguarding Election Candidates Using Reasonable Expenditures, provides a clear framework that allows candidates and elected officials to use campaign funds for professional security services. Read Now

    • Guard Services
Most   Popular

New Products

  • PE80 Series

    PE80 Series by SARGENT / ED4000/PED5000 Series by Corbin Russwin

    ASSA ABLOY, a global leader in access solutions, has announced the launch of two next generation exit devices from long-standing leaders in the premium exit device market: the PE80 Series by SARGENT and the PED4000/PED5000 Series by Corbin Russwin. These new exit devices boast industry-first features that are specifically designed to provide enhanced safety, security and convenience, setting new standards for exit solutions. The SARGENT PE80 and Corbin Russwin PED4000/PED5000 Series exit devices are engineered to meet the ever-evolving needs of modern buildings. Featuring the high strength, security and durability that ASSA ABLOY is known for, the new exit devices deliver several innovative, industry-first features in addition to elegant design finishes for every opening.

  • A8V MIND

    A8V MIND

    Hexagon’s Geosystems presents a portable version of its Accur8vision detection system. A rugged all-in-one solution, the A8V MIND (Mobile Intrusion Detection) is designed to provide flexible protection of critical outdoor infrastructure and objects. Hexagon’s Accur8vision is a volumetric detection system that employs LiDAR technology to safeguard entire areas. Whenever it detects movement in a specified zone, it automatically differentiates a threat from a nonthreat, and immediately notifies security staff if necessary. Person detection is carried out within a radius of 80 meters from this device. Connected remotely via a portable computer device, it enables remote surveillance and does not depend on security staff patrolling the area.

  • HD2055 Modular Barricade

    Delta Scientific’s electric HD2055 modular shallow foundation barricade is tested to ASTM M50/P1 with negative penetration from the vehicle upon impact. With a shallow foundation of only 24 inches, the HD2055 can be installed without worrying about buried power lines and other below grade obstructions. The modular make-up of the barrier also allows you to cover wider roadways by adding additional modules to the system. The HD2055 boasts an Emergency Fast Operation of 1.5 seconds giving the guard ample time to deploy under a high threat situation.