NIST Software Uses Combination Testing To Catch Bugs Quickly

Researchers at the National Institute of Standards and Technology (NIST) have released an updated version of a computer system testing tool that can cut costs by more efficiently finding flaws. A tutorial on using the tool accompanies the new release.

Catching software "bugs" before a program is released enhances computer security because hackers often exploit these flaws to introduce malware, including viruses, to disrupt or take control of computer systems. But it's difficult. A widely cited 2002 study prepared for NIST (Research Triangle Institute, The Economic Impacts of Inadequate Infrastructure for Software Testing, NIST Planning Report 02-3, May 2002) reported that even though 50 percent of software development budgets go to testing, flaws in software still cost the U.S. economy $59.5 billion annually.

Exhaustive checking of all possible combinations of input actions that could cause software failure is not practical, explained NIST's Raghu Kacker, because of the huge number of possibilities, but it's also not necessary. Based on studies of software crashes in applications, including medical devices and Web browsers, NIST's Rick Kuhn and other researchers determined that between 70 and 95 percent of software failures are triggered by only two variables interacting and practically 100 percent of software failures are triggered by no more than six.

"Testing every combination up to six variables can be as good as exhaustive testing," said Kacker.

Working with researcher Jeff Yu Lei and his students from the University of Texas at Arlington, NIST designed Advanced Combinatorial Testing System (ACTS), a freely distributed software tool to generate plans for efficiently testing combinations of two to six interacting variables. The method goes beyond the commonly used "pairwise" approach to software testing, which tests combinations of two variables, so it can detect more obscure flaws.

Kuhn describes the process "as packing as many combinations into a set of tests as efficiently as we know how." For example, testing all possible interactions for a product with 34 on and off switches would require 17 billion tests. Using ACTS, all three-way interactions can be evaluated using only 33 tests and all six-way combinations with just 522 tests, instead of 17 billion.

The first version of ACTS was released in 2008. Since then, it has been distributed at no cost to 465 organizations and individuals in industry, academia and government. "About half of our users are in IT, but other heavy users are in the financial, defense and telecommunications sectors," said Kuhn. In August, NIST and Lockheed Martin initiated a Cooperative Research and Development Agreement to study the application of ACTS in the company's large and complex software applications. The two groups will jointly publish the results.

NIST released the latest update of ACTS in October. The new version includes an improved user interface and a better method of specifying relationships between parameters for testing. This can eliminate the problem, for example, of spending time on tests for invalid combinations, such as using Internet Explorer on a Linux system. Information for requesting ACTS is available at http://csrc.nist.gov/groups/SNS/acts/index.html.

Featured

  • Progressing in Capabilities

    Progressing in Capabilities

    Hazardous areas within industries like oil and gas, manufacturing, agriculture and the like, have long-sought reliable video surveillance cameras and equipment that can operate safely in these harsh and unpredictable environments. Read Now

  • A Comprehensive Nationwide Solution

    A Comprehensive Nationwide Solution

    Across the United States, manufacturing facilities, distribution centers, truck yards, parking lots and car dealerships all have a common concern. They are targets for catalytic converters. In nearly every region, cases of catalytic converter thefts have skyrocketed. Read Now

  • Planning for Your Perimeter

    Planning for Your Perimeter

    The perimeter is an organization’s first line of defense and a critical element of any security and surveillance program. Even if a building’s interior or exterior security is strong, without a solid perimeter surveillance approach any company or business is vulnerable. Read Now

  • The Key Issue

    The Key Issue

    It is February 2014. A woman is getting ready in her room on a cruise ship when she hears a knock on the door; it is a crewmember delivering breakfast. She is not presentable so she tells him to leave it by the door. Read Now

Featured Cybersecurity

New Products

  • BriefCam v6.0

    BriefCam v6.0

    BriefCam has released BriefCam v6.0, which introduces the new deployment option of a multi-site architecture. This enables businesses with multiple, distributed locations to view aggregate data from all remote sites to uncover trends across locations, optimize operations and boost real-time alerting and response – all while continuing to reap the benefits of BriefCam's powerful analytics platform for making video searchable, actionable and quantifiable. 3

  • Videoloft Cloud Video Surveillance VSaaS Solution

    Videoloft Cloud Video Surveillance VSaaS Solution

    Videoloft focuses on transforming traditional professional surveillance systems into cloud connected solutions via the Videoloft Cloud Adapter. 3

  • Tyco Kantech EntraPass security management software

    Tyco Kantech EntraPass security management software

    Johnson Controls, the global leader in smart, healthy and sustainable buildings, and architect of the Open Blue digital connected platforms, has released the newest version of the Tyco Kantech EntraPass security management software. 3