A Smart Breakthrough

Intelligent video surveillance: Get ready for a disruptive technology

• By Alan J. Lipton, Yvonne Cager
• Mar 01, 2006

THE widespread deployment of video surveillance is a potent weapon against terrorists, old-fashioned thieves, intruders and vandals. But it also confronts the security industry with unprecedented challenges. The new reality that overwhelms conventional human-centric solutions is the huge volume of video data.

After acquiring gigabytes of video data, how is it to be transported? Who or what monitors it? How are the security implications observed and understood? How is important information given priority? How are decisions transformed into appropriate actions? How does the security agency make the most-effective use of its limited staff?

To save security personnel from drowning in a flood of video data, leading-edge government agencies and corporations are turning to software intelligence powered by high-performance digital signal processors for a solution.

Implementing intelligent video surveillance (IVS) has critical implications for security systems. It will speed the trend of digital video replacing analog because of the increased effectiveness of the solution. Control will evolve from a centralized model to a distributed one. Security personnel will be taken out of the control room and placed closer to the scene of the action.

As a result, IVS and video data processing requires hardware that is more powerful than conventional surveillance systems, yet flexible enough to handle multiple video standards and codec changes without costly retrofits. IVS software is an order of magnitude more capable than conventional analysis software.

While the concept of such powerful systems may seem like a futuristic scenario, it is a part of everyday operations in many facilities?public and private. IVS systems are being deployed to meet performance and cost targets, and security personnel are being placed where the action is instead of watching TV monitors.

Mastering Video Analysis
At the hardware subsystem level, the components of an IVS system are not markedly different from many conventional systems. They include digital video cameras that acquire the video data, encoders and video servers for processing, routers for transport control, DVRs for storage, monitors for viewing and software to manage the system.

A critical differentiating feature of an IVS system is that the information flowing through the network is not video in a conventional sense. It is metadata -- information about the raw video data that has been stripped of all redundancies and all irrelevant content. The original video stream can be simultaneously captured, compressed and archived for future retrieval and viewing.

The performance of a video surveillance system can be significantly improved by adding intelligence to strategic points in the network and using that intelligence to make decisions based on the metadata being transported though the network. For example:

• Video acquisition. Intelligence embedded in cameras can identify a target by type, and then pan and zoom to provide more detailed information.

• Data processing. The trend toward digital data is clear. The important next step is to process megabytes of raw video into metadata that contains only the information deemed important.

• Distribution. Intelligent security networks require less bandwidth because they are transporting kilobytes of metadata instead of megabytes of video. They also can route information to the right network node without human intervention.

• Decision-making. Smart systems can act on metadata according to rules-based algorithms created by security personnel using a simple point-and-click interface.

• Action and execution. Eliminating redundant or useless information and minimizing the data stream allows security personnel to monitor the system via a mobile device while patrolling, which typically puts them closer to the scene where they can be most effective.

• Archiving. Intelligent security facilitates the indexing and archiving of data for use at a later date.

Smart Cameras, Smart Networks
Today, surveillance systems typically acquire data using digital cameras and apply minimal image processing. The digital data is then converted to analog for distribution over the existing infrastructure. Once it arrives at the command center, it is converted back to digital data for analysis and, finally, storage. There is not much opportunity to act on the information along the way.

IVS systems are quite different. They are end-to-end digital systems. They use the IP network because IP transport offers so much more flexibility than traditional point-to-point analog transport. Each camera becomes, in essence, an intelligent Web camera that can analyze and act upon the video data at any point along the communication path.

In an IVS system, the processing of video data goes far beyond capture, digitization and video compression. Intelligent software could have additional capabilities such as:

• Content analysis. In any scene, the amount of valuable security information is dwarfed by background information, such as static elements like buildings and roads, and even dynamic elements, such as trees blowing in the wind or waves on water. This information can be filtered out, allowing the software to focus on legitimately moving objects, such as people and vehicles.

• Object identification. To be successful in isolating information of security interest, the software must be capable of classifying objects. It could, for example, identify a person, animal, vehicle or some other object.

• Activity analysis. Software can identify an activity of interest and compare it to a set of rules created by security personnel.

Image processing and statistical analysis techniques are capable of identifying objects, their motion and their activity. But this intensive number crunching requires a powerful processor. Digital signal processors (DSPs) are ideal for the task because they are designed to handle fast fourier and other algorithms that are used to describe images mathematically. Using these algorithms, DSPs create the metadata the software uses to inform security personnel of possible problems.

Creating Valuable Data
Creating metadata is a sophisticated process that turns the concept of conventional video upside down. Since surveillance video is viewed by computers, not humans, image enhancement is necessary.

Contrast is a good example of the requirements of computer analysis. Because motion detection and image recognition algorithms depend on identifying the edges of objects, a high-contrast image is best. With the advent of powerful but relatively inexpensive DSPs, that edge-sharpening algorithm can be implemented within the camera subsystem.

Tracking an object moving through the camera's field of view and running it through an activity inference engine that compares the activity against a set of rules typically leads to both a decision and, in some instances, a resulting security activity. The camera's positioning system may be instructed to focus, pan, zoom or turn the camera so the object can be tracked beyond the initial field of view.

These and other scenarios lead to an important hardware consideration that is virtually unknown in conventional video processing: balancing image processing with control functions. From a chip hardware perspective, this requirement mandates a dual-core architecture that includes a DSP core for image processing and a 32-bit microcontroller core for control functions.

The demands on the DSP-based boards that IVS systems require of cameras, DVRs, routers and other network subsystems are imposing. In the audio/video space, for example, the chip must be capable of handling multiple video and audio compression standards, such as MPEG-4, H.264, WMV9, AC3, AAC and WMA. Image resolution options must be flexible and are typically referenced in terms of either 720 x 480 pixels (D1) or 352 x 288 pixels (CIF).

The data processing engine also must be flexible enough to accommodate future algorithms automatically through software upgrades. This can only be accomplished by a programmable processor such as a DSP.

There also are a host of image processing algorithms, such as edge detection, Gaussian masks and a variety of filters, many of which are compute-intensive. The IVS application software implements these algorithms and constructs the information valuable to security personnel by using applications such as inference engines and content analysis engines.

Using TI's 600-MHz, 4800-MIPS DM642 as a benchmark, image processing requires a fraction of the computational power the designer has at his or her disposal. The actual amount varies widely depending on implementation, but 20 to 40 percent is a reasonable estimate.

Speaking Many Interface Languagues
Because it distributes intelligence around the network, IVS mandates support for numerous hardware interfaces. If a single dual-core DSP is to be chosen for all or most of the board-level subsystems to save on component and manufacturing costs, it has to support a glueless camera interface, various network connections, and memory and storage interfaces.

A DDR2 memory interface enables both quick storage of video data and video system flexibility. For remote video surveillance applications, the integrated ATA hard-drive interface is needed to provide a glueless connection to standard hard-disk drives, enabling localized video storage.

Networking requires additional peripherals, including an Ethernet MAC and USB interface because it has become the default plug-and-play method for connecting digital devices.

For the system developers that design the image processing and control board or IVS applications, a peripheral-rich processor solves many of their integration problems. But it also requires leading-edge application software and development environments bundled into development kits.

Moving to the Top
Video surveillance has moved to the top of many government and corporate priority lists in the past few years. The trend from analog to digital systems is well-established. But the widely accepted paradigm of moving compressed video to manned command and control centers also is a backward-looking strategy.

When compared to the bandwidth requirement of streaming video, surprisingly little image information is required to make highly informed choices when software replaces humans at the initial level of detection.

Using software intelligence, including content analysis engines and inference engines, to analyze metadata derived from the video has advantages at every level of the security system. Moreover, integrating as much of this intelligence as possible at the surveillance periphery -- in the camera itself -- further relieves the processing load, enhances the quality of the information and allows actions to be taken quickly.

Storing the relevant information as metadata on a DVR's hard drive also allows rapid forensic analysis of the data days, weeks or months after a security breach. With its wide range of applications and cost advantages, it is by no means hyperbole to call IVS a disruptive technology.