The Edge of Intelligence

Why today’s network cameras are so smart

• By Robert Muehlbauer
• May 01, 2013

When the first network camera was introduced in 1996, its functionality was pretty bare bones: digitize images and send them across the network to a centralized video management system. Its primary purpose was “Web attraction,” where someone could log on to an Internet site and view live situations, such as weather or traffic. But, as resolutions and frame rates improved, it didn’t take long for manufacturers to realize the potential of the camera’s built-in processor and redefine its role in security and surveillance deployments.

By the mid-2000s, manufacturers introduced the first intelligent algorithm to reside inside the camera—and since then, the industry has never looked back.

Processing Power: In-camera is in Demand

With processing power growth exceeding Moore’s Law in surveillance by more than doubling every 18 months, intelligent analytics previously relegated to the video server are now being performed in-camera. The cost-efficiencies of this shift are significant:

• Lower bandwidth consumption—only pre-processed video is being sent across the network rather than massive amounts of constantly streaming footage that must be analyzed on the server side.
• Lower storage requirements—only content-rich video is being streamed to the video storage array rather than every frame that was captured by the camera.
• Lower operating costs—in-camera processing is less expensive than monopolizing CPU cycles on the server.

Motion Detection: Spotting an Incident in Real Time

The first intelligent algorithm brought inside the camera was motion detection. Pre-processing the video in-camera meant the system could be set up so that no video would be sent over the network unless there was movement detected in the field of view. Not only did this analytic cut down on bandwidth usage, it saved on storage since non-event video wasn’t streamed to the server for archiving. On the backend, users saved on CPU cycles because the data center no longer had to analyze all the footage for video motion before triggering an alarmed event.

A common practice of business owners has been to activate motion detection for after-hours surveillance, curtailing video storage costs. For instance, a store owner can stream and save all the video captured during the 12 hours the store was open for business. But once the store was closed for the day, the motion-detection algorithm would only record video if an event occurred. By strategically leveraging the analytics, the store owner could conceivably cut video storage costs in half.

Tamper Alarm: Protecting the Field of View

The next algorithm to migrate to the camera was an analytic to detect tampering attempts. If a camera knocked away from its normal field of view, was bumped out of focus or the lens was blocked—spray painted or covered by an object—it automatically sent an alert to the video management system to notify security of the problem. This was a significant advantage over analog systems, and is still an underused intelligent analytic today.

The camera detects tampering by essentially memorizing the pixel make-up of the scene it’s been trained on, and sending an alarm when the pixels change dramatically. Putting this ability in-camera instead of waiting for the video management system (VMS) to run periodic diagnostic cycles on edge devices assures more immediate alerts about a camera’s sub-performance. If the camera isn’t working or its vision is blocked, the user will be notified.

Tampering alarm analytics have been effective in schools where vandalism and playing pranks with campus cameras are commonplace. Similar to motion detection residing in-camera, the tampering algorithm will automatically alert the VMS to the problem, saving the process-intensive analytic from having to run on the server. But equally important is the immediacy of the alert, which would improve the administration’s chances of catching the culprits in the act and ensuring the camera can see once again.

Cross-line Detection: Monitoring the Perimeter

Another analytic to take up residence in-camera was cross-line detection—monitoring breaches in a perimeter. The breach could occur in either direction— somebody trying to break into a facility or trying to exit the site without authorization, such as a prison inmate attempting an escape. Cross-line detection also could be used as a safety measure, for instance, on a subway platform to ensure no one crossed onto the track. In any case, the camera was now smart enough to detect when a person, vehicle or object crossed the invisible line and send an alarm to the video management system to alert security staff to the event.

Oftentimes, outdoor perimeter installations face challenges with connectivity and bandwidth. Running cross-line detection in-camera provides the means to trigger an immediate alarm, improving security’s chances of apprehending the individual or protecting people who enter restricted areas. Conversely, if security has to wait for the back-end server to process the cross-line detection, the window of opportunity shrinks considerably.

Onboard Storage: SD and microSD Memory Cards

Though the price of storage continues to drop, the build out of server farms to support storage comes at considerable expense, including the increased power consumption to cool an ever-growing collection of servers. In contrast, many of the newer network cameras include slots for SD and microSD memory cards that can store from 32 to 64 GB of video. As Moore’s Law continues to drive down the cost per capacity and physical size of storage devices, cameras and encoders will soon support 2-plus TB of storage. In the realm of video surveillance, this translates to about 55 days of 1080p HDTV video running at 30 frames per second using highly-efficient H.264 video compression.

Greater storage capacity, coupled with increased processing power, opens a whole new range of possibilities for the surveillance industry. For instance:

Self-contained solutions. Individual cameras and video encoders can now be used as complete security systems unto themselves. This provides a cost-effective solution for environments with limited infrastructure options. By embedding the video management system in-camera, users could avoid the cost of external storage and still remotely access the onboard video directly. In outlying areas with limited bandwidth or environments hostile to sensitive computer technology, robust self-contained cameras could provide the eyes, ears and analysis of the scene, as well as store any forensic evidence. For instance, law enforcement could deploy more standalone cameras in remote areas of a city without increasing their existing centralized storage capacity.

Redundancy for network outages. With greater storage capacity in-camera, manufacturers will be able to offer a level of fault tolerance for network outages previously unattainable—even with applications that require high resolution and real-time recording rates. The camera now contains the intelligence to detect an outage and automatically begin storing the images in-camera until the connection is re-established and video can once again be streamed over the network. This is invaluable in deployments, such as casinos where coverage is mandated by law and any loss of video would incur stiff penalties.

Open Application Platforms: Is There an App for That?

Leading IP cameras and video encoders now set aside a certain amount of memory and CPU cycles dedicated to running applications in-camera. Some manufactures also support open application platforms that allow third-party software developers to write custom analytics that can run in-camera. This is creating an Appculture in surveillance like the one pioneered by the iPhone. A few popular third-party analytics include:

People and vehicle counting. These analytics provide customer and operational insight in real time— everything from the volume of foot and vehicle traffic during specific hours to customer dwell time and suspicious loitering. This helps store managers properly staff their operations and manage customer interactions with a higher level of service. Some universities use object counting to help determine when a parking garage is approaching capacity on game day.

License plate recognition. These analytics record license plate images and compare them to images stored on an internal SD memory card. If any plate image matches a user-defined watch list, the camera automatically sends an alert to the video management system to inform security of any suspicious vehicles. To work properly, however, this application requires a camera with high resolution, good low-light capabilities and a powerful processor.

Line management. This helps users facilitate the flow of people in line and enhance customer service— whether at a checkpoint or a checkout line. The software links to an indicator line that tells an individual where to stop and when to proceed to the next available booth, desk or station. The analytic tool can also be used by management to measure line length and trigger an alert to open another checkout bottleneck occurs.

These analytics were developed for the security industry by some of the top surveillance developers and VMS manufacturers. However, the next phase of third-party applications will be the result of attracting and encouraging developers from outside our industry by opening their eyes to the opportunities in video surveillance. Not only will this create more competition to foster innovation, it will open up the market to creative non-traditional surveillance uses.

As an industry, we should build partnerships with academic institutions and support youth initiatives like the FIRST Robotics Competition. This will empower the next generation of software developers to pursue careers in surveillance by providing an early introduction to the true potential of in-camera intelligence. Such initiatives are already starting to spark some amazing and imaginative applications beyond traditional security and surveillance—for everything from analytics that help supervise processes on manufacturing lines to algorithms designed to assist healthcare providers in monitoring patient wellness.

The Future: Higher IQ Cameras

In recent years, the focus in network cameras has centered on image resolution. Having progressed rapidly from the grainy black-and-white images of the analog CCTV days to the megapixel and HDTV full-color fidelity of today’s IP systems, is it reasonable to project 50 and 100GB image processors within the next decade? At some point, excellent image quality will exceed what the eye can perceive and further resolution-based advances will become a point of diminishing returns.

It seems more realistic to expect that increased processing power will be directed towards expanding intelligent analytics at the edge. What might some of those more process-intensive applications be?

Facial recognition. Right now, facial recognition algorithms require too much processing power to be handled in-camera. But, with higher processing performance and greater storage capacity being crammed onto ever-smaller real estate, it’s reasonable to assume that performance will soon reach a level where incamera facial recognition analysis will be possible.

Metadata analysis. Extracting metadata directly from the video in-camera appears to be the next big breakthrough on the horizon. Being able to search that metadata—for everything from the type of object to the particular color of clothing—would be significantly faster and far less expensive than searching through the entire video archive. With sufficient metadata you could narrow the search by designating the type of object you’re looking for, whether it is a person, vehicle, animal or something inanimate like a rock. You could even search the video by the size of the object and the direction the object was moving.

Smarter is Better

Today’s network cameras have matured from the passive eyewitnesses of yesteryear into intelligent computers that can evaluate what they see. The increased processing performance has enabled network cameras to take on a more vital role in analyzing, managing and acting on events within their security systems. By shifting more intelligent applications to the edge, users not only save in bandwidth consumption, storage and central processing costs, but can process events and alerts in real time. This enables faster searches for suspected activity and increases the likelihood of resolving incidents more quickly— turning electronic surveillance from a reactive to a proactive industry.

This article originally appeared in the May 2013 issue of Security Today.