Video Metadata: Describing the Details that Matter

• By Joe Danielson
• Dec 01, 2022

Modern surveillance systems generate an overwhelming (and mostly unused) amount of data. This is especially true when recording video in 24/7 operations, which is essential to capturing evidence, incidents and events. It is not only hard to pick out what really matters in a scene, but also extremely time consuming. Making data more identifiable and actionable is a key problem to solve. Applying metadata to describe key details in a scene allows data to be more identifiable and actionable.

This is why metadata is the foundation for gathering intelligence from surveillance video and/or audio streams. Metadata provides a fast way to find, evaluate, and act on the singular details that matter the most through one, hundreds or thousands of video and audio footage streams. Metadata is now an essential part of effective security and business operations.

What is Metadata?
Typically, Metadata is referred to as ‘data about other data.’ In the context of video surveillance, that translates to ‘data about video data’. Video metadata accurately describes the details that matter in a scene. For instance, attributes for metadata can describe all sort of details about moving objects of interest, e.g. location, time, colors, sizes, shapes, coordinates, volume decibels, speed, direction, etc.
Additionally, foundational details can be added, such as video stream description, codec, time stamps and device identity.

The aforementioned are ‘meta’ descriptions of details in, or related to, a scene. Based on AI machine and deep learning, Meta descriptions can be more (or less) granular. This allows for classifying a group of pixels as a person, animal, vehicle or other pre-defined object classes. Being more precise with more refined descriptions of people or objects e.g. vehicle type, make model, color, speed, direction, etc.

The Value of Metadata
Metadata not only provides details about people, objects and events in a scene. It also allows large amounts of video and recorded footage to quickly group, sort, search, recover and use. As a result, the overall use cases for metadata fit into three areas.

1. Real-time alarm triggering and notifications
2. Post event forensic searching
3. Statistical analysis and reporting

Adding Intelligence to Scenes
Metadata essentially assigns digital meaning to each video frame about the objects and events within it. In other words, it adds interpretation or intelligence about the scene rather than just the raw video footage, which needs to be processed manually by an operator.

Once software can interpret scenes in this way, it can understand the scene details and enable the scene to be acted upon in real-time via events, after events (post-event), via manual search or simply analyzed for statistical analysis. This enables the use of metadata to design baselines that define what is ‘normal’ for any scene feed from any individual camera. In turn, this allows software to recognize any degree of deviation, anomaly or specific behavior or activities, etc. as well as predict what will happen in that scene to a specific probability.

Harnessing the Full Potential of Metadata
Video metadata adds immense value to a video management system. In fact, its true potential is realized when applied to multiple inputs spanning visual, audio, activity, and process-related inputs. In the management of any site, things like RFID tracking, GPS coordinates, tampering alerts, noise detection, and point of sale transactional data, are all high value data sources. Unifying this metadata generated from many different sources means gaining much more insights than one can ever get from each system alone. Interoperability is key, and open-protocols and industry standards are essential to this effort. Ultimately seamless metadata integration will allow us to harness massive amounts of data from all kids of systems and gain a greater understanding of everything around us.

This article originally appeared in the November / December 2022 issue of Security Today.