Augmenting Reality

Transforming communication for the first responder

Collaboration is an essential feature of all office environments.

However, what happens when your office is not in a cubicle, but rather in an emergency operations center that sprung up amidst the tremendous devastation of a hurricane, tsunami or battlefield? What tools do you need when your decisions are guided not by the eight-hour workday but by the minutes or seconds that remain before your team is to be deployed?

In these situations, collaboration quickly transforms from the best business plan to a means of survival for fellow human beings. The brave men and women around the world who serve as first responders have immense responsibility, and access to immediate and secure collaboration is vital to command and control -- and ultimately to the success of these missions.

My job is to provide the tools that first responders need to successfully complete these missions. The most substantial step we’ve taken in developing command and control solutions for first responders is using the Natural User Interface, or NUI, focusing on current and emerging technologies and platforms, such as Microsoft Surface, Microsoft Windows 7, as well as mobile development on devices like the iPhone and BlackBerry.

What is NUI?
NUI allows users to directly interact with content using common hand gestures.

It removes the constraints of the application’s graphical framework, replacing it with a user interface that is both physical and invisible. Users perform tasks through direct touch, and the content becomes the interface. NUI eliminates the proxy controls of the keyboard and the mouse, and it increases the overall immersion for the user. NUI applications are navigated differently as well, navigating intuitively through human assumptions, trial and error, and learned interactions. NUI has combined the concepts of reality and super-reality. For instance, if the user has a photograph lying on the table, he or she can move it, rotate it and flip it. This is a concept of reality.

However, with NUI you can perform those actions, as well as other actions that seem intuitive, in a computer-based environment -- for instance, increasing or decreasing the size of the image by performing certain hand gestures. That is super-reality.

Command and Control
One of NUI’s greatest strengths is its ability to take large amounts of raw data from diverse data sources and provide it to the user in an easy-to-understand, simplified, visual manner. By removing the constraints of a GUI-based application and allowing direct interaction with content, users can process more information faster, and in proper context. These collaborative tools have the ability to transform communications for first responders.

With a cutting-edge application module developed specifically for first responders, emergency personnel could maintain a constant communication link. With a GPS feed coming back to the command center from the farthest edge of the response, decision makers would have real-time information as to the location of staff and equipment assets and could then minimize the time it would take to relocate assets to the areas where they are needed most. Chat and instant messaging would enable a constant flow of imagery coming from an iPhone in the field back to the command center, where it could be evaluated immediately and acted upon, reducing the time it takes for orders to flow from decision makers.

These orders could be accompanied by some type of data set that could also flow in real time, back to the iPhone in the field. The fluid transfer of information in these types of scenarios ensures first responders can act quickly and decisively, ultimately saving lives.

The NUI environment changes the social dynamic. It’s gestural, multi-touch, multi-user, has a 360-degree canvas and environment, and it can directly interact with objects. All of these concepts provide new challenges for us as developers.

We must work through numerous inquiries in order to provide a proper social and collaborative environment.

For instance, how do we handle users’ identification? In a command and control scenario, it is essential to know exactly whose hand is interacting with the application.

After we verify the users, how do we track them and what specific functions they use throughout their entire session?

Session management has the potential to be a big problem when you’re working in the multi-user environment. Orientation becomes important, and the controls need to be fluid and mobile as well as 360-degree accessible. We’re now designing for both the interaction of the system and direct interaction of users.

In future emergency response scenarios, augmented reality will likely play a vital role in command and control. Command center personnel will have the ability to overlay data sets directly on top of a live video feed or other imagery. They will have the ability to send real-time information to a first responder, miles away, who will perhaps be wearing a high-tech pair of sunglasses, where these data sets will be in his field of vision along with whatever is in front of him.

This data, which could conceivably include the arrival time of additional resources or dangerous areas to avoid, will provide the first responder with vital information to aid him as he makes decisions that literally result in life or death.

Technology is ever changing and evolves at a rapid pace. Datasets are becoming increasingly complex, and so is the demand for more intelligent systems. Looking to the future, the distance between users and content will continue to narrow, and because of this we will see platforms that provide both more tactile and visual functionality. Augmented reality will become very prevalent and part of our day-to-day work environment.

This article originally appeared in the issue of .


  • 12 Commercial Crime Sites to Do Your Research

    12 Commercial Crime Sites to Do Your Research

    Understanding crime statistics in your industry and area is crucial for making important decisions about your security budget. With so much information out there, how can you know which statistics to trust? Read Now

  • Boosting Safety and Efficiency

    Boosting Safety and Efficiency

    In alignment with the state of Mississippi’s mission of “Empowering Mississippi citizens to stay connected and engaged with their government,” Salient's CompleteView VMS is being installed throughout more than 150 state boards, commissions and agencies in order to ensure safety for thousands of constituents who access state services daily. Read Now

  • Live From GSX: Post-Show Review

    Live From GSX: Post-Show Review

    This year’s Live From GSX program was a rousing success! Again, we’d like to thank our partners, and IPVideo, for working with us and letting us broadcast their solutions to the industry. You can follow our Live From GSX 2023 page to keep up with post-show developments and announcements. And if you’re interested in working with us in 2024, please don’t hesitate to ask about our Live From programs for ISC West in March or next year’s GSX. Read Now

    • Industry Events
    • GSX
  • People Say the Funniest Things

    People Say the Funniest Things

    By all accounts, GSX version 2023 was completely successful. Apparently, there were plenty of mix-ups with the airlines and getting aircraft from the East Coast into Big D. I am all ears when I am in a gathering of people. You never know when a nugget of information might flip out. Read Now

    • Industry Events
    • GSX

Featured Cybersecurity


New Products

  • PE80 Series

    PE80 Series by SARGENT / ED4000/PED5000 Series by Corbin Russwin

    ASSA ABLOY, a global leader in access solutions, has announced the launch of two next generation exit devices from long-standing leaders in the premium exit device market: the PE80 Series by SARGENT and the PED4000/PED5000 Series by Corbin Russwin. These new exit devices boast industry-first features that are specifically designed to provide enhanced safety, security and convenience, setting new standards for exit solutions. The SARGENT PE80 and Corbin Russwin PED4000/PED5000 Series exit devices are engineered to meet the ever-evolving needs of modern buildings. Featuring the high strength, security and durability that ASSA ABLOY is known for, the new exit devices deliver several innovative, industry-first features in addition to elegant design finishes for every opening. 3

  • ResponderLink


    Shooter Detection Systems (SDS), an company and a global leader in gunshot detection solutions, has introduced ResponderLink, a groundbreaking new 911 notification service for gunshot events. ResponderLink completes the circle from detection to 911 notification to first responder awareness, giving law enforcement enhanced situational intelligence they urgently need to save lives. Integrating SDS’s proven gunshot detection system with Noonlight’s SendPolice platform, ResponderLink is the first solution to automatically deliver real-time gunshot detection data to 911 call centers and first responders. When shots are detected, the 911 dispatching center, also known as the Public Safety Answering Point or PSAP, is contacted based on the gunfire location, enabling faster initiation of life-saving emergency protocols. 3

  • QCS7230 System-on-Chip (SoC)

    QCS7230 System-on-Chip (SoC)

    The latest Qualcomm® Vision Intelligence Platform offers next-generation smart camera IoT solutions to improve safety and security across enterprises, cities and spaces. The Vision Intelligence Platform was expanded in March 2022 with the introduction of the QCS7230 System-on-Chip (SoC), which delivers superior artificial intelligence (AI) inferencing at the edge. 3