Evolution of the Face
Technology puses science fiction out the door
- By Paul Schuepp
- Apr 01, 2013
Ten years ago, face recognition sounded like science fiction.
Biometrics programs were expensive, not very accurate
and largely the domain of researchers and specialists buried
in the labs of government agencies and private companies.
Now, biometrics is the most important security weapon of
all—for detectives, soldiers and intelligence officers. As in industry,
we’ve gone from working with highly-posed, mug shot-style images
and limited photo banks to accurately mapping “faces in the wild”
with mobile hand-held devices for instant checks against massive
cloud-based databases.
Are our faces now like bar codes, easily scanned and instantly
traceable? Not quite. But face recognition has grown into a tool that
every investigator can benefit from. From U.S. Border Control and
Homeland Security agents to local police, soldiers in the field and
Department of Motor Vehicles clerks, there’s a need to identify faces.
Our ability to make faster, more accurate facial identifications regardless
of challenging conditions is helping law enforcement, military
and security officials do their jobs more effectively.
From Mug Shots to Surveillance
Government has always been, and remains, the biggest user and innovator
in the field of biometrics. A diverse array of governmental
agencies are continually looking for more ways to put facial recognition
to good use, mostly for criminal pursuit.
In 2004, mobile facial recognition was barely on anyone’s radar but
biometrics really became a game changer in 2006 in Iraq, according to
a recent annual report from the U.S. Biometrics Identity Management
Agency (BIMA). That’s when the tool started to develop into a genuine
weapons system. This came about largely through the development of
rugged, portable and tactical devices. Battlefield deployment capability
was essential to the growth of facial biometrics.
Face recognition really started with an Army program—the BAT,
or Biometrics Activated Toolset—designed to gather enrollment images,
or pictures taken when a person is booked or detained. The BAT
also acquired fingerprint and iris biometrics. (BAT later became part
of BIMA, which is part of the Army Provost Marshal.) The people who
founded this database had a vision. They took five pictures of each suspect:
one frontal, two profiles and two angulated pictures, left and right.
It was a solid, forward-thinking start. The problem? They didn’t have
a powerful facial recognition search tool for matching the images to a
larger database. That’s where companies like Animetrics soon waded
in, designing programs to work with these five pictures. The legacy facial
recognition systems could only process one frontal image.
Since then, many development cycles have passed. The biggest innovations
over the past decade have come in mobile and cloud technology,
and in the ability to work with challenging images. We no
longer need forward-facing images to compare to other static images;
we have the technology to rotate faces—known as “pose correction—
and to turn flat 2-D images into 3-D avatars. These advancements
have given us the capability to work with challenging images, from a
variety of sources, such as surveillance footage, security cameras and
smartphones. The images we work with now are largely uncontrolled,
unlike the old mug shot or driver’s license pictures.
Thanks to the Internet and explosion in digital and mobile cameras
and video devices, there are so many more images to work with
in the world, and a growing number of processing tools. The Web is
now the major source for investigations. Ironically, it both fuels and
fights crime.
Mobility is critical now. Agencies want investigators in the field
to have biometric technology available at just the touch of a phone
button. And it’s possible. Now, it’s all about integration, custom application
development and harnessing the infinite processing power
of cloud technology. Intelligence agencies are using video analytics to
scour and analyze tons of online images, collecting, indexing, clustering
and analyzing faces.
Accuracy has greatly improved, even as images become less precise.
For the industry as a whole, you still get the best results with
controlled imagery against a controlled database: a 99.7 percent identification
rate, at a false accept rate of 1 in 1,000. (When heads are
turned, the rate falls to 40-50 percent.) But the ability to rotate uncontrolled
images and create avatars produces more accurate matches—
an identification rate of 90 percent or higher at that same false
accept rate. It’s not DNA and it’s not yet admissible in court, but these
matches are incredibly effective in providing new leads for traditional
investigatory follow-up.
Additionally, accessibility to facial recognition tools has increased
greatly in the past decade. Biometrics screening used to be the purview
of maybe one trained specialist in a platoon, or a technician in a crime
lab. Now, we have the capability to give every soldier, border agent and
police officer pocket-sized biometric tools—handheld devices that
can be carried everywhere to make speedy identifications in the field.
The implications for a soldier or officer of immediately knowing who
they’re dealing with—and the ability to instantly match suspects to international
watch lists and criminal records—are revolutionary.
Doing the Math
The science behind these advancements is based in mathematical
concepts that many of us already possess a rudimentary understanding.
It’s all about the math—mapping the face along the three-dimensional
X, Y and Z axes.
Like most biometrics, such as fingerprint and iris matching, basic
face recognition begins with a sensor-created biometric. In this case,
the sensor is a camera that produces a photograph or video frame.
Then, the digital image is scanned by face-detection software to find
the faces, usually starting with the eyes in locating facial features.
Once a face is determined, a set of measurements are created by algorithms,
reducing the photo to a facial biometric template, usually
consisting of a vector of many numbers. These templates are stored in
a file, or an enrollment gallery.
Now, the application can search that gallery for any given face,
seeking a match or close likeness. Searches are conducted by matching
algorithms that compare the given—or suspect—face to the gallery
templates, using a statistical analysis process. Matches are not binary,
for example, match or no match. The matching is completed by finding
a gallery face—or several—that are statistically very close to the given
face, using a weighted score in reference to the other database subjects.
We now have the technology, through the “computational anatomy”
developed by Animetrics founder Dr. Michael Miller at Johns
Hopkins University, to take these images a stage further. Basically,
2-D images are converted to 3-D images by taking the X and Y coordinates,
and synchronizing them into X, Y and Z coordinates. These
3-D templates, which visualize the subject’s facial structure, geometry
and texture, allow for precise comparative analysis of facial features
between a subject face and a larger database. Thus scars, moles, tattoos
and distance measurements between facial features are compared.
The technology is especially effective in increasing matching
accuracy with uncontrolled imagery, such as faces with pose angulations
or those obstructed and shadowed by severe lighting. The 3-D
models can be viewed from any angle, rotated into any plane or used
to create a wireframe image for overlay by a second image for comparison.
These advanced capabilities have greatly expanded the usefulness
and practicality of face recognition.
Street and Field Biometrics
The use cases for facial recognition are growing steadily, as the technology
develops. So many people have smartphones, Androids and
iPhones now, devices that are essentially little computers in our pockets.
Biometrics tools are more accessible than ever before. But the
Department of Defense (DoD), Homeland Security, law enforcement
agencies and the international intelligence community, and forensic
labs around the world, remain the biggest users of facial recognition.
From the FBI and NSA, to the CIA, local police to the intelligence
agency of each armed forces branch, everyone has a use for this technology.
Accurate identity resolution is an absolute necessity on the
modern battlefield, where the ability to track terrorists remains crucial.
It’s also become an important tool for fighting crime on the home
front. Though agencies are largely differentiated by their databases—
some of which are shared, some of which are not and some of which we
wouldn’t want shared—they all share a commitment to the technology.
Here are just a few use cases.
State Department of Motor Vehicles
One hugely successful application of facial recognition is finding duplicate
licenses in department or registry of motor vehicle databases.
Almost all states now run these applications, which gained popularity
after the attacks of Sept. 11, 2001. Local offices now issue temporary
licenses to applicants, which gives staff a few days to research a
person’s information, answering the question: does this driver have
multiple identifications using the same face? Applications scour the
motor vehicle license databases, measuring facial biometrics to identify
duplicates, which can be used for fraudulent bank loans, check
cashing and job applications.
This technology is useful both in preventing the issuance of duplicate
identifications and also in identifying drivers on the road. The
day may come when law enforcement can take the picture of a person
pulled over without a driver’s license, and instantly compare that image
to the state DMV database. The officer will immediately know if
the driver does indeed have a valid license, and if the person really is
who he or she claims to be.
The Military
Facial recognition technology is crucial to DoD, its military branches
and intelligence gathering operations. Military investigators routinely
use facial recognition as a search engine for processing imagery
from surveillance cameras, smartphones, biometric sensors and
hidden devices, checking captured faces against international watch
lists of terrorists and criminals. It is especially critical for agents and
soldiers in the field and check points, constantly on the look-out for
suicide bombers and terrorists, to be able to quickly identify suspects.
While other biometric measures, such as fingerprints, palm prints or
irises, are great for making positive identifications, they’re not always
feasible to collect. Facial biometrics can be used to investigate subjects
in a more covert, less intrusive manner, alerting investigators
that additional action may be warranted.
Rugged handheld hardware and smartphone or tablet devices
designed for use in military intelligence are making in-the-field face recognition a reality, along with remote access to cloud-based databases.
Powerful facial recognition tools can now run as localized
applications, reducing soldiers’ dependency on technology at their
home bases and allowing instant access to watch lists.
Access and control is another key application for government use.
Facial recognition can be used to monitor access to countries, for example,
in border control. But it also can be used to create smaller
lists of internally “approved” faces, for access to Homeland Security
or Transportation Security Administration offices and buildings. Additional
benefits stem from providing government contractors access
to restricted areas. Meanwhile, Security View Systems also are
becoming more prevalent in restricting access to schools and other
security-sensitive buildings.
Law Enforcement
Facial recognition is growing quickly in law enforcement agencies
of all sizes. From Pennsylvania’s statewide JNET system, which allows
all trained officers to compare enhanced 3-D images against a
massive 3.5-million image database, to the Zurich police’s innovative
use of 3-D facial rotation to help victims better identify suspects,
criminal investigators around the world are putting face recognition
to work. Prisons are also making their enrollment databases more
widely available to those working with biometrics in the field.
The FBI is definitely leading the way, with its $1 billion Next
Generation Identification (NGI) program, designed to allow law
enforcement agencies around the country to compare suspect photos
in a massive, national database, beginning in 2014. In the meantime,
the agency also is working to offer free facial recognition software
to law enforcement: the Universal Face Workstation (UFW).
From Sci-fi to Future
Facial recognition will continue to grow in speed, accuracy, accessibility
and use. As the industry gets better at identifying faces from a
distance—gathering images less intrusively—and continue improvements
to surveillance footage and uncontrolled images, video analysis
will continue to grow in importance.
The future is pocked-sized and mobile. In five or 10 years, it’s
possible a soldier will be able to identify everyone around himself
in seconds, merely by holding up a palm-sized device linked to augmented
information in real time. Mobile will be the vehicle for all
communications capabilities, and even most applications for people,
businesses, educators, authorities and law enforcement. And mobile
facial applications will be the norm for security, identity authorization,
investigations and social networking.
Many new applications are on the way, particularly in the areas of
locating missing and exploited children, catching human traffickers
and identifying victims. The near infinite scaling of cloud technologies
will open a wealth of opportunities for unlimited use and expansion
of facial applications, including delivery as Web services and the
indexing of unlimited numbers of photographs in online databases.
We are entering an age of even more information, cooperation
and remote access. Linking field and regional data to national databases
will be critical for faster identification of both our own people
and our enemies.
This article originally appeared in the April 2013 issue of Security Today.