Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now


Real-World Facial-Recognition Tech Catching Up to Its Fictional Counterpart

Tracking suspects might become a largely automated process

Tracking suspects might become a largely automated process

Wayne ColeNetwork television crime shows such as “CSI” and “Person of Interest” frequently use fictional facial recognition technology as a plot device to identify wrongdoers, but in fact such technology might be right around the corner.

Current real-world systems work well when comparing “normalized images” like the photos on drivers’ licenses, passports and, of course, mug shots, all of which are similarly posed, lighted and resolved. However, the real world and TV fantasy diverge when it comes to poorly lit, low resolution, anything-but-head-on imagery from security cameras, cell phones and other consumer devices.


Since the late 1990s, many agencies that deal with photo identifications—driver licenses, passports, workplace badges and military IDs—have gone online, and a 2009 report indicated there were nearly 4 billion hours of security camera footage shot by more than 30 million cameras each week in the United States alone.

Some organizations also have engaged in separate projects to put older data online, with the unintended results of huge image databases containing duplicate images of the same person. For example, a moderate-sized Florida sheriff’s department reportedly has more than 7.5 million facial images in its database. Identifying duplicate subject images and connecting them to, or eliminating them from, an individual’s records is not just a matter of decreasing workloads on examiners, investigators and facial recognition systems; it is essential for finding false identities and exposing identity thefts involving documents containing photographs.

In addition, the U.S. Department of State has some 100 million passport images that are being “de-duplicated” to track fraudulent documents. An effective FRS could detect subjects that might have passports under different names, thereby enabling investigators to separate the innocent (those who have passports with changed names because of court order or marriage) from the nefarious (including drug smugglers, gun and human traffickers, terrorists or spies). Clearly states would have the same interest in de-duplicating driver license and other photo ID databases, but for many agencies de-duplication happens only by accident, usually when an astute investigator stumbles onto and digs into look-alike images. Law enforcement agencies might be more quickly alerted when an alias is being used if a cost-effective, automated FRS is available.


Current FRS technology has computerized methods of comparing facial “geography,” including the relative positions of the eyes, nose and chin, as well as significant marks such as scars, moles, birthmarks and tattoos.

This is typical of what a facial recognition system, or technician, deals with in the real world. The image on the left is from a database of normalized images that have well-lit and focused frontal views. However, the subject on the right is not facing
full front (or even in a full profile), and his face is partially obscured while his features look to be distorted by extreme emotion.
The FRS would quantify and store the facial geography of each image, and when comparing stored imagery to a source image, the FRS would produce a similarity score for the source image. Scores above a set threshold would be considered a potential match and those images would undergo scrutiny by a human examiner. But the accuracy of that procedure—and, in many instances, its admissibility as evidence—depends on having two “normalized” images for comparison. That is, the images need to have the same pose, similar expression, equivalent lighting, good resolution (i.e., no visible blurring or lens distortion), and be close enough in age for the similarity in overall appearance to be obvious.

Unfortunately, bank robbery suspects often wear hats that cast shadows onto the suspects’ faces. In addition, bank cameras routinely are mounted near the ceiling, and low-resolution and extreme wide-angle lenses can distort face images. Suspects also usually display expressions of extreme emotion while shouting instructions to the employees and customers. Such an image being used to compare against photos from driver licenses or mug shots would be almost useless.

In such a situation, investigators would be lucky if there is a single, nearly head-on shot that could be pre-processed into an image and used to get good information from the FRS. But that would require intensive work by a skilled image enhancement technician, and such technicians are far more numerous on TV than in actual investigative agencies.


The Institute of Electrical and Electronics Engineers has been reporting on research and development progress in facial recognition. According to the IEEE, some projects focus on source image normalization while others are attempting to use composite sketches as FRS input.

A promising effort involves the quantization of facial images as 3D topographies instead of 2D geographies. With such information attached to all the images in a database of known persons, the known face can be rotated and distorted for differences in pose and expression when compared to a query image. An automated system with that ability would lower a technician’s pre-processing to simple lighting and resolution enhancement.

Currently, the same procedure for creating a 3D facial mesh, mapping an image to it, then rotating and distorting that model to compare to a database of normalized images is very expensive and time consuming. As a result, budget-strapped agencies are unable to maximize the use of available facial image data.

Aging is another area where increased accuracy has been achieved with the 3D facial mapping approach. Standard age compensation approaches available in some commercial off-the-shelf software can work from normalized facial images—that is similar pose, lighting and expression—and achieve matching accuracy over 90 percent when the age difference in the images’ subject is less than a year. However, when there is an age difference of 10 to 20 years in images of a subject, accuracy can drop below 70 percent. It only gets worse as the age difference increases, especially if the transition time spans from childhood to adulthood. However, applying aging algorithms to a 3D facial map appears to improve the result significantly over longer age ranges.


The merger of electronic background investigations and real-time electronic “tailing” is moving technology closer to the fictional machine used in crime dramas.

In addition, a federal district court ruling may have established the legality of such warrantless electronic monitoring. The decision said that using the GPS on a “burner” cell phone to track a suspect’s movement did not require a warrant since it was like using a license plate to follow, or locate, a suspect in a public space where there is no reasonable expectation of privacy. A more recent decision admitted warrantless surveillance video from cameras set up by law enforcement in the woods around a subject property. It is likely that the same legal arguments will apply to faces recorded on video surveillance.

With the deployment of some of those newer facial recognition technologies, tracking suspects around a city, or the globe, may become an accurate and largely automated process sooner than expected.