DeepFace

• 1. Introduction

  ›

  DeepFace is a facial recognition system built by Facebook's research team that can identify a human face with 97.35% accuracy. That number sits just below the 97.53% benchmark for human beings doing the same task. The gap is smaller than most people would guess, and in some conditions the machine beats the person. How did a social media company end up at the frontier of face recognition? What does a nine-layer neural network trained on four million images actually do to a photograph? And when more than a billion people's face scans were on the line, who decided what to delete, and what to keep?

• 2. Building the System

  ›

  Yaniv Taigman joined Facebook when the company acquired Face.com in 2012, bringing expertise that would anchor the DeepFace project. He worked alongside Facebook research scientist Ming Yang and Lior Wolf, a faculty member from Tel Aviv University. Together they built a neural network with over 120 million connection weights, trained on four million images uploaded by Facebook users.

  The network processes a face through four sequential modules: 2D alignment, 3D alignment, frontalization, and finally a neural network that outputs a 4096-dimensional feature vector. That vector is the system's internal fingerprint of a face. To recognize someone, DeepFace compares the vector against stored vectors of known individuals and finds the closest match.

  The 2D alignment stage begins by finding six specific points: the centers of the eyes, the tip of the nose, and the corners and center of the mouth. These anchor points let the system warp the image into a standard orientation. Flat geometry has limits, though. Faces tilted out of the camera plane defeat 2D correction alone.

  For those cases, a 3D model with 67 fiducial points takes over. The system fits a virtual camera to the image that minimizes distortion, then uses that camera to pull the face into a frontal-facing view. The frontalization stage reduces remaining errors by blending portions of the image with their mirror-symmetric counterparts. The 2014 research paper describing the system added a final classification layer capable of sorting an input into one of 4030 known identities.

• 3. Competitive Landscape

  ›

  The FBI's facial recognition system operates at 85% accuracy. DeepFace, at 97%, was a meaningful step beyond what law enforcement had deployed. The standard benchmark used for comparison is called Labeled Faces in the Wild, a data set designed to test recognition under real-world conditions.

  Google's FaceNet surpassed DeepFace on that same benchmark, reaching 99.63% accuracy. FaceNet draws on images from Google Photos, giving it a similarly vast training pool. Neeraj Kumar, a researcher at the University of Washington, attributed Facebook's performance to scale: access to enormous numbers of labeled images produces what he called a higher-capacity model, one that generalizes better than systems trained on smaller data sets.

  AI researcher Ben Goertzel described DeepFace as having "pretty convincingly solved face recognition," while cautioning that deep learning alone is not a complete path to artificial intelligence. Facebook did not publicize DeepFace's capabilities through press announcements when the research paper came out. According to a Slate analysis, the company stayed quiet because it feared another wave of negative headlines about surveillance and privacy.

• 4. Privacy in the EU and Beyond

  ›

  When DeepFace first rolled out, Facebook users were enrolled by default. They could turn the feature off, but they were never told it had been switched on. That design choice blocked the technology's release in the European Union. EU data protection law requires that users affirmatively consent before a company collects their biometric information. Silent opt-in did not qualify.

  Some European governments had already ordered Facebook to delete facial recognition data before DeepFace's wider deployment. A Huffington Post piece described the technology as "creepy" and pointed to those government actions as evidence of broader concern. Broadcasting and Cable reported that Facebook and Google were both invited to a 2014 National Telecommunications and Information Administration meeting convened by the Center for Digital Democracy to help draft a consumer privacy Bill of Rights. Both companies declined to attend.

  Facebook's director of artificial intelligence research stated that DeepFace was not designed to invade individual privacy. The stated purpose was protective: when someone's face appeared in a newly uploaded photo, DeepFace would alert that person. They could then choose to remove their face from the image. Facebook committed not to share facial templates with third parties and said it would delete templates if a user deleted their account or removed a tag.

• 5. The BIPA Lawsuit

  ›

  Illinois passed the most comprehensive biometric privacy statute in the United States. The Biometric Information Privacy Act, known as BIPA, requires any company collecting biometric data to give written notice, obtain a signed release, and disclose how long the data will be stored. Facebook users filed a class action lawsuit arguing that the tag-suggestion tool violated every one of those requirements.

  The Ninth Circuit denied Facebook's motion to dismiss and certified the case as a class action. Facebook sought to appeal that certification and ultimately succeeded in getting a hearing. The company argued that plaintiffs had not alleged any concrete harm beyond the statutory violation itself. Courts found that argument insufficient. Facebook first proposed a $550 million settlement. The court rejected it. After the company raised the offer to $650 million, the settlement was accepted. In early March 2021, Facebook was ordered to pay. Each of the 1.6 million Illinois residents in the class would receive at least $345.

  In 2019, before the settlement was finalized, Facebook removed the automatic facial recognition tagging feature entirely, shifting to an opt-in model. Meta later announced a larger shutdown: the deletion of more than one billion facial recognition templates by December 2021. The underlying DeepFace software was not scheduled for deletion, and Meta did not rule out using facial recognition in future products.

• 6. Algorithmic Bias

  ›

  Facial recognition algorithms achieve over 90% accuracy on some populations while failing systematically on others. Systems including DeepFace identify Black and Asian faces incorrectly at a rate 10 to 100 times higher than they do for white faces. Women and young people also experience lower accuracy. The root cause is training data composition: algorithms built primarily from images of white men perform poorly when applied to faces that differ from that dominant sample.

  In July 2020, Facebook announced that it was assembling dedicated teams to examine racism in its algorithms. Those teams would work within Facebook's Responsible AI division. What specific reforms would follow remained unclear at the time of that announcement.

  The 2019 "Ten Year Challenge" illustrated how these concerns extend into everyday platform behavior. More than 5 million people, including many celebrities, posted a photo from a decade earlier alongside a recent one. Writer Kate O'Neill published an op-ed in Wired raising the possibility that Facebook had designed the challenge to produce paired images useful for training age-progression models. Facebook denied any role in originating the challenge. Whether the challenge was engineered or organic, it captured a real anxiety: every photo uploaded to Facebook potentially improves the same database that more than a billion users had their face scanned into, and that database remains the largest facial recognition dataset in existence.