Fairness (machine learning)

• 1. Introduction

  ›

  Fairness in machine learning is a field driven by specific, documented harms. In 2016, ProPublica published an analysis of COMPAS software, used by US courts to predict whether defendants would reoffend. The finding was pointed: black defendants were almost twice as likely as white defendants to be incorrectly labeled as high risk. White defendants faced the opposite error more often. The maker of COMPAS, Northepointe Inc., disputed the findings and accused ProPublica of statistical errors. ProPublica refuted that rebuttal. U.S. Attorney General Eric Holder had flagged a related concern two years earlier. He warned that risk assessment tools might unfairly weigh factors no defendant can control. Education level and socioeconomic background were among those factors. The exchange exposed a deeper problem. There is no universal definition of what it means for an automated system to be fair. Different formal definitions can contradict each other mathematically. Bias appears across domains as distinct as courtrooms, hiring platforms, image recognition systems, and language models. How to detect it, measure it, and correct it remains one of the most contested questions in computer science.

• 2. An Earlier Reckoning

  ›

  A vivid scientific discussion of quantitative fairness in decision-making flourished during the mid-1960s and 1970s. Its primary catalyst was the passage of the U.S. Civil Rights Act of 1964 and the civil rights movement that preceded it. Researchers debated how to measure discrimination and when one definition of fairness should take priority over another. By the end of the 1970s, that debate had largely faded. Competing notions of fairness offered no clear basis for choosing among them. The conversation collapsed under its own ambiguity and largely disappeared. Decades later, algorithmic decision-making brought the same unresolved questions back into view. Since 2016, research into algorithmic fairness has grown sharply. Computer scientists, statisticians, legal scholars, and ethicists have converged on the topic from different disciplines. They often bring conflicting frameworks and no shared starting definition. The range of bias types under scrutiny reflects how far the problem has spread since those first debates. Racial disparities in criminal risk scores sit alongside language models that treat non-English perspectives as noise.

• 3. Failures on the Record

  ›

  In 2015, Google apologized after Google Photos incorrectly labeled a black couple as gorillas. Around the same time, Flickr's auto-tag feature labeled some black people as apes and animals. A 2016 international beauty contest judged by an AI algorithm favored contestants with lighter skin; researchers traced the likely cause to bias in the training data. A 2018 study examined three commercial gender classification systems and found a consistent result: all three performed best with light-skinned males and worst with dark-skinned females. In 2020, Twitter's image cropping tool was shown to prefer lighter-skinned faces. In 2022, the creators of DALL-E 2 acknowledged that the model's generated images were significantly stereotyped by gender and race.

  Amazon used software to screen job applications that penalized resumes containing the word women. In 2019, Apple's algorithm for setting credit limits on the new Apple Card gave significantly higher limits to male applicants than to female ones, even among couples who shared their finances. A 2021 report by The Markup found that US mortgage-approval algorithms were more likely to reject non-white applicants.

  Research by Luo and colleagues found that large language models trained primarily on English data treat Anglo-American viewpoints as authoritative. Non-English perspectives are systematically treated as irrelevant or wrong. When asked to describe liberalism, ChatGPT drew on the Anglo-American tradition, emphasizing human rights and equality. Perspectives embedded in Vietnamese, Chinese, Japanese, Korean, French, and German corpora were absent from the response. Political viewpoints face the same pressure. A model trained on data where certain political positions appear more frequently may generate responses that favor those positions. No mechanism currently signals this imbalance to the user.

• 4. Competing Definitions

  ›

  Three mathematical criteria dominate the formal research on fairness in machine learning. Independence requires that a model's predictions be statistically independent of sensitive characteristics such as race or gender. Under this criterion, the rate at which individuals are assigned to any predicted class must be equal across groups. Separation takes a more permissive position. It allows predictions to correlate with sensitive characteristics, but only insofar as the true outcome variable justifies that correlation. Any difference in treatment must be traceable to the target variable, not to the model's bias. Sufficiency is the third criterion. Two individuals who receive the same prediction should have equal probability of that prediction being correct, regardless of which group they come from.

  Total fairness, the simultaneous satisfaction of all three criteria, is mathematically achievable only in narrow theoretical cases. When sensitive attributes and true outcomes are not statistically independent, which describes most real-world data, independence and separation cannot both hold. The same incompatibility applies to separation and sufficiency. Each criterion is coherent in isolation; together they are, in most realistic settings, irreconcilable.

  Fairness through unawareness, the most intuitive individual-level approach, proposes removing sensitive attributes from a model's inputs entirely. Its main weakness is that other features in the data often correlate with the excluded attribute. A designer with discriminatory intent can introduce a proxy variable and effectively recover the sensitive information anyway. Cynthia Dwork and collaborators addressed this in a 2012 paper, calling their framework fairness through awareness. It requires a model to treat similar individuals similarly. The challenge is building a distance metric that captures what similarity means in a given context. A related concept, counterfactual fairness, asks a different question: would a decision remain the same in a hypothetical world where the individual's sensitive attributes were different? Researchers including Kusner and colleagues have proposed formal conditions for this, and the idea has been extended to equalized odds in counterfactual scenarios as well.

• 5. Strategies for Reduction

  ›

  Preprocessing adjusts the training data before a model ever sees it. Simply removing the sensitive variable does not solve the problem, because other features often correlate with it. One technique maps each individual to an intermediate representation where group membership cannot be recovered, while preserving as much other information as possible. Zemel and colleagues demonstrated this approach using a multinomial random variable as the intermediate representation. Reweighing is a simpler preprocessing method. It assigns each data point a numerical weight: lower for members of favored groups, higher for members of unfavored groups. The weighted discrimination across the dataset then falls to zero.

  Inprocessing adds fairness constraints directly to the model's training objective. A constraint might require the false positive rate to be equal across groups. Adding such constraints can make the optimization problem harder to solve. Researchers sometimes relax the constraints to keep training feasible. Adversarial debiasing is one specific inprocessing technique. It trains a predictor and an adversary simultaneously using gradient descent. The predictor tries to forecast the target variable. The adversary tries to predict the sensitive attribute from the predictor's output. The predictor is updated to minimize its own error while maximizing the adversary's error. That way, the adversary cannot extract sensitive information from what the predictor outputs. Training this way has been shown to improve demographic parity.

  Postprocessing works on a trained model's outputs rather than on the model itself. A threshold is set for each group independently so that error rates align across them. One tool is the ROC curve, which plots the true positive rate against the false positive rate at various threshold settings. The selected threshold is the point where rates converge across groups. Reject option classification handles uncertain cases differently. When a model's confidence score falls near the classification boundary, the outcome is uncertain. In such cases, individuals from disadvantaged groups are labeled positive rather than following the model's biased default. Sendhil Mullainathan has argued that these trade-offs should be made explicit through social welfare functions. Such functions express both fairness and accuracy in terms of their concrete effects on the people the algorithm affects.

• 6. Industry Efforts and Limits

  ›

  IBM built tools for both Python and R containing multiple algorithms to detect and reduce software bias. Google published guidelines and tools for studying and combating bias in machine learning. Facebook reported using an internal tool called Fairness Flow to detect bias in its AI. Critics found the effort limited in practice. Few employees used the tool. It could not be applied to all of Facebook's programs, and even where it could, use remained optional.

  Mathematical approaches to fairness require sorting individuals into predefined social groups. That categorization can be too coarse to capture how discrimination actually operates. Sensitive characteristics interact with each other in ways that simple group-level models do not reflect. There is also no settled philosophical or legal definition of non-discrimination that technical research can use as a shared foundation. Different researchers are therefore building different tools with different implicit assumptions about what fairness means.

  Human operators who consult algorithmic systems may accept recommendations only when they align with their existing biases. When that happens, the algorithm's formal fairness does not produce fair outcomes in practice. A model can satisfy a mathematical criterion and still fail the people it was designed to protect. One study offered a concrete counterpoint: using an algorithmic tool for pretrial detention rather than unaided human judgment reduced detention rates for Black, Hispanic, and other racial minority defendants while the overall crime rate remained constant. That result supports the case for designing with explicit social welfare functions. Such functions make the trade-off between efficiency and equity legible, rather than leaving it buried in the model's structure.