GEPI《How AI Fails Women》報告檢視算法如何放大性別偏見

A hiring score, a credit assessment, or a clinical recommendation can look impartial while carrying forward the inequalities in the data behind it. That is the central warning of the Gender Equity Policy Institute’s new report, How AI Fails Women: When Bias Becomes Amplified, which brings together research on how automated systems can reproduce—and sometimes intensify—gender discrimination.

Written by machine learning scientist and GEPI senior fellow Myriam Shiran, PhD, the report reviews academic and gray literature across economic opportunity, healthcare, criminal justice, generated content, consumer AI design, and AI-enabled image-based sexual abuse. It is a synthesis of existing evidence, rather than a new experiment measuring every AI system against a single benchmark.

Its most consequential argument is about what happens after a biased result leaves a model. Recommendations influence decisions, generated content shapes expectations, and some outputs become material for future training. An unfair pattern can therefore become harder to identify as it moves through successive systems.

Gender-Blind Inputs Do Not Guarantee Fair Outcomes

Removing a gender field is an intuitive response to discrimination. GEPI’s review explains why that step alone is insufficient: names, education, employment histories, and other details can still carry gender-related signals. Historical outcomes may also encode unequal treatment before a model ever encounters them.

One hiring audit cited in the report, by Wilson and Caliskan in 2024, involved more than three million simulated résumé comparisons across nine occupations. The tested screening models significantly favored male-associated names in 51.9% of comparisons and female-associated names in 11.1%. The remaining 37% did not show a significant preference. These are results from controlled model tests, not percentages of real applicants denied jobs.

The evidence also varies by test design. The report describes a separate 2025 audit in which several language models showed no gender difference when researchers changed names alone, while changes to education details affected scores. A narrow test can miss information that matters elsewhere in the same workflow.

In lending, the review points to research on roughly 80,000 unsecured personal loans from a major U.S. fintech lender. Women’s loans generated higher returns for the lender, a finding the study linked to a model overestimating their default risk and pricing loans accordingly. Here, the issue is the relationship between predicted risk, actual repayment, and the price a borrower pays—not simply whether gender appears in the application.

Medical AI Needs More Than a Strong Average Score

Healthcare offers a particularly clear example of why population-wide performance can conceal important differences. A model may perform well overall while making more errors for people underrepresented in its training or validation data.

GEPI cites a 2024 study by Zack and colleagues that presented GPT-4 with otherwise identical chest-pain cases described as involving a man or a woman. Across 200 runs for each description, the model rated cardiac stress testing as highly important in 71% of the male cases and 58% of the female cases—a 13-percentage-point difference.

That finding concerns model recommendations in a controlled evaluation, not observed treatment decisions or patient outcomes. The distinction matters, particularly because the report includes exploratory studies of different models, diseases, and clinical tasks conducted at different times.

The review also makes an essential qualification: clinically meaningful sex differences should not simply be erased. The challenge is to determine whether a difference in a model’s advice reflects relevant biology or an unsupported assumption. That requires representative evaluation data and careful examination of errors, rather than a demand that every recommendation be identical.

Bias Can Shape Representation as Well as Decisions

The report distinguishes harm to an individual’s access to resources—such as a loan or medical care—from harm in how groups are portrayed. Generated recommendation letters, news articles, and occupational images can reinforce stereotypes even when no immediate allocation decision follows.

Research summarized by GEPI found language models describing women more through interpersonal qualities and men more through leadership or ability. Studies of image generators found exaggerated gender segregation across occupations. Those results apply to the models and versions examined; they should not be read as a fresh audit of every currently available system.

Consumer assistants and companions raise a different question. Their voices, personalities, and responses to harassment are deliberate product choices. Evaluating only the statistical balance of their answers can overlook the assumptions embedded in what a product is designed to be.

The review also examines nonconsensual synthetic sexual imagery, where AI can make abuse easier to produce and distribute. This is more than an inaccurate depiction: the use of a real person’s likeness without consent can create direct harm. It calls for a different response from correcting an occupational stereotype in generated artwork.

Why Independent Testing Matters

GEPI describes four recurring mechanisms: learning from unequal historical records, reproducing stereotypes, presenting judgments as objective measurements, and allowing biased outputs to feed subsequent decisions or models.

Independent audits help make those mechanisms visible. Researchers can vary one attribute in otherwise comparable inputs and examine how outputs change, even without access to a proprietary model’s internal architecture. But a successful test on one prompt or benchmark does not establish fairness across an entire deployment.

For organizations adopting AI, the practical implication is to test the actual use case. A recruitment team needs to examine the route from screening scores to interview selection. A lender needs to compare risk estimates with repayment outcomes. A healthcare organization needs to understand which patients are represented in validation data and where errors concentrate. These checks answer different questions from whether a model is fluent or accurate on average.

What the Report Establishes—and What Remains Unknown

The report draws primarily on U.S. evidence while identifying international studies where relevant. Its sources span different technologies, years, and methods. Some document outcomes; others reveal concerning behavior in simulations. The review does not establish one universal rate of discrimination, nor does it show that every system produces the same bias.

It also identifies a measurement problem: researchers often cannot access the institutional data needed to connect an output with its consequences. Seeing a disparity is important, but understanding its real-world cost may require records held by employers, vendors, lenders, or healthcare providers.

GEPI’s contribution is to connect findings that are often discussed separately and ask whether testing and accountability follow AI into the decisions it influences. The full report, How AI Fails Women: When Bias Becomes Amplified, includes the methodology, study references, and sector-by-sector evidence behind that argument.