Content deleted Content added
Mpoynter18 (talk | contribs) adding a study at the end of introduction section, plus a reference. |
m →Racial and ethnic discrimination: Change "www.timesofisrael.com" to "The Times of Israel" |
||
| (15 intermediate revisions by 12 users not shown) | |||
Line 12:
As algorithms expand their ability to organize society, politics, institutions, and behavior, sociologists have become concerned with the ways in which unanticipated output and manipulation of data can impact the physical world. Because algorithms are often considered to be neutral and unbiased, they can inaccurately project greater authority than human expertise (in part due to the psychological phenomenon of [[automation bias]]), and in some cases, reliance on algorithms can displace human responsibility for their outcomes. Bias can enter into algorithmic systems as a result of pre-existing cultural, social, or institutional expectations; by how features and labels are chosen; because of technical limitations of their design; or by being used in unanticipated contexts or by audiences who are not considered in the software's initial design.<ref>{{Cite book |last1=Suresh |first1=Harini |last2=Guttag |first2=John |title=Equity and Access in Algorithms, Mechanisms, and Optimization |chapter=A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle |date=2021-11-04 |chapter-url=https://dl.acm.org/doi/10.1145/3465416.3483305 |series=EAAMO '21 |___location=New York, NY, USA |publisher=Association for Computing Machinery |pages=1–9 |doi=10.1145/3465416.3483305 |isbn=978-1-4503-8553-4|s2cid=235436386 }}</ref>
Algorithmic bias has been cited in cases ranging from election outcomes to the spread of [[online hate speech]]. It has also arisen in criminal justice,<ref>{{Cite journal |last=Krištofík |first=Andrej |date=2025-04-28 |title=Bias in AI (Supported) Decision Making: Old Problems, New Technologies |journal=International Journal for Court Administration |language=en |volume=16 |issue=1 |doi=10.36745/ijca.598 |issn=2156-7964|doi-access=free }}</ref> healthcare, and hiring, compounding existing racial, socioeconomic, and gender biases. The relative inability of facial recognition technology to accurately identify darker-skinned faces has been linked to multiple wrongful arrests of black men, an issue stemming from imbalanced datasets. Problems in understanding, researching, and discovering algorithmic bias persist due to the proprietary nature of algorithms, which are typically treated as trade secrets. Even when full transparency is provided, the complexity of certain algorithms poses a barrier to understanding their functioning. Furthermore, algorithms may change, or respond to input or output in ways that cannot be anticipated or easily reproduced for analysis. In many cases, even within a single website or application, there is no single "algorithm" to examine, but a network of many interrelated programs and data inputs, even between users of the same service.
A 2021 survey identified multiple forms of algorithmic bias, including historical, representation, and measurement biases, each of which can contribute to unfair outcomes.<ref>{{Cite journal |
== Definitions ==
Line 56:
and Transparency (FAT) of algorithms has emerged as its own interdisciplinary research area with an annual conference called FAccT.<ref>{{Cite web|url=https://facctconference.org/2021/press-release.html|title=ACM FAccT 2021 Registration
|website=fatconference.org|access-date=2021-11-14}}</ref> Critics have suggested that FAT initiatives cannot serve effectively as independent watchdogs when many are funded by corporations building the systems being studied.<ref name="Ochigame">{{cite web |last1=Ochigame |first1=Rodrigo |title=The Invention of "Ethical AI": How Big Tech Manipulates Academia to Avoid Regulation |url=https://theintercept.com/2019/12/20/mit-ethical-ai-artificial-intelligence/ |website=The Intercept |access-date=11 February 2020 |date=20 December 2019}}</ref>
NIST’s AI Risk Management Framework 1.0 and its 2024 Generative AI Profile provide practical guidance for governing and measuring bias mitigation in AI systems.[https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf]
== Types ==
Line 74 ⟶ 76:
==== Selection bias ====
[[Selection bias]] refers the inherent tendency of large language models to favor certain option identifiers irrespective of the actual content of the options. This bias primarily stems from token bias—that is, the model assigns a higher a priori probability to specific answer tokens (such as “A”) when generating responses. As a result, when the ordering of options is altered (for example, by systematically moving the correct answer to different positions), the model’s performance can fluctuate significantly. This phenomenon undermines the reliability of large language models in multiple-choice settings.<ref>{{Citation |last1=Choi |first1=Hyeong Kyu |last2=Xu |first2=Weijie |last3=Xue |first3=Chi |last4=Eckman |first4=Stephanie |last5=Reddy |first5=Chandan K. |title=Mitigating Selection Bias with Node Pruning and Auxiliary Options |date=2024-09-27 |arxiv=2409.18857}}</ref><ref>{{Citation |last1=Zheng |first1=Chujie |last2=Zhou |first2=Hao |last3=Meng |first3=Fandong |last4=Zhou |first4=Jie |last5=Huang |first5=Minlie |title=Large Language Models Are Not Robust Multiple Choice Selectors |date=2023-09-07 |arxiv=2309.03882}}</ref>
==== Gender bias ====
[[Gender bias]] refers to the tendency of these models to produce outputs that are unfairly prejudiced towards one gender over another. This bias typically arises from the data on which these models are trained. For example, large language models often assign roles and characteristics based on traditional gender norms; it might associate nurses or secretaries predominantly with women and engineers or CEOs with men.<ref>{{Cite book |last1=Busker |first1=Tony |last2=Choenni |first2=Sunil |last3=Shoae Bargh |first3=Mortaza |chapter=Stereotypes in ChatGPT: An empirical study |date=2023-11-20 |title=Proceedings of the 16th International Conference on Theory and Practice of Electronic Governance |chapter-url=https://dl.acm.org/doi/10.1145/3614321.3614325 |series=ICEGOV '23 |___location=New York, NY, USA |publisher=Association for Computing Machinery |pages=24–32 |doi=10.1145/3614321.3614325 |isbn=979-8-4007-0742-1}}</ref><ref>{{Cite book |last1=Kotek |first1=Hadas |last2=Dockum |first2=Rikker |last3=Sun |first3=David |chapter=Gender bias and stereotypes in Large Language Models |date=2023-11-05 |title=Proceedings of the ACM Collective Intelligence Conference |chapter-url=https://dl.acm.org/doi/10.1145/3582269.3615599 |series=CI '23 |___location=New York, NY, USA |publisher=Association for Computing Machinery |pages=12–24 |doi=10.1145/3582269.3615599 |arxiv=2308.14921 |isbn=979-8-4007-0113-9}}</ref>
==== Stereotyping ====
Beyond gender and race, these models can reinforce a wide range of
A recent focus in research has been on the complex interplay between the grammatical properties of a language and real-world biases that can become embedded in AI systems, potentially perpetuating harmful stereotypes and assumptions. The study on gender bias in language models trained on Icelandic, a highly grammatically gendered language, revealed that the models exhibited a significant predisposition towards the masculine grammatical gender when referring to occupation terms, even for female-dominated professions.<ref>{{Citation |last1=Friðriksdóttir|first1=Steinunn Rut|title=Gendered Grammar or Ingrained Bias? Exploring Gender Bias in Icelandic Language Models|journal=Lrec-Coling 2024|date=2024|pages=7596–7610|url=https://aclanthology.org/2024.lrec-main.671/|last2=Einarsson|first2=Hafsteinn}}</ref> This suggests the models amplified societal gender biases present in the training data.
==== Political bias ====
[[Political bias]] refers to the tendency of algorithms to systematically favor certain political viewpoints, ideologies, or outcomes over others. Language models may also exhibit political biases. Since the training data includes a wide range of political opinions and coverage, the models might generate responses that lean towards particular political ideologies or viewpoints, depending on the prevalence of those views in the data.<ref>{{Cite journal |last1=Feng |first1=Shangbin |last2=Park |first2=Chan Young |last3=Liu |first3=Yuhan |last4=Tsvetkov |first4=Yulia |date=July 2023 |editor-last=Rogers |editor-first=Anna |editor2-last=Boyd-Graber |editor2-first=Jordan |editor3-last=Okazaki |editor3-first=Naoaki |title=From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models |url=https://aclanthology.org/2023.acl-long.656 |journal=Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |___location=Toronto, Canada |publisher=Association for Computational Linguistics |pages=11737–11762 |doi=10.18653/v1/2023.acl-long.656|doi-access=free |arxiv=2305.08283 }}</ref><ref>{{Cite web |last=Dolan |first=Eric W. |date=2025-02-14 |title=Scientists reveal ChatGPT's left-wing bias — and how to "jailbreak" it |url=https://www.psypost.org/scientists-reveal-chatgpts-left-wing-bias-and-how-to-jailbreak-it/ |access-date=2025-02-14 |website=PsyPost - Psychology News |language=en-US}}</ref>
[[Racism|Racial bias]] refers to the tendency of machine learning models to produce outcomes that unfairly discriminate against or stereotype individuals based on race or ethnicity. This bias often stems from training data that reflects historical and systemic inequalities. For example, AI systems used in hiring, law enforcement, or healthcare may disproportionately disadvantage certain racial groups by reinforcing existing stereotypes or underrepresenting them in key areas. Such biases can manifest in ways like facial recognition systems misidentifying individuals of certain racial backgrounds or healthcare algorithms underestimating the medical needs of minority patients. Addressing racial bias requires careful examination of data, improved transparency in algorithmic processes, and efforts to ensure fairness throughout the AI development lifecycle.<ref>{{Cite web |last=Lazaro |first=Gina |date=May 17, 2024 |title=Understanding Gender and Racial Bias in AI |url=https://www.sir.advancedleadership.harvard.edu/articles/understanding-gender-and-racial-bias-in-ai |access-date=December 11, 2024 |website=Harvard Advanced Leadership Initiative Social Impact Review}}</ref><ref>{{Cite journal |last=Jindal |first=Atin |date=September 5, 2022 |title=Misguided Artificial Intelligence: How Racial Bias is Built Into Clinical Models |journal=Journal of Brown Hospital Medicine |volume=2 |issue=1 |page=38021 |doi=10.56305/001c.38021 |doi-access=free |pmid=40046549 |pmc=11878858 }}</ref>
=== Technical ===
Line 129 ⟶ 131:
=== Gender discrimination ===
In 2016, the professional networking site [[LinkedIn]] was discovered to recommend male variations of women's names in response to search queries. The site did not make similar recommendations in searches for
In 2012, the department store franchise [[Target (company)|Target]] was cited for gathering data points to infer when
Web search algorithms have also been accused of bias. Google's results may prioritize pornographic content in search terms related to sexuality, for example, "lesbian". This bias extends to the search engine showing popular but sexualized content in neutral searches. For example, "Top 25 Sexiest Women Athletes" articles displayed as first-page results in searches for "women athletes".<ref name="Noble">{{cite journal|last1=Noble|first1=Safiya|author-link=Safiya Noble|title=Missed Connections: What Search Engines Say about Women|journal=[[Bitch (magazine)|Bitch]] |date=2012|volume=12|issue=4|pages=37–41|url=https://safiyaunoble.files.wordpress.com/2012/03/54_search_engines.pdf}}</ref>{{rp|31}} In 2017, Google adjusted these results along with others that surfaced [[hate groups]], racist views, child abuse and pornography, and other upsetting and offensive content.<ref name="Guynn2">{{cite news|last1=Guynn|first1=Jessica|title=Google starts flagging offensive content in search results|url=https://www.usatoday.com/story/tech/news/2017/03/16/google-flags-offensive-content-search-results/99235548/|access-date=19 November 2017|work=USA TODAY|agency=USA Today|date=16 March 2017}}</ref> Other examples include the display of higher-paying jobs to male applicants on job search websites.<ref name="SimoniteMIT">{{cite web|url=https://www.technologyreview.com/s/539021/probing-the-dark-side-of-googles-ad-targeting-system/|title=Study Suggests Google's Ad-Targeting System May Discriminate|last1=Simonite|first1=Tom|website=MIT Technology Review|publisher=Massachusetts Institute of Technology|access-date=17 November 2017}}</ref> Researchers have also identified that machine translation exhibits a strong tendency towards male defaults.<ref>{{Cite arXiv|eprint = 1809.02208|last1 = Prates|first1 = Marcelo O. R.|last2 = Avelar|first2 = Pedro H. C.|last3 = Lamb|first3 = Luis|title = Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate|year = 2018|class = cs.CY}}</ref> In particular, this is observed in fields linked to unbalanced gender distribution, including [[Science, technology, engineering, and mathematics|STEM]] occupations.<ref>{{Cite journal |doi = 10.1007/s00521-019-04144-6|title = Assessing gender bias in machine translation: A case study with Google Translate|journal = Neural Computing and Applications|year = 2019|last1 = Prates|first1 = Marcelo O. R.|last2 = Avelar|first2 = Pedro H.|last3 = Lamb|first3 = Luís C.|volume = 32|issue = 10|pages = 6363–6381|arxiv = 1809.02208|s2cid = 52179151}}</ref> In fact, current machine translation systems fail to reproduce the real world distribution of female workers.<ref>{{cite news |last1=Claburn |first1=Thomas |title=Boffins bash Google Translate for sexism |url=https://www.theregister.com/2018/09/10/boffins_bash_google_translate_for_sexist_language/ |access-date=28 April 2022 |work=The Register |date=10 September 2018 |language=en}}</ref>
In 2015, [[Amazon.com]] turned off an AI system it developed to screen job applications when they realized it was biased against women.<ref>{{cite news |last1=Dastin |first1=Jeffrey |title=Amazon scraps secret AI recruiting tool that showed bias against women |url=https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G |work=Reuters |date=October 9, 2018}}</ref> The recruitment tool excluded applicants who attended all-women's colleges and resumes that included the word "women's".<ref>{{Cite web|last=Vincent|first=James|date=10 October 2018|title=Amazon reportedly scraps internal AI recruiting tool that was biased against women|url=https://www.theverge.com/2018/10/10/17958784/ai-recruiting-tool-bias-amazon-report|website=The Verge}}</ref> A similar problem emerged with music streaming services—In 2019, it was discovered that the recommender system algorithm used by Spotify was biased against
=== Racial and ethnic discrimination ===
Line 150 ⟶ 152:
Another study, published in August 2024, on [[Large language model]] investigates how language models perpetuate covert racism, particularly through dialect prejudice against speakers of African American English (AAE). It highlights that these models exhibit more negative stereotypes about AAE speakers than any recorded human biases, while their overt stereotypes are more positive. This discrepancy raises concerns about the potential harmful consequences of such biases in decision-making processes.<ref>Hofmann, V., Kalluri, P.R., Jurafsky, D. et al. AI generates covertly racist decisions about people based on their dialect. Nature 633, 147–154 (2024). https://doi.org/10.1038/s41586-024-07856-5</ref>
A study published by the [[Anti-Defamation League]] in 2025 found that several major LLMs, including [[ChatGPT]], [[Llama (language model)|Llama]], [[Claude (language model)|Claude]], and [[Gemini (language model)|Gemini]] showed antisemitic bias.<ref>{{
A 2018 study found that commercial gender classification systems had significantly higher error rates for darker-skinned women, with error rates up to 34.7%, compared to near-perfect accuracy for lighter-skinned men.<ref>{{Cite conference |last1=Buolamwini |first1=J. |last2=Gebru |first2=T. |title=Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification |book-title=Proceedings of the 1st Conference on Fairness, Accountability and Transparency |pages=77–91 |year=2018 |url=https://proceedings.mlr.press/v81/buolamwini18a.html |access-date=April 30, 2025}}</ref>
==== Law enforcement and legal proceedings ====
Line 164 ⟶ 168:
In 2017 a [[Facebook]] algorithm designed to remove online hate speech was found to advantage white men over black children when assessing objectionable content, according to internal Facebook documents.<ref name="AngwinGrassegger">{{cite web|url=https://www.propublica.org/article/facebook-hate-speech-censorship-internal-documents-algorithms|title=Facebook's Secret Censorship Rules Protect White Men From Hate Speech But Not Black Children — ProPublica|last1=Angwin|first1=Julia|last2=Grassegger|first2=Hannes|date=28 June 2017|website=ProPublica|access-date=20 November 2017}}</ref> The algorithm, which is a combination of computer programs and human content reviewers, was created to protect broad categories rather than specific subsets of categories. For example, posts denouncing "Muslims" would be blocked, while posts denouncing "Radical Muslims" would be allowed. An unanticipated outcome of the algorithm is to allow hate speech against black children, because they denounce the "children" subset of blacks, rather than "all blacks", whereas "all white men" would trigger a block, because whites and males are not considered subsets.<ref name="AngwinGrassegger" /> Facebook was also found to allow ad purchasers to target "Jew haters" as a category of users, which the company said was an inadvertent outcome of algorithms used in assessing and categorizing data. The company's design also allowed ad buyers to block African-Americans from seeing housing ads.<ref name="AngwinVarnerTobin">{{cite news|url=https://www.propublica.org/article/facebook-enabled-advertisers-to-reach-jew-haters|title=Facebook Enabled Advertisers to Reach 'Jew Haters' — ProPublica|last1=Angwin|first1=Julia|date=14 September 2017|work=ProPublica|access-date=20 November 2017|last2=Varner|first2=Madeleine|last3=Tobin|first3=Ariana}}</ref>
While algorithms are used to track and block hate speech, some were found to be 1.5 times more likely to flag information posted by Black users and 2.2 times likely to flag information as hate speech if written in [[African American English]].<ref>{{Cite conference|url=https://homes.cs.washington.edu/~msap/pdfs/sap2019risk.pdf|title=The Risk of Racial Bias in Hate Speech Detection|last1=Sap|first1=Maarten|last2=Card|first2=Dallas|last3=Gabriel|first3=Saadia|last4=Choi|first4=Yejin|last5=Smith|first5=Noah A.|book-title=Proceedings of the 57th Annual Meeting of the Association for Computational Linguist|publisher=Association for Computational Linguistics|___location=Florence, Italy|date=28 July – 2 August 2019|pages=1668–1678|url-status=live|archive-url=https://web.archive.org/web/20190814194616/https://homes.cs.washington.edu/~msap/pdfs/sap2019risk.pdf |archive-date=2019-08-14 }}</ref>
==== Surveillance ====
| |||