Content deleted Content added
m →References: clean up; http->https after permanent move of Wikimedia to HTTPS using AWB |
m →History |
||
| (40 intermediate revisions by 24 users not shown) | |||
Line 1:
{{short description|Experiment using randomness in some aspect, usually to aid in removal of bias}}
[[Image:Flowchart of Phases of Parallel Randomized Trial - Modified from CONSORT 2010.png|thumb|250px|right|Flowchart of four phases (enrollment, intervention allocation, follow-up, and data analysis) of a parallel randomized trial of two groups, modified from the [[Consolidated Standards of Reporting Trials|CONSORT 2010 Statement]]<ref name="Schulz-2010">{{Cite journal | author = Schulz KF, Altman DG, Moher D; for the CONSORT Group | title = CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials | journal = BMJ | volume = 340 | pages = c332 | year = 2010 | doi = 10.1136/bmj.c332 | url= | pmid = 20332509 | pmc = 2844940 }}</ref>]] In [[scientific method|science]], '''randomized experiments''' are the [[experiment]]s that allow the greatest reliability and validity of statistical estimates of treatment effects. Randomization-based inference is especially important in [[experimental design]] and in [[survey sampling]].
== Overview ==
Line 9:
Randomization of treatment in [[clinical trials]] pose ethical problems. In some cases, randomization reduces the therapeutic options for both physician and patient, and so randomization requires [[clinical equipoise]] regarding the treatments.
==Online
Web sites can run randomized controlled experiments<ref>{{cite book
| last = Kohavi
| first = Ron
Line 25 ⟶ 23:
| publisher = Springer
| year = 2015
| chapter-url = http://www.exp-platform.com/Documents/2015%20Online%20Controlled%20Experiments_EncyclopediaOfMLDM.pdf
}}</ref> to create a feedback loop.<ref name="surveyarticle">{{cite journal
|author1=Kohavi, Ron |author2=Longbotham, Roger |author3=Sommerfield, Dan |author4=Henne, Randal M.
| title = Controlled experiments on the web: survey and practical guide
| journal = Data Mining and Knowledge Discovery
Line 35 ⟶ 31:
| issue = 1
| pages = 140–181
| year = 2009
| issn = 1384-5810
| doi = 10.1007/s10618-008-0114-1
| doi-access = free
}}</ref> Key differences between offline experimentation and online experiments include:<ref name="surveyarticle"/><ref name="puzzlingResults">{{cite conference
| url= http://www.exp-platform.com/Pages/PuzzingOutcomesExplained.aspx
| title=Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained
Line 47 ⟶ 41:
| first = Ron
|author2=Deng, Alex |author3=Frasca, Brian |author4=Longbotham, Roger |author5=Walker, Toby |author6= Xu Ya
|
| year = 2012}}</ref>
* Logging: user interactions can be logged reliably.
* Number of users: large sites, such as Amazon, Bing/Microsoft, and Google run experiments, each with over a million users.
* Number of concurrent experiments: large sites run tens of overlapping, or concurrent, experiments.<ref name="ExPScale">{{cite
| last = Kohavi
| first = Ron
|author2=Deng Alex |author3=Frasca Brian |author4=Walker Toby |author5=Xu Ya |author6= Nils Pohlmann
| title = Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
| chapter = Online controlled experiments at large scale
| date = 2013
| volume = 19
| pages = 1168–1176
| publisher = ACM
| ___location = Chicago, Illinois, USA
| doi = 10.1145/2487575.2488217
| isbn = 9781450321747
| s2cid = 13224883
}}</ref>
* Robots, whether [[web crawlers]] from valid sources or malicious [[internet bots]].{{clarify|reason=what about them? do they affect the reliability of the results?|date=May 2019}}
* Ability to ramp-up experiments from low percentages to higher percentages.
* Speed / performance has significant impact on key metrics.<ref name="surveyarticle" /><ref name="ExPRulesOfThumb">
{{cite
| last = Kohavi
| first = Ron
| author2=Deng Alex |author3=Longbotham Roger |author4=Xu Ya
| title = Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining
|
| date = 2014
| chapter-url = http://www.exp-platform.com/Pages/SevenRulesofThumbforWebSiteExperimenters.aspx
| volume = 20
| pages = 1857–1866
| publisher = ACM
| ___location = New York, New York, USA
| doi = 10.1145/2623330.2623341
| isbn = 9781450329569
| s2cid = 207214362
}}</ref>
* Ability to use the pre-experiment period as an A/A test to reduce variance.<ref name="cuped">{{cite conference
Line 88 ⟶ 85:
| title=Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data
| last = Deng
| first = Alex |author2=Xu, Ya |author3=Kohavi, Ron |author4=Walker, Toby
| book-title = WSDM 2013: Sixth ACM International Conference on Web Search and Data Mining
| year = 2013}}</ref>
Line 97 ⟶ 93:
{{main|History of experiments}}
| last = Neuhauser
|author2=Diaz, M
| title = Daniel: using the Bible to teach quality improvement methods
| journal = Quality and Safety in Health Care
| volume = 13
| doi = 10.1136/qshc.2003.009480
| pmid=15069225
| pmc=1743807
}}</ref><ref>{{cite book
| last = Angrist
Line 120 ⟶ 116:
</ref>
Randomized experiments were institutionalized in psychology and education in the late eighteen-hundreds, following the invention of randomized experiments by [[Charles Sanders Peirce|C. S. Peirce]].<ref>{{cite journal| author=[[Charles Sanders Peirce]] and [[Joseph Jastrow]]| year=1885|title=On Small Differences in Sensation| journal=Memoirs of the National Academy of Sciences|volume=3|pages=73–83|url=http://psychclassics.yorku.ca/Peirce/small-diffs.htm}} http://psychclassics.yorku.ca/Peirce/small-diffs.htm</ref><ref>{{cite journal| doi=10.1086/354775| first=Ian |last=Hacking|
Outside of psychology and education, randomized experiments were popularized by [[R.A. Fisher]] in his book ''[[Statistical Methods for Research Workers]]'', which also introduced additional principles of experimental design.
==Statistical
{{Expand section|date=September 2012}}
The [[Rubin Causal Model]] provides a common way to describe a randomized experiment. While the Rubin Causal Model provides a framework for defining the causal parameters (i.e., the effects of a randomized treatment on an outcome), the analysis of experiments can take a number of forms. The model assumes that there are two potential outcomes for each unit in the study: the outcome if the unit receives the treatment and the outcome if the unit does not receive the treatment. The difference between these two potential outcomes is known as the treatment effect, which is the causal effect of the treatment on the outcome. Most commonly, randomized experiments are analyzed using [[ANOVA]], [[
The Rubin Causal Model is a useful a framework for understanding how to estimate the causal effect of the treatment, even when there are confounding variables that may affect the outcome. This model specifies that the causal effect of the treatment is the difference in the outcomes that would have been observed for each individual if they had received the treatment and if they had not received the treatment. In practice, it is not possible to observe both potential outcomes for the same individual, so statistical methods are used to estimate the causal effect using data from the experiment.
==Empirical evidence that randomization makes a difference==
Empirically differences between randomized and non-randomized studies,<ref>{{cite journal| doi=10.1002/14651858.MR000034.pub2|
== Directed acyclic graph (DAG) explanation of randomization ==
Randomization is the cornerstone of many scientific claims. To randomize, means that we can eliminate the confounding factors. Say we study the effect of '''A''' on '''B.''' Yet, there are many unobservables '''U''' that potentially affect '''B''' and confound our estimate of the finding. To explain these kinds of issues, statisticians or econometricians nowadays use [[directed acyclic graph]].{{Needs update|date=July 2024}}
==See also==
*[[A/B testing]]
*[[Allocation concealment]]
*[[Random assignment]]
*[[Randomized block design]]
*[[Randomized controlled trial]]
==References==
{{Reflist}}
* {{cite book |author1 = Caliński, Tadeusz
|
|name-list-style = amp
|title = Block designs: A Randomization approach, Volume '''I''': Analysis
|series = Lecture Notes in Statistics
|volume = 150
|publisher = Springer-Verlag
|___location = New York
|year = 2000
|isbn = 978-0-387-98578-7
|url-access = registration
|url = https://archive.org/details/blockdesignsrand0002cali
}}
* {{cite book |author1 = Caliński, Tadeusz
|
|name-list-style = amp
|title = Block designs: A Randomization approach, Volume '''II''': Design
|series = Lecture Notes in Statistics
|volume = 170
|publisher = Springer-Verlag
|___location = New York
|year = 2003
|isbn = 978-0-387-95470-7
|url-access = registration
|url = https://archive.org/details/blockdesignsrand0002cali
}}
* {{cite journal|doi=10.1086/354775|first=Ian |last=Hacking|
*{{cite book|
* {{cite book| last=Kempthorne|first=Oscar |chapter=Intervention experiments, randomization and inference|title=Current Issues in Statistical Inference—Essays in Honor of D. Basu | editor=Malay Ghosh and Pramod K. Pathak | pages=13–31 | publisher=Institute for Mathematical Statistics |___location=Hayward, CA | chapter-url=http://projecteuclid.org/euclid.lnms/1215458836 | doi=10.1214/lnms/1215458836 | mr=1194407|
{{Experimental design}}
| |||