Content deleted Content added
m Restored reference to Master Metrics (now both available) |
m WP:CHECKWIKI error fix for #61. Punctuation goes before References. Do general fixes if a problem exists. - using AWB (10901) |
||
Line 52:
* Logging: user interactions can be logged reliably.
* Number of users: large sites, such as Amazon, Bing/Microsoft, and Google run experiments, each with over a million users.
* Number of concurrent experiments: large sites run tens of overlapping, or concurrent, experiments.<ref name="ExPScale">{{cite conference
| last = Kohavi
| first = Ron
Line 65:
| url = http://dx.doi.org/10.1145/2487575.2488217
| doi = 10.1145/2487575.2488217
}}</ref>
* Robots, whether [[web crawlers]] from valid sources or malicious [[internet bots]].
* Ability to ramp-up experiments from low percentages to higher percentages.
* Speed / performance has significant impact on key metrics.<ref name="surveyarticle"
{{cite conference
| last = Kohavi
Line 77:
| journal = Proceedings of the 20th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
| volume = 20
| pages =
| publisher = ACM
| ___location = New York, New York, USA
| year = 2014
| doi = 10.1145/2623330.2623341
}}</ref>
* Ability to use the pre-experiment period as an A/A test to reduce variance.<ref name="cuped">{{cite conference
| url= http://www.exp-platform.com/Pages/CUPED.aspx
| title=Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data
Line 91:
| coauthors = Xu, Ya; Kohavi, Ron; Walker, Toby
| booktitle = WSDM 2013: Sixth ACM International Conference on Web Search and Data Mining
| year = 2013}}</ref>
==History==
Line 98:
The earliest controlled experiments appears to have been suggested in the Old Testament's Book of Daniel. King Nebuchadnezzar proposed that some Israelites eat "a daily amount of food and wine from the king's table." Daniel preferred a vegetarian diet, but the official was concerned that the king would "see you looking worse than the other young men your age? The king would then have my head because of you."
Daniel then proposed the following controlled experiment: "Test your servants for ten days. Give us nothing but vegetables to eat and water to drink. Then compare our appearance with that of the young men who eat the royal food, and treat your servants in accordance with what you see” (Daniel 1, 12– 13).<ref>{{cite journal
| last = Neuhauser
| first = D
Line 106:
| volume = 13
| issue = 2
| pages =
| year = 2004
| url = http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1743807/pdf/v013p00153.pdf
Line 118:
| year = 2014
| page = 31}}
</ref>
Randomized experiments were institutionalized in psychology and education in the late eighteen-hundreds, following the invention of randomized experiments by [[Charles Sanders Peirce|C. S. Peirce]].<ref>{{cite journal| author=[[Charles Sanders Peirce]] and [[Joseph Jastrow]]|year=1885|title=On Small Differences in Sensation| journal=Memoirs of the National Academy of Sciences|volume=3|pages=73–83|url=http://psychclassics.yorku.ca/Peirce/small-diffs.htm}} http://psychclassics.yorku.ca/Peirce/small-diffs.htm</ref><ref>{{cite journal|doi=10.1086/354775|first=Ian |last=Hacking| authorlink=Ian Hacking | title=Telepathy: Origins of Randomization in Experimental Design|journal=[[Isis (journal)|Isis]]|issue=3|volume=79|date=September 1988 |pages=427–451| mr = 1013489| jstor=234674}}</ref><ref>{{cite journal|doi=10.1086/444032|author=[[Stephen M. Stigler]]|title=A Historical View of Statistical Concepts in Psychology and Educational Research| journal=American Journal of Education| volume=101|issue=1|date=November 1992|pages=60–70}}</ref><ref>{{cite journal|doi=10.1086/383850|author=Trudy Dehue|title=Deception, Efficiency, and Random Groups: Psychology and the Gradual Origination of the Random Group Design|journal=[[Isis (journal)|Isis]]|volume=88|issue=4|date=December 1997|pages=653–673|pmid=9519574}}</ref>
| |||