Content deleted Content added
| (41 intermediate revisions by 29 users not shown) | |||
Line 1:
{{Short description|Aspect of scientific research}}
{{About|the reproducibility of scientific research results|reproductive capacity of organisms|fertility|and|fecundity|reproducibility in the context of computer software|Reproducible builds}}
'''Reproducibility''', closely related to '''replicability''' and '''repeatability''', is a major principle underpinning the [[scientific method]]. For the findings of a study to be reproducible means that results obtained by an [[experiment]] or an [[observational study]] or in a [[statistical analysis]] of a [[data set]] should be achieved again with a high degree of reliability when the study is replicated. There are different kinds of replication<ref>{{Cite journal|last1=Tsang|first1=Eric W. K.|last2=Kwan|first2=Kai-man|date=1999|title=Replication and Theory Development in Organizational Science: A Critical Realist Perspective|url=http://dx.doi.org/10.5465/amr.1999.2553252|journal=Academy of Management Review|volume=24|issue=4|pages=759–780|doi=10.5465/amr.1999.2553252|issn=0363-7425|url-access=subscription}}</ref> but typically replication studies involve different researchers using the same methodology. Only after one or several such successful replications should a result be recognized as scientific knowledge.
==History==
[[File:Boyle air pump.jpg|thumb|right|Boyle's air pump was, in terms of the 17th century, a complicated and expensive scientific apparatus, making reproducibility of results difficult.]]
The first to stress the importance of reproducibility in science was the Anglo-Irish chemist [[Robert Boyle]], in [[England]] in the 17th century. Boyle's [[air pump]] was designed to generate and study [[vacuum]], which at the time was a very controversial concept. Indeed, distinguished philosophers such as [[René Descartes]] and [[Thomas Hobbes]] denied the very possibility of vacuum existence. [[History of science|Historians of science]] [[Steven Shapin]] and [[Simon Schaffer]], in their 1985 book ''[[Leviathan and the Air-Pump]]'', describe the debate between Boyle and Hobbes, ostensibly over the nature of vacuum, as fundamentally an argument about how useful knowledge should be gained. Boyle, a pioneer of the [[experimental method]], maintained that the foundations of knowledge should be constituted by experimentally produced facts, which can be made believable to a scientific community by their reproducibility. By repeating the same experiment over and over again, Boyle argued, the certainty of fact will emerge.
The air pump, which in the 17th century was a complicated and expensive apparatus to build, also led to one of the first documented disputes over the reproducibility of a particular [[scientific phenomenon]]. In the 1660s, the Dutch scientist [[Christiaan Huygens]] built his own air pump in [[Amsterdam]], the first one outside the direct management of Boyle and his assistant at the time [[Robert Hooke]]. Huygens reported an effect he termed "anomalous suspension", in which water appeared to levitate in a glass jar inside his air pump (in fact suspended over an air bubble), but Boyle and Hooke could not replicate this phenomenon in their own pumps. As Shapin and Schaffer describe, "it became clear that unless the phenomenon could be produced in England with one of the two pumps available, then no one in England would accept the claims Huygens had made, or his competence in working the pump". Huygens was finally invited to England in 1663, and under his personal guidance Hooke was able to replicate anomalous suspension of water. Following this Huygens was elected a Foreign Member of the [[Royal Society]]. However, Shapin and Schaffer also note that "the accomplishment of replication was dependent on contingent acts of judgment. One cannot write down a formula saying when replication was or was not achieved".<ref>[[Steven Shapin]] and [[Simon Schaffer]], ''[[Leviathan and the Air-Pump]]'', Princeton University Press, Princeton, New Jersey (1985).</ref>
Line 19 ⟶ 15:
''Replicability'' and ''repeatability'' are related terms broadly or loosely synonymous with reproducibility (for example, among the general public), but they are often usefully differentiated in more precise senses, as follows.
▲When new data is obtained in the attempt to achieve it, the term ''replicability'' is often used, and the new study is a ''replication'' or ''replicate'' of the original one. Obtaining the same results when analyzing the data set of the original study again with the same procedures, many authors use the term ''reproducibility'' in a narrow, technical sense coming from its use in computational research.
==Measures of reproducibility and repeatability==
In chemistry, the terms reproducibility and repeatability are used with a specific quantitative meaning.<ref>{{Cite journal |last= |first= |title=IUPAC - reproducibility (R05305) |url=https://goldbook.iupac.org/terms/view/R05305 |access-date=2022-03-04 |website=[[International Union of Pure and Applied Chemistry]]|doi= 10.1351/goldbook.R05305|doi-access=free|url-access=subscription}}</ref> In inter-laboratory experiments, a concentration or other quantity of a chemical substance is measured repeatedly in different laboratories to assess the variability of the measurements. Then, the standard deviation of the difference between two values obtained within the same laboratory is called repeatability. The standard deviation for the difference between two measurement from different laboratories is called ''reproducibility''.<ref name="ASTM E177">{{cite web|url=https://www.astm.org/Standards/E177.htm |title=Standard Practice for Use of the Terms Precision and Bias in ASTM Test Methods |year=2014 |author=Subcommittee E11.20 on Test Method Evaluation and Quality Control |publisher=ASTM International |id=ASTM E177}}{{
These measures are related to the more general concept of [[variance component]]s in [[metrology]].
Line 34 ⟶ 27:
===Reproducible research method===
The term ''reproducible research'' refers to the idea that scientific results should be documented in such a way that their deduction is fully transparent. This requires a detailed description of the methods used to obtain the data<ref>{{Cite journal|last=King|first=Gary|date=1995|title=Replication, Replication|journal=PS: Political Science and Politics|volume=28|issue=3|pages=444–452|doi=10.2307/420301|jstor=420301|s2cid=250480339 |issn=1049-0965|url=http://nrs.harvard.edu/urn-3:HUL.InstRepos:4266312}}</ref><ref>{{cite journal|last1=Kühne |first1=Martin |last2=Liehr |first2=Andreas W. |year=2009 |title=Improving the Traditional Information Management in Natural Sciences |doi=10.2481/dsj.8.18 |journal=Data Science Journal |volume=8 |issue=1 |pages=18–27 |url=https://datascience.codata.org/jms/article/download/dsj.8.18/198 |doi-access=free}}</ref>
and making the full dataset and the code to calculate the results easily accessible.<ref>{{cite journal|last1=Fomel |first1=Sergey |author-link2=Jon Claerbout |last2=Claerbout |first2=Jon |year=2009 |title=Guest Editors' Introduction: Reproducible Research |journal=Computing in Science and Engineering |volume=11 |issue=1 |pages=5–7 |doi=10.1109/MCSE.2009.14 |bibcode=2009CSE....11a...5F}}</ref><ref name="buckheit1995" /><ref>{{cite journal|title=The Yale Law School Round Table on Data and Core Sharing: "Reproducible Research" |journal=Computing in Science and Engineering |volume=12 |issue=5 |pages=8–12 |doi=10.1109/MCSE.2010.113 |year=2010 |doi-access=
This is the essential part of [[open science]].
Line 41 ⟶ 34:
A basic workflow for reproducible research involves data acquisition, data processing and data analysis. Data acquisition primarily consists of obtaining primary data from a primary source such as surveys, field observations, experimental research, or obtaining data from an existing source. Data processing involves the processing and review of the raw data collected in the first stage, and includes data entry, data manipulation and filtering and may be done using software. The data should be digitized and prepared for data analysis. Data may be analysed with the use of software to interpret or visualise statistics or data to produce the desired results of the research such as quantitative results including figures and tables. The use of software and automation enhances the reproducibility of research methods.<ref>{{cite book |last1=Kitzes |first1=Justin |last2=Turek |first2=Daniel |last3=Deniz |first3=Fatma |title=The practice of reproducible research case studies and lessons from the data-intensive sciences |date=2018 |publisher=University of California Press |___location=Oakland, California |isbn=9780520294745 |pages=19–30 |jstor=10.1525/j.ctv1wxsc7 |url=http://www.jstor.org/stable/10.1525/j.ctv1wxsc7}}</ref>
There are systems that facilitate such documentation, like the [[R (programming language)|R]] [[Markdown]] language<ref>{{cite journal|last1=Marwick|first1=Ben|last2=Boettiger|first2=Carl|last3=Mullen|first3=Lincoln|title=Packaging data analytical work reproducibly using R (and friends)|journal=The American Statistician|volume=72|date=29 September 2017|pages=80–88|doi=10.1080/00031305.2017.1375986|s2cid=125412832|url=http://ro.uow.edu.au/cgi/viewcontent.cgi?article=6445&context=smhpapers}}</ref>
or the [[Jupyter]] notebook.<ref>{{cite conference|title=Jupyter Notebooks–a publishing format for reproducible computational workflows |url=https://eprints.soton.ac.uk/403913/1/STAL9781614996491-0087.pdf |archive-url=https://web.archive.org/web/20180110174609/https://eprints.soton.ac.uk/403913/1/STAL9781614996491-0087.pdf |archive-date=2018-01-10 |url-status=live |book-title=Positioning and Power in Academic Publishing: Players, Agents and Agendas |editor1-last=Loizides |editor1-first=F |editor2-last=Schmidt |editor2-first=B |publisher=IOS Press |last1=Kluyver |first1=Thomas |last2=Ragan-Kelley |first2=Benjamin |last3=Perez |first3=Fernando |last4=Granger |first4=Brian |last5=Bussonnier
|first5=Matthias |last6=Frederic |first6=Jonathan |last7=Kelley
Line 48 ⟶ 41:
===Reproducible research in practice===
Psychology has seen a renewal of internal concerns about irreproducible results (see the entry on [[replicability crisis]] for empirical results on success rates of replications). Researchers showed in a 2006 study that, of 141 authors of a publication from the American Psychological Association (APA) empirical articles, 103 (73%) did not respond with their data over a six-month period.<ref>{{Cite journal|last1=Wicherts |first1=J. M. |last2=Borsboom |first2=D. |last3=Kats |first3=J. |last4=Molenaar |first4=D. |title=The poor availability of psychological research data for reanalysis |doi=10.1037/0003-066X.61.7.726 |journal=American Psychologist |volume=61 |issue=7 |pages=726–728 |year=2006 |pmid=17032082}}</ref> In a follow
In economics, concerns have been raised in relation to the credibility and reliability of published research. In other sciences, reproducibility is regarded as fundamental and is often a prerequisite to research being published, however in economic sciences it is not seen as a priority of the greatest importance. Most peer-reviewed economic journals do not take any substantive measures to ensure that published results are reproducible, however, the top economics journals have been moving to adopt mandatory data and code archives.<ref>{{cite journal |last1=McCullough |first1=Bruce |title=Open Access Economics Journals and the Market for Reproducible Economic Research |journal=Economic Analysis and Policy |date=March 2009 |volume=39 |issue=1 |pages=117–126 |doi=10.1016/S0313-5926(09)50047-1|doi-access=
A 2018 study published in the journal ''[[PLOS ONE]]'' found that 14.4% of a sample of public health statistics researchers had shared their data or code or both.<ref>{{Cite journal|date=2018|title=Use of reproducible research practices in public health: A survey of public health analysts.|journal=PLOS ONE|volume=13|issue=9|pages=e0202447|issn=1932-6203|oclc=7891624396|bibcode=2018PLoSO..1302447H|last1=Harris|first1=Jenine K.|last2=Johnson|first2=Kimberly J.|last3=Carothers|first3=Bobbi J.|last4=Combs|first4=Todd B.|last5=Luke|first5=Douglas A.|last6=Wang|first6=Xiaoyan|doi=10.1371/journal.pone.0202447|pmid=30208041|pmc=6135378|doi-access=free}}</ref>
There have been initiatives to improve reporting and hence reproducibility in the medical literature for many years, beginning with the [[Consolidated Standards of Reporting Trials|CONSORT]] initiative, which is now part of a wider initiative, the [[EQUATOR Network]].
This group has recently turned its attention to how better reporting might reduce waste in research,<ref>{{Cite web|title=Research Waste/EQUATOR Conference {{!}} Research Waste |url=http://researchwaste.net/research-wasteequator-conference/ |website=researchwaste.net |url-status=dead |archive-url=https://web.archive.org/web/20161029015313/http://researchwaste.net:80/research-wasteequator-conference/ |archive-date=29 October 2016}}</ref> especially biomedical research.
Reproducible research is key to new discoveries in [[pharmacology]]. A Phase I discovery will be followed by Phase II reproductions as a drug develops towards commercial production. In recent decades Phase II success has fallen from 28% to 18%. A 2011 study found that 65% of medical studies were inconsistent when re-tested, and only 6% were completely reproducible.<ref>{{Cite journal|last1=Prinz |first1=F. |last2=Schlange |first2=T. |last3=Asadullah |first3=K. |doi=10.1038/nrd3439-c1 |title=Believe it or not: How much can we rely on published data on potential drug targets? |journal=Nature Reviews Drug Discovery |volume=10 |issue=9 |page=712 |year=2011 |pmid=21892149 |doi-access=free}}</ref>
Some efforts have been made to increase replicability beyond the social and biomedical sciences. Studies in the humanities tend to rely more on expertise and hermeneutics which may make replicability more difficult. Nonetheless, some efforts have been made to call for more transparency and documentation in the humanities.<ref>{{Cite journal |last1=Van Eyghen |first1=Hans |last2= Van den Brink |first2=Gijsbert |last3= Peels |first3= Rik |title=Brooke on the Merton Thesis: A Direct Replication of John Hedley Brooke's Chapter on Scientific and Religious Reform |journal=Zygon: Journal of Religion and Science |volume=59 |issue=2 |year=2024|url=https://www.zygonjournal.org/article/id/11497/| doi=10.16995/zygon.11497|doi-access=free }}</ref>
==Noteworthy irreproducible results==
Line 64 ⟶ 59:
In March 1989, [[University of Utah]] chemists Stanley Pons and Martin Fleischmann reported the production of excess heat that could only be explained by a nuclear process ("[[cold fusion]]"). The report was astounding given the simplicity of the equipment: it was essentially an [[electrolysis]] cell containing [[heavy water]] and a [[palladium]] [[cathode]] which rapidly absorbed the [[deuterium]] produced during electrolysis. The news media reported on the experiments widely, and it was a front-page item on many newspapers around the world (see [[science by press conference]]). Over the next several months others tried to replicate the experiment, but were unsuccessful.<ref>{{cite journal|title=Physicists Debunk Claim Of a New Kind of Fusion|newspaper=New York Times|last=Browne|first=Malcolm|url=http://partners.nytimes.com/library/national/science/050399sci-cold-fusion.html|date=3 May 1989|access-date=3 February 2017}}</ref>
[[Nikola Tesla]] claimed as early as 1899 to have used a high frequency current to light gas-filled lamps from over {{convert|25|mi|km}} away [[Wireless energy transfer|without using wires]]. In 1904 he built [[Wardenclyffe Tower]] on [[Shoreham, New York|Long Island]] to demonstrate means to send and receive power without connecting wires. The facility was never fully operational and was not completed due to economic problems, so no attempt to reproduce his first result was ever carried out.<ref>[[Margaret Cheney (author)|Cheney, Margaret]] (1999), ''Tesla, Master of Lightning'', New York: Barnes & Noble Books, {{ISBN|0-7607-1005-8}}, pp. 107.; "Unable to overcome his financial burdens, he was forced to close the laboratory in 1905."</ref>
Other examples which contrary evidence has refuted the original claim:
* [[N-ray]]s, a hypothesized form of radiation subsequently found to be illusory
* [[Polywater]], a hypothesized polymerized form of water found to be just water with common contaminations
* [[Stimulus-triggered acquisition of pluripotency]], revealed to be the result of fraud
* [[GFAJ-1]], a bacterium that could purportedly incorporate [[arsenic biochemistry|arsenic]] into its DNA in place of phosphorus
* [[MMR vaccine controversy
* [[Schön scandal]] — semiconductor "breakthroughs" revealed to be fraudulent
* [[Power posing]] — a [[social psychology]] phenomenon that went viral after being the subject of a very popular [[TED talk]], but was unable to be replicated in dozens of studies<ref name="NYT_2017_Cuddy">{{cite news |url=https://www.nytimes.com/2017/10/18/magazine/when-the-revolution-came-for-amy-cuddy.html |title=When the Revolution Came for Amy Cuddy |first=Susan |last=Dominus |date=October 18, 2017 |work=New York Times Magazine}}</ref>
Line 104 ⟶ 101:
==Further reading==
* {{cite web|title = Scientists on Science: Reproducibility|date = October 2006|url = https://arstechnica.com/science/2006/10/5744/|author = Timmer, John|work = [[Ars Technica]]}}
* {{cite web|title = Is redoing scientific research the best way to find truth? During replication attempts, too many studies fail to pass muster |date = January 2015 |url = https://www.sciencenews.org/article/redoing-scientific-research-best-way-find-truth |author = Saey, Tina Hesman |work = [[Science News]]}} "Science is not irrevocably broken, [epidemiologist John Ioannidis] asserts. It just needs some improvements. "Despite the fact that
==External links==
{{Wiktionary}}
* [https://www.cos.io/our-services/top-guidelines Transparency and Openness Promotion Guidelines] from the [[Center for Open Science]]
* [https://www.nist.gov/pml/nist-technical-note-1297 Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results] of the [[National Institute of Standards and Technology]]
* [https://cTuning.org/ae Reproducible papers with artifacts] by the [[CTuning foundation]]
* [https://www.reproducibleresearch.net ReproducibleResearch.net]
{{Medical research studies}}
| |||