Reproducibility: Difference between revisions

Revision as of 19:51, 13 November 2023

Revision as of 03:28, 17 November 2023

Line 2:

{{About|the reproducibility of scientific research results|reproductive capacity of organisms|fertility|and|fecundity|reproducibility in the context of computer software|Reproducible builds}}

'''Reproducibility''', closely related to '''replicability''' and '''repeatability''', is a major principle underpinning the [[scientific method]]. For the findings of a study to be reproducible means that results obtained by an [[experiment]] or an [[observational study]] or in a [[statistical analysis]] of a [[data set]] should be achieved again with a high degree of reliability when the study is replicated. There are different kinds of replication<ref>{{Cite journal|last1=Tsang|first1=Eric W. K.|last2=Kwan|first2=Kai-man|date=1999|title=Replication and Theory Development in Organizational Science: A Critical Realist Perspective|url=http://dx.doi.org/10.5465/amr.1999.2553252|journal=Academy of Management Review|volume=24|issue=4|pages=759–780|doi=10.5465/amr.1999.2553252|issn=0363-7425}}</ref> but typically replication studies involve different researchers using the same methodology. Only after one or several such successful replications should a result be recognized as scientific knowledge.

With a narrower scope, ''reproducibility'' has been introduced in [[computational science]]s: Any results should be documented by making all data and code available in such a way that the computations can be executed again with identical results.

Line 48:

===Reproducible research in practice===

Psychology has seen a renewal of internal concerns about non-reproducibleirreproducible results (see the entry on [[replicability crisis]] for empirical results on success rates of replications). Researchers showed in a 2006 study that, of 141 authors of a publication from the American Psychological Association (APA) empirical articles, 103 (73%) did not respond with their data over a six-month period.<ref>{{Cite journal|last1=Wicherts |first1=J. M. |last2=Borsboom |first2=D. |last3=Kats |first3=J. |last4=Molenaar |first4=D. |title=The poor availability of psychological research data for reanalysis |doi=10.1037/0003-066X.61.7.726 |journal=American Psychologist |volume=61 |issue=7 |pages=726–728 |year=2006 |pmid=17032082}}</ref> In a follow-up study published in 2015, it was found that 246 out of 394 contacted authors of papers in APA journals did not share their data upon request (62%).<ref>{{Cite journal|last1=Vanpaemel |first1=W. |last2=Vermorgen |first2=M. |last3=Deriemaecker |first3=L. |last4=Storms |first4=G. |title=Are we wasting a good crisis? The availability of psychological research data after the storm |doi=10.1525/collabra.13 |journal=Collabra |volume=1 |issue=1 |pages=1–5 |year=2015 |doi-access=free}}</ref> In a 2012 paper, it was suggested that researchers should publish data along with their works, and a dataset was released alongside as a demonstration.<ref>{{Cite journal|last1=Wicherts |first1=J. M. |last2=Bakker |first2=M. |doi=10.1016/j.intell.2012.01.004 |title=Publish (your data) or (let the data) perish! Why not publish your data too? |journal=Intelligence |volume=40 |issue=2 |pages=73–76 |year=2012}}</ref> In 2017, an article published in ''[[Scientific Data (journal)|Scientific Data]]'' suggested that this may not be sufficient and that the whole analysis context should be disclosed.<ref>{{cite journal|last1=Pasquier|first1=Thomas|last2=Lau|first2=Matthew K.|last3=Trisovic|first3=Ana|last4=Boose|first4=Emery R.|last5=Couturier|first5=Ben|last6=Crosas|first6=Mercè|last7=Ellison|first7=Aaron M.|last8=Gibson|first8=Valerie|last9=Jones|first9=Chris R.|last10=Seltzer|first10=Margo|title=If these data could talk|journal=Scientific Data|date=5 September 2017|volume=4|issue=1 |pages=170114|doi=10.1038/sdata.2017.114|pmid=28872630|pmc=5584398|bibcode=2017NatSD...470114P}}</ref>

In economics, concerns have been raised in relation to the credibility and reliability of published research. In other sciences, reproducibility is regarded as fundamental and is often a prerequisite to research being published, however in economic sciences it is not seen as a priority of the greatest importance. Most peer-reviewed economic journals do not take any substantive measures to ensure that published results are reproducible, however, the top economics journals have been moving to adopt mandatory data and code archives.<ref>{{cite journal |last1=McCullough |first1=Bruce |title=Open Access Economics Journals and the Market for Reproducible Economic Research |journal=Economic Analysis and Policy |date=March 2009 |volume=39 |issue=1 |pages=117–126 |doi=10.1016/S0313-5926(09)50047-1|doi-access= }}</ref> There is low or no incentives for researchers to share their data, and authors would have to bear the costs of compiling data into reusable forms. Economic research is often not reproducible as only a portion of journals have adequate disclosure policies for datasets and program code, and even if they do, authors frequently do not comply with them or they are not enforced by the publisher. A Study of 599 articles published in 37 peer-reviewed journals revealed that while some journals have achieved significant compliance rates, significant portion have only partially complied, or not complied at all. On an article level, the average compliance rate was 47.5%; and on a journal level, the average compliance rate was 38%, ranging from 13% to 99%.<ref>{{cite journal |last1=Vlaeminck |first1=Sven |last2=Podkrajac |first2=Felix |title=Journals in Economic Sciences: Paying Lip Service to Reproducible Research? |journal=IASSIST Quarterly |date=2017-12-10 |volume=41 |issue=1–4 |page=16 |doi=10.29173/iq6 |url=https://iassistquarterly.com/index.php/iassist/article/view/6/905|hdl=11108/359 |s2cid=96499437 |hdl-access=free }}</ref>

Line 59:

Reproducible research is key to new discoveries in [[pharmacology]]. A Phase I discovery will be followed by Phase II reproductions as a drug develops towards commercial production. In recent decades Phase II success has fallen from 28% to 18%. A 2011 study found that 65% of medical studies were inconsistent when re-tested, and only 6% were completely reproducible.<ref>{{Cite journal|last1=Prinz |first1=F. |last2=Schlange |first2=T. |last3=Asadullah |first3=K. |doi=10.1038/nrd3439-c1 |title=Believe it or not: How much can we rely on published data on potential drug targets? |journal=Nature Reviews Drug Discovery |volume=10 |issue=9 |page=712 |year=2011 |pmid=21892149 |doi-access=free}}</ref>

==Noteworthy non-reproducibleirreproducible results==

[[Hideyo Noguchi]] became famous for correctly identifying the bacterial agent of [[syphilis]], but also claimed that he could culture this agent in his laboratory. Nobody else has been able to produce this latter result.<ref name="Tan Furubayashi 2014 pp. 550–551">{{cite journal|last1=Tan |first1=SY |last2=Furubayashi |first2=J |title=Hideyo Noguchi (1876-1928): Distinguished bacteriologist |journal=Singapore Medical Journal |volume=55 |issue=10 |year=2014 |issn=0037-5675 |pmid=25631898 |pmc=4293967 |doi=10.11622/smedj.2014140 |pages=550–551}}</ref>