Content deleted Content added
m Rollback edit(s) by Letscontributes (talk): LLM output (UV 0.1.6) |
|||
| (181 intermediate revisions by 81 users not shown) | |||
Line 1:
{{
[[File:Common_Test_Statistics_Chart.png|thumb|The above image shows a table with some of the most common [[test statistic]]s and their corresponding tests or models.]]
A '''statistical hypothesis test''' is a method of statistical inference used to decide whether the data provide sufficient evidence to reject a particular hypothesis. A statistical hypothesis test typically involves a calculation of a [[test statistic]]. Then a decision is made, either by comparing the test statistic to a [[Critical value (statistics)|critical value]] or equivalently by evaluating a [[p-value|''p''-value]] computed from the test statistic. Roughly 100 [[list of statistical tests|specialized statistical tests]] are in use and noteworthy.<ref>{{cite book |last1=Lewis |first1=Nancy D. |last2=Lewis |first2=Nigel Da Costa |last3=Lewis |first3=N. D. |title=100 Statistical Tests in R: What to Choose, how to Easily Calculate, with Over 300 Illustrations and Examples |date=2013 |publisher=Heather Hills Press |isbn=978-1-4840-5299-0 |url=https://books.google.com/books?id=wIs7mwEACAAJ |language=en}}</ref><ref>{{cite book |last1=Kanji |first1=Gopal K. |title=100 Statistical Tests |date=18 July 2006 |publisher=SAGE |isbn=978-1-4462-2250-8 |url=https://books.google.com/books?id=c16MhjA4pHgC |language=en}}</ref>
==History==
{{See also|History of probability}}
While hypothesis testing was popularized early in the 20th century, early forms were used in the 1700s. The first use is credited to [[John Arbuthnot]] (1710),<ref name="Bellhouse2001">{{Citation|last=Bellhouse|first=P.|title=in Statisticians of the Centuries by C.C. Heyde and E. Seneta|pages=39–42|year=2001|chapter=John Arbuthnot|publisher=Springer|isbn=978-0-387-95329-8}}
</ref> followed by [[Pierre-Simon Laplace]] (1770s), in analyzing the [[human sex ratio]] at birth; see {{slink||Human sex ratio}}.
===Choice of null hypothesis===
[[Paul Meehl]] has argued that the [[epistemological]] importance of the choice of null hypothesis has gone largely unacknowledged. When the null hypothesis is predicted by theory, a more precise experiment will be a more severe test of the underlying theory. When the null hypothesis defaults to "no difference" or "no effect", a more precise experiment is a less severe test of the theory that motivated performing the experiment.<ref>{{cite journal|last=Meehl|first=P|year=1990|title=Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles That Warrant It|url=http://rhowell.ba.ttu.edu/meehl1.pdf|journal=Psychological Inquiry|volume=1|issue=2|pages=108–141|doi=10.1207/s15327965pli0102_1}}</ref> An examination of the origins of the latter practice may therefore be useful:
'''1778:''' [[Pierre Laplace]] compares the birthrates of boys and girls in multiple European cities. He states: "it is natural to conclude that these possibilities are very nearly in the same ratio". Thus, the null hypothesis in this case that the birthrates of boys and girls should be equal given "conventional wisdom".<ref name="Laplace 1778" />
'''1900:''' [[Karl Pearson]] develops the [[chi squared test]] to determine "whether a given form of frequency curve will effectively describe the samples drawn from a given population." Thus the null hypothesis is that a population is described by some distribution predicted by theory. He uses as an example the numbers of five and sixes in the [[Walter Frank Raphael Weldon|Weldon dice throw data]].<ref name="Pearson 1900">{{cite journal|last=Pearson|first=K|year=1900|title=On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling|url=http://www.economics.soton.ac.uk/staff/aldrich/1900.pdf|journal=The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science|volume=5|issue=50|pages=157–175|doi=10.1080/14786440009463897}}</ref>
'''1904:''' [[Karl Pearson]] develops the concept of "[[contingency table|contingency]]" in order to determine whether outcomes are [[statistical independence|independent]] of a given categorical factor. Here the null hypothesis is by default that two things are unrelated (e.g. scar formation and death rates from smallpox).<ref name="Pearson 1904">{{cite journal|last=Pearson|first=K|year=1904|title=On the Theory of Contingency and Its Relation to Association and Normal Correlation|url=https://archive.org/details/cu31924003064833|journal=Drapers' Company Research Memoirs Biometric Series|volume=1|pages=1–35}}</ref> The null hypothesis in this case is no longer predicted by theory or conventional wisdom, but is instead the [[principle of indifference]] that led [[Ronald Fisher|Fisher]] and others to dismiss the use of "inverse probabilities".<ref>{{cite journal|last=Zabell|first=S|year=1989|title=R. A. Fisher on the History of Inverse Probability|journal=Statistical Science|volume=4|issue=3|pages=247–256|doi=10.1214/ss/1177012488|jstor=2245634|doi-access=free}}</ref>
=={{anchor|Controversy}}Modern origins and early controversy==
Modern significance testing is largely the product of [[Karl Pearson]] ([[p-value|''p''-value]], [[Pearson's chi-squared test]]), [[William Sealy Gosset]] ([[Student's t-distribution]]), and [[Ronald Fisher]] ("[[null hypothesis]]", [[analysis of variance]], "[[statistical significance|significance test]]"), while hypothesis testing was developed by [[Jerzy Neyman]] and [[Egon Pearson]] (son of Karl). Ronald Fisher began his life in statistics as a Bayesian (Zabell 1992), but Fisher soon grew disenchanted with the subjectivity involved (namely use of the [[principle of indifference]] when determining prior probabilities), and sought to provide a more "objective" approach to inductive inference.<ref name="ftp.isds.duke">Raymond Hubbard, [[M. J. Bayarri]], ''[http://ftp.isds.duke.edu/WorkingPapers/03-26.pdf P Values are not Error Probabilities] {{webarchive|url=https://web.archive.org/web/20130904000350/http://ftp.isds.duke.edu/WorkingPapers/03-26.pdf|date=September 4, 2013}}''. A working paper that explains the difference between Fisher's evidential ''p''-value and the Neyman–Pearson Type I error rate <math>\alpha</math>.</ref>
Fisher emphasized rigorous experimental design and methods to extract a result from few samples assuming [[Gaussian distribution]]s. Neyman (who teamed with the younger Pearson) emphasized mathematical rigor and methods to obtain more results from many samples and a wider range of distributions. Modern hypothesis testing is an inconsistent hybrid of the Fisher vs Neyman/Pearson formulation, methods and terminology developed in the early 20th century.
Fisher popularized the "significance test". He required a null-hypothesis (corresponding to a population frequency distribution) and a sample. His (now familiar) calculations determined whether to reject the null-hypothesis or not. Significance testing did not utilize an alternative hypothesis so there was no concept of a [[Type II error]] (false negative).
The ''p''-value was devised as an informal, but objective, index meant to help a researcher determine (based on other knowledge) whether to modify future experiments or strengthen one's [[Fiducial inference|faith]] in the null hypothesis.<ref name="Fisher 1955 69–78">{{cite journal|last=Fisher|first=R|year=1955|title=Statistical Methods and Scientific Induction|url=http://www.phil.vt.edu/dmayo/PhilStatistics/Triad/Fisher%201955.pdf|journal=Journal of the Royal Statistical Society, Series B|volume=17|issue=1|pages=69–78|doi=10.1111/j.2517-6161.1955.tb00180.x}}</ref> Hypothesis testing (and Type I/II errors) was devised by Neyman and Pearson as a more objective alternative to Fisher's ''p''-value, also meant to determine researcher behaviour, but without requiring any [[inductive inference]] by the researcher.<ref name="Neyman 289–337">{{cite journal|last1=Neyman|first1=J|last2=Pearson|first2=E. S.|date=January 1, 1933|title=On the Problem of the most Efficient Tests of Statistical Hypotheses|journal=[[Philosophical Transactions of the Royal Society A]]|volume=231|issue=694–706|pages=289–337|bibcode=1933RSPTA.231..289N|doi=10.1098/rsta.1933.0009|doi-access=free}}</ref><ref>{{cite journal|last=Goodman|first=S N|date=June 15, 1999|title=Toward evidence-based medical statistics. 1: The P Value Fallacy|journal=Ann Intern Med|volume=130|issue=12|pages=995–1004|doi=10.7326/0003-4819-130-12-199906150-00008|pmid=10383371|s2cid=7534212}}</ref>
Neyman & Pearson considered a different problem to Fisher (which they called "hypothesis testing"). They initially considered two simple hypotheses (both with frequency distributions). They calculated two probabilities and typically selected the hypothesis associated with the higher probability (the hypothesis more likely to have generated the sample). Their method always selected a hypothesis. It also allowed the calculation of both types of error probabilities.
Fisher and Neyman/Pearson clashed bitterly. Neyman/Pearson considered their formulation to be an improved generalization of significance testing (the defining paper<ref name="Neyman 289–337" /> was [[Neyman–Pearson lemma|abstract]]; Mathematicians have generalized and refined the theory for decades<ref name="Lehmann93" />). Fisher thought that it was not applicable to scientific research because often, during the course of the experiment, it is discovered that the initial assumptions about the null hypothesis are questionable due to unexpected sources of error. He believed that the use of rigid reject/accept decisions based on models formulated before data is collected was incompatible with this common scenario faced by scientists and attempts to apply this method to scientific research would lead to mass confusion.<ref>{{cite journal|last=Fisher|first=R N|year=1958|title=The Nature of Probability|url=http://www.york.ac.uk/depts/maths/histstat/fisher272.pdf|journal=Centennial Review|volume=2|pages=261–274|quote=We are quite in danger of sending highly trained and highly intelligent young men out into the world with tables of erroneous numbers under their arms, and with a dense fog in the place where their brains ought to be. In this century, of course, they will be working on guided missiles and advising the medical profession on the control of disease, and there is no limit to the extent to which they could impede every sort of national effort.}}
</ref>
The dispute between Fisher and Neyman–Pearson was waged on philosophical grounds, characterized by a philosopher as a dispute over the proper role of models in statistical inference.<ref name="Lenhard">{{cite journal|last=Lenhard|first=Johannes|year=2006|title=Models and Statistical Inference: The Controversy between Fisher and Neyman–Pearson|journal=Br. J. Philos. Sci.|volume=57|pages=69–91|doi=10.1093/bjps/axi152|s2cid=14136146}}</ref>
Events intervened: Neyman accepted a position in the [[University of California, Berkeley]] in 1938, breaking his partnership with Pearson and separating the disputants (who had occupied the same building). [[World War II]] provided an intermission in the debate. The dispute between Fisher and Neyman terminated (unresolved after 27 years) with Fisher's death in 1962. Neyman wrote a well-regarded eulogy.<ref>{{cite journal|last1=Neyman|first1=Jerzy|year=1967|title=RA Fisher (1890—1962): An Appreciation.|journal=Science|volume=156|issue=3781|pages=1456–1460|bibcode=1967Sci...156.1456N|doi=10.1126/science.156.3781.1456|pmid=17741062|s2cid=44708120}}</ref> Some of Neyman's later publications reported ''p''-values and significance levels.<ref>{{cite journal|last1=Losavich|first1=J. L.|last2=Neyman|first2=J.|last3=Scott|first3=E. L.|last4=Wells|first4=M. A.|year=1971|title=Hypothetical explanations of the negative apparent effects of cloud seeding in the Whitetop Experiment.|journal=Proceedings of the National Academy of Sciences of the United States of America|volume=68|issue=11|pages=2643–2646|bibcode=1971PNAS...68.2643L|doi=10.1073/pnas.68.11.2643|pmc=389491|pmid=16591951|doi-access=free}}</ref>
==={{anchor|NHST}}Null hypothesis significance testing (NHST)===
The modern version of hypothesis testing is generally called the '''null hypothesis significance testing (NHST)'''<ref name=nickerson /> and is a hybrid of the Fisher approach with the Neyman-Pearson approach. In 2000, [[Raymond S. Nickerson]] wrote an article stating that NHST was (at the time) "arguably the most widely used method of analysis of data collected in psychological experiments and has been so for about 70 years" and that it was at the same time "very controversial".<ref name=nickerson />
This fusion resulted from confusion by writers of statistical textbooks (as predicted by Fisher) beginning in the 1940s<ref name="Halpin 625–653">{{cite journal|last1=Halpin|first1=P F|last2=Stam|first2=HJ|date=Winter 2006|title=Inductive Inference or Inductive Behavior: Fisher and Neyman: Pearson Approaches to Statistical Testing in Psychological Research (1940–1960)|journal=The American Journal of Psychology|volume=119|issue=4|pages=625–653|doi=10.2307/20445367|jstor=20445367|pmid=17286092}}</ref> (but [[Detection theory|signal detection]], for example, still uses the Neyman/Pearson formulation). Great conceptual differences and many caveats in addition to those mentioned above were ignored. Neyman and Pearson provided the stronger terminology, the more rigorous mathematics and the more consistent philosophy, but the subject taught today in introductory statistics has more similarities with Fisher's method than theirs.<ref name="Gigerenzer">{{cite book|last=Gigerenzer|first=Gerd|title=The Empire of Chance: How Probability Changed Science and Everyday Life|author2=Zeno Swijtink|author3=Theodore Porter|author4=Lorraine Daston|author5=John Beatty|author6=Lorenz Kruger|publisher=Cambridge University Press|year=1989|isbn=978-0-521-39838-1|pages=70–122|chapter=Part 3: The Inference Experts}}</ref>
Sometime around 1940,<ref name="Halpin 625–653" /> authors of statistical text books began combining the two approaches by using the ''p''-value in place of the [[test statistic]] (or data) to test against the Neyman–Pearson "significance level".
{| class="wikitable"
|+ A comparison between Fisherian, frequentist (Neyman–Pearson)
|-
! #
! Fisher's null hypothesis testing !! Neyman–Pearson decision theory
|-
| 1
| Set up a statistical null hypothesis. The null need not be a nil hypothesis (i.e., zero difference).
| Set up two statistical hypotheses, H1 and H2, and decide about α, β, and sample size before the experiment, based on subjective cost-benefit considerations. These define a rejection region for each hypothesis.
|-
| 2
| Report the exact level of significance (e.g. p = 0.051 or p = 0.049). Do not refer to "accepting" or "rejecting" hypotheses. If the result is "not significant", draw no conclusions and make no decisions, but suspend judgement until further data is available.
| If the data falls into the rejection region of H1, accept H2; otherwise accept H1. Accepting a hypothesis does not mean that you believe in it, but only that you act as if it were true.
|-
| 3
| Use this procedure only if little is known about the problem at hand, and only to draw provisional conclusions in the context of an attempt to understand the experimental situation.
| The usefulness of the procedure is limited among others to situations where you have a disjunction of hypotheses (e.g. either μ1 = 8 or μ2 = 10 is true) and where you can make meaningful cost-benefit trade-offs for choosing alpha and beta.
|}
==Philosophy==
Hypothesis testing and philosophy intersect. [[Inferential statistics]], which includes hypothesis testing, is applied probability. Both probability and its application are intertwined with philosophy. Philosopher [[David Hume]] wrote, "All knowledge degenerates into probability." Competing practical definitions of [[Probability#Interpretations|probability]] reflect philosophical differences. The most common application of hypothesis testing is in the scientific interpretation of experimental data, which is naturally studied by the [[philosophy of science]].
Fisher and Neyman opposed the subjectivity of probability. Their views contributed to the objective definitions. The core of their historical disagreement was philosophical.
Many of the philosophical criticisms of hypothesis testing are discussed by statisticians in other contexts, particularly [[correlation does not imply causation]] and the [[design of experiments]].
Hypothesis testing is of continuing interest to philosophers.<ref name="Lenhard" /><ref name="doi10.1093/bjps/axl003">
{{Cite journal|last1=Mayo|first1=D. G.|last2=Spanos|first2=A.|year=2006|title=Severe Testing as a Basic Concept in a Neyman–Pearson Philosophy of Induction|journal=The British Journal for the Philosophy of Science|volume=57|issue=2|pages=323–357|citeseerx=10.1.1.130.8131|doi=10.1093/bjps/axl003|s2cid=7176653}}</ref>
==Education==
{{main|Statistics education}}
Statistics is increasingly being taught in schools with hypothesis testing being one of the elements taught.<ref>[http://www.corestandards.org/the-standards/mathematics/hs-statistics-and-probability/introduction/ Mathematics > High School: Statistics & Probability > Introduction] {{webarchive|url=https://archive.today/20120728122912/http://www.corestandards.org/the-standards/mathematics/hs-statistics-and-probability/introduction/|date=July 28, 2012}} Common Core State Standards Initiative (relates to USA students)</ref><ref>[http://www.collegeboard.com/student/testing/ap/sub_stats.html College Board Tests > AP: Subjects > Statistics] The College Board (relates to USA students)</ref> Many conclusions reported in the popular press (political opinion polls to medical studies) are based on statistics. Some writers have stated that statistical analysis of this kind allows for thinking clearly about problems involving mass data, as well as the effective reporting of trends and inferences from said data, but caution that writers for a broad public should have a solid understanding of the field in order to use the terms and concepts correctly.<ref name="Huff8">{{cite book|last=Huff|first=Darrell|url=https://archive.org/details/howtoliewithstat00huff/page/8|title=How to lie with statistics|publisher=Norton|year=1993|isbn=978-0-393-31072-6|___location=New York|page=[https://archive.org/details/howtoliewithstat00huff/page/8 8]}}'Statistical methods and statistical terms are necessary in reporting the mass data of social and economic trends, business conditions, "opinion" polls, the census. But without writers who use the words with honesty and readers who know what they mean, the result can only be semantic nonsense.'</ref><ref name="S&C">{{cite book|last1=Snedecor|first1=George W.|title=Statistical Methods|last2=Cochran|first2=William G.|publisher=Iowa State University Press|year=1967|edition=6|___location=Ames, Iowa|page=3}} "...the basic ideas in statistics assist us in thinking clearly about the problem, provide some guidance about the conditions that must be satisfied if sound inferences are to be made, and enable us to detect many inferences that have no good logical foundation."</ref> An introductory college statistics class places much emphasis on hypothesis testing – perhaps half of the course. Such fields as literature and divinity now include findings based on statistical analysis (see the [[Bible Analyzer]]). An introductory statistics class teaches hypothesis testing as a cookbook process. Hypothesis testing is also taught at the postgraduate level. Statisticians learn how to create good statistical test procedures (like ''z'', Student's ''t'', ''F'' and chi-squared). Statistical hypothesis testing is considered a mature area within statistics,<ref name="Lehmann97" /> but a limited amount of development continues.
An academic study states that the cookbook method of teaching introductory statistics leaves no time for history, philosophy or controversy. Hypothesis testing has been taught as received unified method. Surveys showed that graduates of the class were filled with philosophical misconceptions (on all aspects of statistical inference) that persisted among instructors.<ref>{{cite journal|last1=Sotos|first1=Ana Elisa Castro|last2=Vanhoof|first2=Stijn|last3=Noortgate|first3=Wim Van den|last4=Onghena|first4=Patrick|year=2007|title=Students' Misconceptions of Statistical Inference: A Review of the Empirical Evidence from Research on Statistics Education|url=https://lirias.kuleuven.be/bitstream/123456789/136347/1/CastroSotos.pdf|journal=Educational Research Review|volume=2|issue=2|pages=98–113|doi=10.1016/j.edurev.2007.04.001}}</ref> While the problem was addressed more than a decade ago,<ref>{{cite journal|last=Moore|first=David S.|year=1997|title=New Pedagogy and New Content: The Case of Statistics|url=http://www.stat.auckland.ac.nz/~iase/publications/isr/97.Moore.pdf|journal=International Statistical Review|volume=65|issue=2|pages=123–165|doi=10.2307/1403333|jstor=1403333}}</ref> and calls for educational reform continue,<ref>{{Cite journal |last1=Hubbard |first1=Raymond|last2=Armstrong|first2=J. Scott|author-link2=J. Scott Armstrong|year=2006|title=Why We Don't Really Know What Statistical Significance Means: Implications for Educators|journal=Journal of Marketing Education|volume=28 |issue=2|pages=114–120 |doi=10.1177/0273475306288399 |hdl-access=free |hdl=2092/413 |s2cid=34729227}}</ref> students still graduate from statistics classes holding fundamental misconceptions about hypothesis testing.<ref>{{cite journal|last1=Sotos|first1=Ana Elisa Castro|last2=Vanhoof|first2=Stijn|last3=Noortgate|first3=Wim Van den|last4=Onghena|first4=Patrick|year=2009|title=How Confident Are Students in Their Misconceptions about Hypothesis Tests?|journal=Journal of Statistics Education|volume=17|doi=10.1080/10691898.2009.11889514|doi-access=free|number=2}}</ref> Ideas for improving the teaching of hypothesis testing include encouraging students to search for statistical errors in published papers, teaching the history of statistics and emphasizing the controversy in a generally dry subject.<ref name="Gigerenzer 2004 391–408">{{cite book|last=Gigerenzer|first=G.|title=The SAGE Handbook of Quantitative Methodology for the Social Sciences|year=2004|isbn=9780761923596|pages=391–408|chapter=The Null Ritual What You Always Wanted to Know About Significant Testing but Were Afraid to Ask|doi=10.4135/9781412986311|chapter-url=http://library.mpib-berlin.mpg.de/ft/gg/GG_Null_2004.pdf}}</ref>
Raymond S. Nickerson commented:
{{block quote|The debate about NHST has its roots in unresolved disagreements among major contributors to the development of theories of inferential statistics on which modern approaches are based. [[Gigerenzer]] et al. (1989) have reviewed in considerable detail the controversy between R. A. Fisher on the one hand and Jerzy Neyman and Egon Pearson on the other as well as the disagreements between both of these views and those of the followers of Thomas Bayes. They noted the remarkable fact that little hint of the historical and ongoing controversy is to be found in most textbooks that are used to teach NHST to its potential users. The resulting lack of an accurate historical perspective and understanding of the complexity and sometimes controversial philosophical foundations of various approaches to statistical inference may go a long way toward explaining the apparent ease with which statistical tests are misused and misinterpreted.<ref name=nickerson />}}
==Performing a frequentist hypothesis test in practice==
The typical steps involved in performing a frequentist hypothesis test in practice are:
# Define a hypothesis (claim which is testable using data).
# Select a relevant statistical test with associated [[test statistic]] <var>T</var>.
# Derive the distribution of the test statistic under the null hypothesis from the assumptions. In standard cases this will be a well-known result. For example, the test statistic might follow a [[Student's t distribution]] with known degrees of freedom, or a [[normal distribution]] with known mean and variance.
# Select a significance level (''α''), the maximum acceptable [[false positive rate]]. Common values are 5% and 1%.
# Compute from the observations the observed value <var>t</var><sub>obs</sub> of the test statistic <var>T</var>.
# Decide to either reject the null hypothesis in favor of the alternative or not reject it. The [[Neyman–Pearson lemma|Neyman-Pearson]] decision rule is to reject the null hypothesis <var>H</var><sub>0</sub> if the observed value <var>t</var><sub>obs</sub> is in the critical region, and not to reject the null hypothesis otherwise.<ref>{{Cite journal |date=2005 |title=Testing Statistical Hypotheses |url=https://link.springer.com/book/10.1007/0-387-27605-X |journal=Springer Texts in Statistics |language=en |doi=10.1007/0-387-27605-x |isbn=978-0-387-98864-1 |issn=1431-875X|url-access=subscription }}</ref>
=== Practical example ===
The difference in the two processes applied to the radioactive suitcase example (below):
* "The Geiger-counter reading is 10. The limit is 9. Check the suitcase."
* "The Geiger-counter reading is high; 97% of safe suitcases have lower readings. The limit is 95%. Check the suitcase."
The former report is adequate, the latter gives a more detailed explanation of the data and the reason why the suitcase is being checked.
Not rejecting the null hypothesis does not mean the null hypothesis is "accepted" per se (though Neyman and Pearson used that word in their original writings; see the [[
The processes described here are perfectly adequate for computation. They seriously neglect the [[design of experiments]] considerations.<ref>{{cite book|
It is particularly critical that appropriate sample sizes be estimated before conducting the experiment.
Line 45 ⟶ 105:
===Interpretation===
When the null hypothesis is true and statistical assumptions are met, the probability that the p-value will be less than or equal to the significance level <math>\alpha</math> is at most <math>\alpha</math>. This ensures that the hypothesis test maintains its specified false positive rate (provided that statistical assumptions are met).<ref name="LR" />
The ''p''-value is the probability that a test statistic which is at least as extreme as the one obtained would occur under the null hypothesis. At a significance level of 0.05, a fair coin would be expected to (incorrectly) reject the null hypothesis (that it is fair) in 1 out of 20 tests on average. The ''p''-value does not provide the probability that either the null hypothesis or its opposite is correct (a common source of confusion).<ref>{{Cite journal|last=Nuzzo|first=Regina|author-link= Regina Nuzzo |date=2014|title=Scientific method: Statistical errors|journal=Nature|volume=506|issue=7487|pages=150–152|bibcode=2014Natur.506..150N|doi=10.1038/506150a|pmid=24522584|doi-access=free|hdl=11573/685222|hdl-access=free}}</ref>
If the ''p''-value is less than the chosen significance threshold (equivalently, if the observed test statistic is in the critical region), then we say the null hypothesis is rejected at the chosen level of significance. If the ''p''-value is ''not'' less than the chosen significance threshold (equivalently, if the observed test statistic is outside the critical region), then the null hypothesis is not rejected at the chosen level of significance.
In the "lady tasting tea" example (below), Fisher required the lady to properly categorize all of the cups of tea to justify the conclusion that the result was unlikely to result from chance. His test revealed that if the lady was effectively guessing at random (the null hypothesis), there was a 1.4% chance that the observed results (perfectly ordered tea) would occur.
===Use and importance===
Line 94 ⟶ 141:
A statistical analysis of misleading data produces misleading conclusions. The issue of data quality can be more subtle. In [[forecasting]] for example, there is no agreement on a measure of forecast accuracy. In the absence of a consensus measurement, no decision based on measurements will be without controversy.
Publication bias: Statistically nonsignificant results may be less likely to be published, which can bias the literature.
Multiple testing: When multiple true null hypothesis tests are conducted at once without adjustment, the overall probability of Type I error is higher than the nominal alpha level.<ref>{{cite journal |last1=Ranganathan |first1=Priya |last2=Pramesh |first2=C. S |last3=Buyse |first3=Marc |title=Common pitfalls in statistical analysis: The perils of multiple testing |journal=Perspect Clin Res |date=April–June 2016 |volume=7 |issue=2 |pages=106–107 |doi=10.4103/2229-3485.179436 |pmid=27141478|pmc=4840791 |doi-access=free }}</ref>
Those making critical decisions based on the results of a hypothesis test are prudent to look at the details rather than the conclusion alone. In the physical sciences most results are fully accepted only when independently confirmed. The general advice concerning statistics is, "Figures never lie, but liars figure" (anonymous).
==Definition of terms==
{{See also|Notation in probability and statistics}}
The following definitions are mainly based on the exposition in the book by Lehmann and Romano:<ref name="LR">{{cite book|title=Testing Statistical Hypotheses|edition=3E|isbn=978-0-387-98864-1|last1=Lehmann|first1=E. L.|first2=Joseph P.|last2=Romano|year=2005|publisher=Springer|___location=New York}}</ref>
*'''Statistical hypothesis''': A statement about the parameters describing a [[Statistical population|population]] (not a [[Statistical sample|sample]]).
*Test statistic: A value calculated from a sample without any unknown parameters, often to summarize the sample for comparison purposes.
*{{visible anchor|Simple hypothesis}}: Any hypothesis which specifies the population distribution completely.
*Composite hypothesis: Any hypothesis which does ''not'' specify the population distribution completely.
*[[Null hypothesis]] (H<sub>0</sub>)
*Positive data: Data that enable the investigator to reject a null hypothesis.
*[[Alternative hypothesis]] (H<sub>1</sub>)
[[File:One tailed critical value with significance level alpha.jpg|thumb|260x260px|Suppose the data can be realized from an N(0,1) distribution. For example, with a chosen significance level α = 0.05, from the Z-table, a one-tailed critical value of approximately 1.645 can be obtained. The one-tailed critical value C<sub>α</sub> ≈ 1.645 corresponds to the chosen significance level. The critical region [C<sub>α</sub>, ∞) is realized as the tail of the standard normal distribution.]]
*'''{{vanchor|Critical value}}s''' of a statistical test are the boundaries of the acceptance region of the test.<ref>{{cite book |first1=Ann J. |last1=Hughes |first2=Dennis E. |last2=Grawoig |title=Statistics: A Foundation for Analysis |___location=Reading, Mass. |publisher=Addison-Wesley |year=1971 |isbn=0-201-03021-7 |page=[https://archive.org/details/trent_0116302260611/page/191 191] |url=https://archive.org/details/trent_0116302260611 |url-access=registration }}</ref> The acceptance region is the set of values of the test statistic for which the null hypothesis is not rejected. Depending on the shape of the acceptance region, there can be one or more than one critical value.
**'''{{vanchor|Region of rejection}}''' / '''{{vanchor|Critical region}}''': The set of values of the test statistic for which the null hypothesis is rejected.
*'''[[statistical power|Power of a test]] (1 − ''β'')'''
* [[Size (statistics)|'''Size''']]: For simple hypotheses, this is the test's probability of ''incorrectly'' rejecting the null hypothesis. The [[false positive]] rate. For composite hypotheses this is the supremum of the probability of rejecting the null hypothesis over all cases covered by the null hypothesis. The complement of the false positive rate is termed '''specificity''' in [[biostatistics]]. ("This is a specific test. Because the result is positive, we can confidently say that the patient has the condition.") See [[sensitivity and specificity]] and [[type I and type II errors]] for exhaustive definitions.
*[[Significance level]] of a test (''α)''
*'''[[p-value|''p''-value]]'''
*'''{{vanchor|Statistical significance test}}''': A predecessor to the statistical hypothesis test (see the Origins section). An experimental result was said to be [[statistical significance|statistically significant]] if a sample was sufficiently inconsistent with the (null) hypothesis. This was variously considered common sense, a pragmatic heuristic for identifying meaningful experimental results, a convention establishing a threshold of statistical evidence or a method for drawing conclusions from data. The statistical hypothesis test added mathematical rigor and philosophical consistency to the concept by making the alternative hypothesis explicit. The term is loosely used for the modern version which is now part of statistical hypothesis testing.
*Conservative test: A test is conservative if, when constructed for a given nominal significance level, the true probability of ''incorrectly'' rejecting the null hypothesis is never greater than the nominal level.
*[[Exact test]]
A statistical hypothesis test compares a test statistic (''z'' or ''t'' for examples) to a threshold. The test statistic (the formula found in the table below) is based on optimality. For a fixed level of Type I error rate, use of these statistics minimizes Type II error rates (equivalent to maximizing power). The following terms describe tests in terms of such optimality:
*Most powerful test: For a given ''size'' or ''significance level'', the test with the greatest power (probability of rejection) for a given value of the parameter(s) being tested, contained in the alternative hypothesis.
*[[Uniformly most powerful test]] (UMP)
==Nonparametric bootstrap hypothesis testing==
{{main|Bootstrapping (statistics)}}
Bootstrap-based [[Resampling (statistics)|resampling]] methods can be used for null hypothesis testing. A bootstrap creates numerous simulated samples by randomly resampling (with replacement) the original, combined sample data, assuming the null hypothesis is correct. The bootstrap is very versatile as it is distribution-free and it does not rely on restrictive parametric assumptions, but rather on empirical approximate methods with asymptotic guarantees. Traditional parametric hypothesis tests are more computationally efficient but make stronger structural assumptions. In situations where computing the probability of the test statistic under the null hypothesis is hard or impossible (due to perhaps inconvenience or lack of knowledge of the underlying distribution), the bootstrap offers a viable method for statistical inference.<ref>Hall, P. and Wilson, S.R., 1991. Two guidelines for bootstrap hypothesis testing. Biometrics, pp.757-762.
</ref><ref>Tibshirani, R.J. and Efron, B., 1993. An introduction to the bootstrap. Monographs on statistics and applied probability, 57(1).
</ref><ref>Martin, M.A., 2007. Bootstrap hypothesis testing for some common statistical problems: A critical evaluation of size and power properties. Computational Statistics & Data Analysis, 51(12), pp.6321-6342.
</ref><ref>Horowitz, J.L., 2019. Bootstrap methods in econometrics. Annual Review of Economics, 11, pp.193-224.
I'm</ref>
==Examples==
===Human sex ratio===
{{main|Human sex ratio}}
The earliest use of statistical hypothesis testing is generally credited to the question of whether male and female births are equally likely (null hypothesis), which was addressed in the 1700s by [[John Arbuthnot]] (1710),<ref>{{cite journal|author=John Arbuthnot
Arbuthnot examined birth records in London for each of the 82 years from 1629 to 1710, and applied the [[sign test]], a simple [[non-parametric test]].<ref name="Conover1999">{{Citation|last=Conover|first=W.J.|title=Practical Nonparametric Statistics|pages=157–176|year=1999|chapter=Chapter 3.4: The Sign Test|edition=Third|publisher=Wiley|isbn=978-0-471-16068-7}}</ref><ref name="Sprent1989">{{Citation|last=Sprent|first=P.|title=Applied Nonparametric Statistical Methods|year=1989|edition=Second|publisher=Chapman & Hall|isbn=978-0-412-44980-2}}</ref><ref>{{cite book|last=Stigler|first=Stephen M.|title=The History of Statistics: The Measurement of Uncertainty Before 1900|publisher=Harvard University Press|year=1986|isbn=978-0-67440341-3|pages=[https://archive.org/details/historyofstatist00stig/page/225 225–226]}}</ref> In every year, the number of males born in London exceeded the number of females. Considering more male or more female births as equally likely, the probability of the observed outcome is 0.5<sup>82</sup>, or about 1 in 4,836,000,000,000,000,000,000,000; in modern terms, this is the ''p''-value. Arbuthnot concluded that this is too small to be due to chance and must instead be due to divine providence: "From whence it follows, that it is Art, not Chance, that governs." In modern terms, he rejected the null hypothesis of equally likely male and female births at the ''p'' = 1/2<sup>82</sup> significance level.
Laplace considered the statistics of almost half a million births. The statistics showed an excess of boys compared to girls.<ref name="Laplace 1778">{{cite journal|
{{cite web|last=Laplace|first=P.|title=Mémoire sur les probabilités|translator-first=Richard J.|translator-last=Pulskam|date=August 21, 2010|url=http://cerebro.xu.edu/math/Sources/Laplace/memoir_probabilities.pdf
|archive-date=April 27, 2015|archive-url=https://web.archive.org/web/20150427142452/http://cerebro.xu.edu/math/Sources/Laplace/memoir_probabilities.pdf|url-status=dead}}</ref> He concluded by calculation of a ''p''-value that the excess was a real, but unexplained, effect.<ref>{{cite book|last=Stigler|first=Stephen M.|url=https://archive.org/details/historyofstatist00stig/page/134|title=The History of Statistics: The Measurement of Uncertainty before 1900|publisher=Belknap Press of Harvard University Press|year=1986|isbn=978-0-674-40340-6|___location=Cambridge, Mass|page=[https://archive.org/details/historyofstatist00stig/page/134 134]}}</ref>
===Lady tasting tea===
{{main|Lady tasting tea}}
In a famous example of hypothesis testing, known as the ''Lady tasting tea'',<ref name="fisher">{{cite book|last=Fisher|first=Sir Ronald A.|
===Courtroom trial===
Line 138 ⟶ 204:
In the start of the procedure, there are two hypotheses <math>H_0</math>: "the defendant is not guilty", and <math>H_1</math>: "the defendant is guilty". The first one, <math>H_0</math>, is called the ''[[null hypothesis]]''. The second one, <math>H_1</math>, is called the ''alternative hypothesis''. It is the alternative hypothesis that one hopes to support.
The hypothesis of innocence is rejected only when an error is very unlikely, because one
{|class="wikitable"
Line 155 ⟶ 221:
A criminal trial can be regarded as either or both of two decision processes: guilty vs not guilty or evidence vs a threshold ("beyond a reasonable doubt"). In one view, the defendant is judged; in the other view the performance of the prosecution (which bears the burden of proof) is judged. A hypothesis test can be regarded as either a judgment of a hypothesis or as a judgment of evidence.
===Clairvoyant card game===
A person (the subject) is tested for [[clairvoyance]]. They are shown the
As we try to find evidence of their clairvoyance, for the time being the null hypothesis is that the person is not clairvoyant.<ref>{{cite book|last1=Jaynes|first1=E. T.|title=Probability theory : the logic of science|date=2007|publisher=Cambridge Univ. Press
If the null hypothesis is valid, the only thing the test person can do is guess. For every card, the probability (relative frequency) of any single suit appearing is 1/4. If the alternative is valid, the test subject will predict the suit correctly with probability greater than 1/4. We will call the probability of guessing correctly ''p''. The hypotheses, then, are:
Line 185 ⟶ 234:
When the test subject correctly predicts all 25 cards, we will consider them clairvoyant, and reject the null hypothesis. Thus also with 24 or 23 hits. With only 5 or 6 hits, on the other hand, there is no cause to consider them so. But what about 12 hits, or 17 hits? What is the critical number, ''c'', of hits, at which point we consider the subject to be clairvoyant? How do we determine the critical value ''c''? With the choice ''c''=25 (i.e. we only accept clairvoyance when all cards are predicted correctly) we're more critical than with ''c''=10. In the first case almost no test subjects will be recognized to be clairvoyant, in the second case, a certain number will pass the test. In practice, one decides how critical one will be. That is, one decides how often one accepts an error of the first kind – a [[false positive]], or Type I error. With ''c'' = 25 the probability of such an error is:
:{{nowrap|<math>P(\text{reject }H_0 \mid H_0 \text{ is valid}) = P\left(X = 25\mid p=\
and hence, very small. The probability of a false positive is the probability of randomly guessing correctly all 25 times.
Being less critical, with ''c'' = 10, gives:
:{{nowrap|<math>P(\text{reject }H_0 \mid H_0 \text{ is valid}) = P\left(X \ge 10 \mid p=\
Thus, ''c'' = 10 yields a much greater probability of false positive.
Line 197 ⟶ 246:
Before the test is actually performed, the maximum acceptable probability of a Type I error (''α'') is determined. Typically, values in the range of 1% to 5% are selected. (If the maximum acceptable error rate is zero, an infinite number of correct guesses is required.) Depending on this Type 1 error rate, the critical value ''c'' is calculated. For example, if we select an error rate of 1%, ''c'' is calculated thus:
:{{nowrap|<math>P(\text{reject }H_0 \mid H_0 \text{ is valid}) = P\left(X \ge c\mid p=\
From all the numbers c, with this property, we choose the smallest, in order to minimize the probability of a Type II error, a [[false negative]]. For the above example, we select: <math>c=13</math>.
Line 203 ⟶ 252:
But what if the subject did not guess any cards at all? Having zero correct answers is clearly an oddity too. Without any clairvoyant skills the probability.
:<math>P(X=0 \mid H_0 \text{ is valid}) = P\left(X = 0\mid p=\
This is highly unlikely (less than 1 in a 1000 chance). While the subject can't guess the cards correctly, dismissing H<sub>0</sub> in favour of H<sub>1</sub> would be an error. In fact, the result would suggest a trait on the subject's part of avoiding calling the correct card. A test of this could be formulated: for a selected 1% error rate the subject would have to answer correctly at least twice, for us to believe that card calling is based purely on guessing. -->
== Variations and sub-classes ==
Statistical hypothesis testing is a key technique of both [[frequentist inference]] and [[Bayesian inference]], although the two types of inference have notable differences. Statistical hypothesis tests define a procedure that controls (fixes) the probability of incorrectly ''deciding'' that a default position ([[null hypothesis]]) is incorrect. The procedure is based on how likely it would be for a set of observations to occur if the null hypothesis were true.
One naïve [[Bayesian statistics|Bayesian]] approach to hypothesis testing is to base decisions on the [[posterior probability]],<ref>Schervish, M (1996) ''Theory of Statistics'', p. 218. Springer {{isbn|0-387-94546-6}}</ref><ref>{{cite book|title=Reference Manual on Scientific Evidence|publisher=West National Academies Press|chapter=Reference Guide on Statistics|first1=David H.|last1=Kaye|first2=David A.|last2=Freedman|chapter-url=http://www.nap.edu/openbook.php?record_id=13163&page=211|___location=Eagan, MN; Washington, D.C.|year=2011|edition=3rd|page=259|isbn=978-0-309-21421-6}}</ref> but this fails when comparing point and continuous hypotheses. Other approaches to decision making, such as [[Bayesian decision theory]], attempt to balance the consequences of incorrect decisions across all possibilities, rather than concentrating on a single null hypothesis. A number of other approaches to reaching a decision based on data are available via [[decision theory]] and [[optimal decision]]s, some of which have desirable properties. Hypothesis testing, though, is a dominant approach to data analysis in many fields of science. Extensions to the theory of hypothesis testing include the study of the [[statistical power|power]] of tests, i.e. the probability of correctly rejecting the null hypothesis given that it is false. Such considerations can be used for the purpose of [[sample size determination]] prior to the collection of data.
==
An example of Neyman–Pearson hypothesis testing (or null hypothesis statistical significance testing) can be made by a change to the radioactive suitcase example. If the "suitcase" is actually a shielded container for the transportation of radioactive material, then a test might be used to select among three hypotheses: no radioactive source present, one present, two (all) present. The test could be required for safety, with actions required in each case. The [[Neyman–Pearson lemma]] of hypothesis testing says that a good criterion for the selection of hypotheses is the ratio of their probabilities (a [[likelihood-ratio test|likelihood ratio]]). A simple method of solution is to select the hypothesis with the highest probability for the Geiger counts observed. The typical result matches intuition: few counts imply no source, many counts imply two sources and intermediate counts imply one source. Notice also that usually there are problems for [[Philosophic burden of proof#Proving a negative|proving a negative]]. Null hypotheses should be at least [[Falsifiability|falsifiable]].
Neyman–Pearson theory can accommodate both prior probabilities and the costs of actions resulting from decisions.<ref name="Ash">{{cite book | last = Ash | first = Robert | title = Basic probability theory | publisher = Wiley | ___location = New York | year = 1970 | isbn = 978-0471034506 }}Section 8.2</ref> The former allows each test to consider the results of earlier tests (unlike Fisher's significance tests). The latter allows the consideration of economic issues (for example) as well as probabilities. A likelihood ratio remains a good criterion for selecting among hypotheses.
Line 336 ⟶ 283:
==Criticism==
{{see also|p-value#
Much of the criticisms of statistical hypothesis testing can be summarized by the following issues:
* The interpretation of a ''p''-value is dependent upon [[stopping rule]] and definition of multiple comparison. The former often changes during the course of a study and the latter is unavoidably ambiguous. (i.e. "p values depend on both the (data) observed and on the other possible (data) that might have been observed but weren't").<ref>{{cite journal|last=Cornfield|first=Jerome|title=Recent Methodological Contributions to Clinical Trials| journal=American Journal of Epidemiology|volume=104|issue=4|pages=408–421|year=1976|url=http://www.epidemiology.ch/history/PDF%20bg/Cornfield%20J%201976%20recent%20methodological%20contributions.pdf|doi=10.1093/oxfordjournals.aje.a112313|pmid= 788503}}</ref>
* Confusion resulting (in part) from combining the methods of Fisher and Neyman–Pearson which are conceptually distinct.<ref name="Tukey60">{{cite journal|last=Tukey|first=John W.|title=Conclusions vs decisions|journal= Technometrics|volume=26|issue=4|pages=423–433|year=1960|doi=10.1080/00401706.1960.10489909}} "Until we go through the accounts of testing hypotheses, separating [Neyman–Pearson] decision elements from [Fisher] conclusion elements, the intimate mixture of disparate elements will be a continual source of confusion." ... "There is a place for both "doing one's best" and "saying only what is certain," but it is important to know, in each instance, both which one is being done, and which one ought to be done."</ref>
* Emphasis on statistical significance to the exclusion of estimation and confirmation by repeated experiments.<ref>{{cite journal|last=Yates|first=Frank|title=The Influence of Statistical Methods for Research Workers on the Development of the Science of Statistics|journal=Journal of the American Statistical Association|volume=46|issue=253|pages=19–34|year=1951|doi=10.1080/01621459.1951.10500764}} "The emphasis given to formal tests of significance throughout [R.A. Fisher's] Statistical Methods ... has caused scientific research workers to pay undue attention to the results of the tests of significance they perform on their data, particularly data derived from experiments, and too little to the estimates of the magnitude of the effects they are investigating." ... "The emphasis on tests of significance and the consideration of the results of each experiment in isolation, have had the unfortunate consequence that scientific workers have often regarded the execution of a test of significance on an experiment as the ultimate objective."</ref>
* Rigidly requiring statistical significance as a criterion for publication, resulting in [[publication bias]].<ref>{{cite journal|last1=Begg|first1=Colin B.|last2=Berlin|first2=Jesse A.|title=Publication bias: a problem in interpreting medical data|journal=Journal of the Royal Statistical Society, Series A|volume=151|issue=3|pages=419–463|year=1988|doi=10.2307/2982993|jstor=2982993|s2cid=121054702 }}</ref> Most of the criticism is indirect. Rather than being wrong, statistical hypothesis testing is misunderstood, overused and misused.
* When used to detect whether a difference exists between groups, a paradox arises. As improvements are made to experimental design (e.g. increased precision of measurement and sample size), the test becomes more lenient. Unless one accepts the absurd assumption that all sources of noise in the data cancel out completely, the chance of finding statistical significance in either direction approaches 100%.<ref>{{cite journal|last=Meehl|first=Paul E.|title= Theory-Testing in Psychology and Physics: A Methodological Paradox|journal=Philosophy of Science|volume=34|issue=2|pages=103–115|year=1967|url=http://mres.gmu.edu/pmwiki/uploads/Main/Meehl1967.pdf|doi=10.1086/288135|s2cid=96422880| url-status=dead|archive-url=https://web.archive.org/web/20131203010657/http://mres.gmu.edu/pmwiki/uploads/Main/Meehl1967.pdf|archive-date=December 3, 2013|df=mdy-all}} Thirty years later, Meehl acknowledged statistical significance theory to be mathematically sound while continuing to question the default choice of null hypothesis, blaming instead the "social scientists' poor understanding of the logical relation between theory and fact" in "The Problem Is Epistemology, Not Statistics: Replace Significance Tests by Confidence Intervals and Quantify Accuracy of Risky Numerical Predictions" (Chapter 14 in Harlow (1997)).</ref> However, this absurd assumption that the mean difference between two groups cannot be zero implies that the data cannot be independent and identically distributed (i.i.d.) because the expected difference between any two subgroups of i.i.d. random variates is zero; therefore, the i.i.d. assumption is also absurd.
*Layers of philosophical concerns. The probability of statistical significance is a function of decisions made by experimenters/analysts.<ref name="bakan66">
{{cite journal |last=Bakan |first=David |year=1966 |title=The test of significance in psychological research |journal=Psychological Bulletin |volume=66 |issue=6 |pages=423–437 |doi=10.1037/h0020412 |pmid=5974619}}</ref> If the decisions are based on convention they are termed arbitrary or mindless<ref name="Gigerenzer 587–606">{{cite journal|last=Gigerenzer|first=G|title=Mindless statistics|journal=The Journal of Socio-Economics|date=November 2004|volume=33|issue=5|pages=587–606|doi=10.1016/j.socec.2004.09.033}}</ref> while those not so based may be termed subjective. To minimize type II errors, large samples are recommended. In psychology practically all null hypotheses are claimed to be false for sufficiently large samples so "...it is usually nonsensical to perform an experiment with the ''sole'' aim of rejecting the null hypothesis."<ref>{{cite journal | last = Nunnally | first = Jum | title = The place of statistics in psychology | journal = Educational and Psychological Measurement | volume = 20 | number = 4 | pages = 641–650 | year = 1960 | doi=10.1177/001316446002000401| s2cid = 144813784}}</ref> "Statistically significant findings are often misleading" in psychology.<ref>{{cite journal | last = Lykken | first = David T. | title = What's wrong with psychology, anyway? | journal = Thinking Clearly About Psychology | volume = 1 | pages = 3–39 | year = 1991}}</ref> Statistical significance does not imply practical significance, and [[correlation does not imply causation]]. Casting doubt on the null hypothesis is thus far from directly supporting the research hypothesis. *"[I]t does not tell us what we want to know".<ref name=cohen94>{{cite journal|author=Jacob Cohen|title=The Earth Is Round (p < .05)|journal=American Psychologist|volume=49|issue=12|pages=997–1003|date=December 1994|doi=10.1037/0003-066X.49.12.997|s2cid=380942}} This paper lead to the review of statistical practices by the APA. Cohen was a member of the Task Force that did the review.</ref> Lists of dozens of complaints are available.<ref name=kline>{{cite book|last=Kline|first=Rex|title=Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research|publisher=American Psychological Association|___location=Washington, D.C. |year=2004|isbn=9781591471189 }}</ref><ref name="nickerson">{{cite journal|author=Nickerson, Raymond S.|title=Null Hypothesis Significance Tests: A Review of an Old and Continuing Controversy|journal=Psychological Methods|volume=5|issue=2|pages=241–301|year=2000|url=https://psycnet.apa.org/doiLanding?doi=10.1037%2F1082-989X.5.2.241|doi=10.1037/1082-989X.5.2.241|pmid=10937333|s2cid=28340967|archive-url=
Critics and supporters are largely in factual agreement regarding the characteristics of null hypothesis significance testing (NHST): While it can provide critical information, it is ''inadequate as the sole tool for statistical analysis''. ''Successfully rejecting the null hypothesis may offer no support for the research hypothesis.'' The continuing controversy concerns the selection of the best statistical practices for the near-term future given the existing practices. However, adequate research design can minimize this issue. Critics would prefer to ban NHST completely, forcing a complete departure from those practices,<ref>{{cite journal |last1=Hunter |first1=John E. |title=Needed: A Ban on the Significance Test |journal=Psychological Science |date=January 1997 |volume=8 |issue=1 |pages=3–7 |doi=10.1111/j.1467-9280.1997.tb00534.x|s2cid=145422959 }}</ref> while supporters suggest a less absolute change.{{citation needed|date=December 2015}}
Controversy over significance testing, and its effects on publication bias in particular, has produced several results. The [[American Psychological Association]] has strengthened its statistical reporting requirements after review,<ref name=wilkinson>{{cite journal|author=Wilkinson, Leland|title=Statistical Methods in Psychology Journals; Guidelines and Explanations|journal=American Psychologist|volume=54|issue=8|pages=594–604|year=1999|doi=10.1037/0003-066X.54.8.594|s2cid=428023 }} "Hypothesis tests. It is hard to imagine a situation in which a dichotomous accept-reject decision is better than reporting an actual p value or, better still, a confidence interval." (p 599). The committee used the cautionary term "forbearance" in describing its decision against a ban of hypothesis testing in psychology reporting. (p 603)</ref> [[medical journal]] publishers have recognized the obligation to publish some results that are not statistically significant to combat publication bias,<ref>{{cite web|url=http://www.icmje.org/publishing_1negative.html|title=ICMJE: Obligation to Publish Negative Studies|access-date=September 3, 2012|quote=Editors should seriously consider for publication any carefully done study of an important question, relevant to their readers, whether the results for the primary or any additional outcome are statistically significant. Failure to submit or publish findings because of lack of statistical significance is an important cause of publication bias.|url-status=dead|archive-url=https://web.archive.org/web/20120716211637/http://www.icmje.org/publishing_1negative.html|archive-date=July 16, 2012|df=mdy-all}}</ref> and a journal (''Journal of Articles in Support of the Null Hypothesis'') has been created to publish such results exclusively.<ref name=JASNH>''Journal of Articles in Support of the Null Hypothesis'' website: [http://www.jasnh.com/ JASNH homepage]. Volume 1 number 1 was published in 2002, and all articles are on psychology-related subjects.</ref> Textbooks have added some cautions,<ref>{{cite book|title=Statistical Methods for Psychology|last=Howell|first=David|year=2002|publisher=Duxbury|edition=5|isbn=978-0-534-37770-0|page=[https://archive.org/details/statisticalmetho0000howe/page/94 94]|url= https://archive.org/details/statisticalmetho0000howe/page/94}}</ref> and increased coverage of the tools necessary to estimate the size of the sample required to produce significant results.
==Alternatives==
Line 355 ⟶ 303:
{{See also|Confidence interval#Statistical hypothesis testing}}
A unifying position of critics is that statistics should not lead to an accept-reject conclusion or decision, but to an estimated value with an [[interval estimate]]; this data-analysis philosophy is broadly referred to as [[estimation statistics]]. Estimation statistics can be accomplished with either frequentist<ref>{{Cite
[[Bayesian inference]] is one proposed alternative to significance testing. (Nickerson cited 10 sources suggesting it, including Rozeboom (1960)).<ref name="nickerson"/> For example, Bayesian [[parameter estimation]] can provide rich information about the data from which researchers can draw inferences, while using uncertain [[Prior probability|priors]] that exert only minimal influence on the results when enough data is available. Psychologist [[John K. Kruschke]] has suggested Bayesian estimation as an alternative for the [[Student's t-test|''t''-test]]<ref name="Kruschke 2012" /> and has also contrasted Bayesian estimation for assessing null values with Bayesian model comparison for hypothesis testing.<ref name="Kruschke 2018" /> Two competing models/hypotheses can be compared using [[Bayes factors]].<ref>{{cite report |last=Kass |first=R. E. |title=Bayes factors and model uncertainty |year=1993|url=http://www.stat.washington.edu/research/reports/1993/tr254.pdf |publisher=Department of Statistics, University of Washington}}</ref> Bayesian methods could be criticized for requiring information that is seldom available in the cases where significance testing is most heavily used. Neither the prior probabilities nor the [[probability distribution]] of the test statistic under the alternative hypothesis are often available in the social sciences.<ref name="nickerson"/>
Advocates of a Bayesian approach sometimes claim that the goal of a researcher is most often to [[objectivity (science)|objectively]] assess the [[probability]] that a [[hypothesis]] is true based on the data they have collected.<ref>{{Cite journal | last = Rozeboom | first = William W
Line 372 ⟶ 318:
| title = The Case for Objective Bayesian Analysis
| journal = Bayesian Analysis | volume = 1 | issue = 3 | pages = 385–402 | year = 2006 | doi=10.1214/06-ba115| doi-access = free }} In listing the competing definitions of "objective" Bayesian analysis, "A major goal of statistics (indeed science) is to find a completely coherent objective Bayesian methodology for learning from data." The author expressed the view that this goal "is not attainable".</ref> Neither [[Ronald Fisher|Fisher]]'s significance testing, nor [[Neyman–Pearson lemma|Neyman–Pearson]] hypothesis testing can provide this information, and do not claim to. The probability a hypothesis is true can only be derived from use of [[Bayes' Theorem]], which was unsatisfactory to both the Fisher and Neyman–Pearson camps due to the explicit use of [[subjectivity]] in the form of the [[prior probability]].<ref name="Neyman 289–337"/><ref>{{cite journal|last=Aldrich|first=J|title=R. A. Fisher on Bayes and Bayes' theorem|journal=Bayesian Analysis|year=2008|volume=3|issue=1|pages=161–170|doi=10.1214/08-BA306|df=mdy-all|doi-access=free}}</ref> Fisher's strategy is to sidestep this with the [[p-value|''p''-value]] (an objective ''index'' based on the data alone) followed by ''inductive inference'', while Neyman–Pearson devised their approach of ''inductive behaviour''.
==See also==
Line 403 ⟶ 334:
* [[Look-elsewhere effect]]
* [[Modifiable areal unit problem]]
* [[Modifiable temporal unit problem]]
* [[Multivariate hypothesis testing]]
* [[Omnibus test]]
* [[Dichotomous thinking]]
*[[Almost sure hypothesis testing]]
*[[Akaike information criterion]]
*[[Bayesian information criterion]]
*[[E-values]]{{div col end}}
==References==
Line 418 ⟶ 352:
==External links==
{{Commons category|Hypothesis testing}}
* {{springer|title=Statistical hypotheses, verification of|id=p/s087400}}
* [http://www.cs.ucsd.edu/users/goguen/courses/275f00/stat.html Bayesian critique of classical hypothesis testing]
* [https://web.archive.org/web/20051124221846/http://www.npwrc.usgs.gov/resource/methods/statsig/stathyp.htm Critique of classical hypothesis testing highlighting long-standing qualms of statisticians]
* [https://web.archive.org/web/20091029162244/http://www.wiwi.uni-muenster.de/ioeb/en/organisation/pfaff/stat_overview_table.html Statistical Tests Overview:] How to choose the correct statistical test
* [https://arxiv.org/abs/1401.2851] Statistical Analysis based Hypothesis Testing Method in Biological Knowledge Discovery; Md. Naseef-Ur-Rahman Chowdhury, Suvankar Paul, Kazi Zakia Sultana
===Online calculators===
* Some [http://www.schramm.cc/link/Statistics-calculator.php p-value and hypothesis test calculators].
{{Statistics|inference||state=collapsed}}
{{Public health}}
{{Authority control}}
[[Category:Statistical hypothesis testing| ]]
[[Category:Design of experiments]]
[[Category:Logic and statistics]]
[[Category:Mathematical and quantitative methods (economics)]]
| |||