Statistical model validation: Difference between revisions

Revision as of 17:48, 25 January 2023

Revision as of 00:44, 2 March 2023

Line 21:

==Methods for validating==

When doing a validation, there are three notable causes of potential difficulty, according to the ''[[Encyclopedia of Statistical Sciences]]''.<ref name="ESS06">{{citation| first= M. L. | last= Deaton | title= Simulation models, validation of | encyclopedia= [[Encyclopedia of Statistical Sciences]] | editor1-first= S. | editor1-last= Kotz | editor1-link= Samuel Kotz |display-editors=etal | year= 2006 | publisher= [[Wiley (publisher)|Wiley]]}}.</ref> The three causes are these: lack of data; lack of control of the input variables; uncertainty about the underlying probability distributions and correlations. The usual methods for dealing with difficulties in validation include the following: checking the assumptions made in constructing the model; examining the available data and related model outputs; applying expert judgment.<ref name="NRC12" /> Note that expert judgment commonly requires expertise in the application area.<ref name="NRC12">{{citation | chapter= Chapter 5: Model validation and prediction | chapter-url= https://www.nap.edu/read/13395/chapter/7 | author= [[National Academies of Sciences, Engineering, and Medicine|National Research Council]] | year= 2012 | title= Assessing the Reliability of Complex Models: Mathematical and statistical foundations of verification, validation, and uncertainty quantification | ___location= Washington, DC | publisher= [[National Academies Press]] | pages= 52–85 | doi= 10.17226/13395 | isbn= 978-0-309-25634-6 }}. </ref>

Expert judgment can sometimes be used to assess the validity of a prediction ''without'' obtaining real data: e.g. for the curve in Figure&nbsp;1, an expert might well be able to assess that a substantial extrapolation will be invalid. Additionally, expert judgment can be used in [[Turing test|Turing]]-type tests, where experts are presented with both real data and related model outputs and then asked to distinguish between the two.<ref name= "MB93">{{citation | author1-first= D. G. | author1-last=Mayer | author2-first= D.G. | author2-last= Butler | title= Statistical validation | journal= [[Ecological Modelling]] | year= 1993 | volume= 68 | issue=1–2 | pages= 21–32 | doi= 10.1016/0304-3800(93)90105-2}}.</ref>

For some classes of statistical models, specialized methods of performing validation are available. As an example, if the statistical model was obtained via a [[regression analysis|regression]], then specialized analyses for [[regression model validation]] exist and are generally employed.

Line 59:

==Further reading==

*{{citation | author-first= Y. | author-last= Barlas | title= Formal aspects of model validity and validation in system dynamics | journal= System Dynamics Review | year= 1996 | doi= 10.1002/(SICI)1099-1727(199623)12:3<183::AID-SDR103>3.0.CO;2-4 | volume= 12 | issue= 3 | pages= 183–210 }}

*{{citation | author1-first= P. I. | author1-last= Good | author1-link= Phillip Good | author2-first= J. W. | author2-last= Hardin | title= Common Errors in Statistics | chapter= Chapter 15: Validation | publisher= [[John Wiley & Sons]] | pages= 277–285 | year= 2012 | edition= Fourth }}

*{{citation | author-first= P. J. | author-last= Huber | author-link= Peter J. Huber | chapter= Chapter 3: Approximate models | pages= 25–41 | editor1-first= C. | editor1-last= Huber-Carol | editor2-first= N. | editor2-last=Balakrishnan | editor3-first= M. S. | editor3-last= Nikulin | editor4-first= M. | editor4-last= Mesbah | title= Goodness-of-Fit Tests and Model Validity | publisher= [[Springer Science+Business Media|Springer]] | year= 2002}}