Discriminant function analysis: Difference between revisions

Revision as of 20:59, 9 June 2016

Latest revision as of 11:54, 12 October 2023

(11 intermediate revisions by 9 users not shown)

Line 1:

{{Merge#REDIRECT |[[Linear discriminant analysis|date=May 2016}}]]

{{Proposed deletion/dated

{{Redirect category shell|

|concern = methodology being discussed is [[Linear Discriminant Analysis]]; this page is a less rigorous discussion. best off merging select bits from here into LDA and deleting this page.

{{R from merge}}

|timestamp = 20160609205936

|help =

}}

'''Discriminant function analysis''' is a statistical analysis to predict a [[categorical variable|categorical]] [[dependent variable|dependent]] [[Variable (mathematics)#Applied statistics|variable]] (called a grouping variable) by one or more [[continuous variable|continuous]] or [[Binary variable|binary]] [[independent variable|independent]] variables (called predictor variables). The original dichotomous discriminant analysis was developed by Sir [[Ronald Fisher]] in 1936.<ref name="cohen">Cohen et al. Applied Multiple Regression/Correlation Analysis for the Behavioural Sciences 3rd ed. (2003). Taylor & Francis Group.</ref> It is different from an [[ANOVA]] or [[MANOVA]], which is used to predict one (ANOVA) or multiple (MANOVA) continuous dependent variables by one or more independent categorical variables. Discriminant function analysis is useful in determining whether a set of variables is effective in predicting category membership.<ref name="green">Green, S.B. Salkind, N. J. & Akey, T. M. (2008). Using SPSS for Windows and Macintosh: Analyzing and understanding data. New Jersey: Prentice Hall.</ref>

Discriminant analysis is used when groups are known a priori (unlike in [[cluster analysis]]). Each case must have a score on one or more quantitative predictor measures, and a score on a group measure.<ref name="buy">BÖKEOĞLU ÇOKLUK, Ö, & BÜYÜKÖZTÜRK, Ş. (2008). Discriminant function analysis: Concept and application. Eğitim araştırmaları dergisi, (33), 73-92.</ref> In simple terms, discriminant function analysis is classification - the act of distributing things into groups, classes or categories of the same type.

Moreover, it is a useful follow-up procedure to a MANOVA instead of doing a series of one-way ANOVAs, for ascertaining how the groups differ on the composite of dependent variables. In this case, a significant F test allows classification based on a linear combination of predictor variables. Terminology can get confusing here, as in MANOVA, the dependent variables are the predictor variables, and the independent variables are the grouping variables.<ref name="green"/>

==Assumptions==

The assumptions of discriminant analysis are the same as those for MANOVA. The analysis is quite sensitive to outliers and the size of the smallest group must be larger than the number of predictor variables.<ref name="buy"/>

*[[Multivariate normal distribution|Multivariate normality]]: Independent variables are normal for each level of the grouping variable.<ref name="green"/><ref name="buy"/>

*Homogeneity of variance/covariance ([[homoscedasticity]]): Variances among group variables are the same across levels of predictors. Can be tested with [[Box's M test|Box's M]] statistic.<ref name="green"/>{{page needed|date=April 2012}} It has been suggested, however, that [[linear discriminant analysis]] be used when covariances are equal, and that [[quadratic classifier#Quadratic discriminant analysis|quadratic discriminant analysis]] may be used when covariances are not equal.<ref name="buy"/>

*[[Multicollinearity]]: Predictive power can decrease with an increased correlation between predictor variables.<ref name="buy"/>

*[[statistical independence|Independence]]: Participants are assumed to be randomly sampled, and a participant’s score on one variable is assumed to be independent of scores on that variable for all other participants.<ref name="green"/><ref name="buy"/>

It has been suggested that discriminant analysis is relatively robust to slight violations of these assumptions,<ref>Lachenbruch, P. A. (1975). ''Discriminant analysis''. NY: Hafner</ref> and it has also been shown that discriminant analysis may still be reliable when using dichotomous variables (where multivariate normality is often violated).<ref>Klecka, William R. (1980). ''Discriminant analysis''. Quantitative Applications in the Social Sciences Series, No. 19. Thousand Oaks, CA: Sage Publications.</ref>

==Discriminant functions==

Discriminant analysis works by creating one or more linear combinations of predictors, creating a new [[latent variable]] for each function. These functions are called discriminant functions. The number of functions possible is either ''Ng''-1 where ''Ng'' = number of groups, or ''p'' (the number of predictors), whichever is smaller. The first function created maximizes the differences between groups on that function. The second function maximizes differences on that function, but also must not be correlated with the previous function. This continues with subsequent functions with the requirement that the new function not be correlated with any of the previous functions.

Given group <math>j</math>, with <math>\mathbb{R}_j</math> sets of sample space, there is a discriminant rule such that if <math>x \in\mathbb{R}_j</math>, then <math>x\in j</math>. Discriminant analysis then, finds “good” regions of <math>\mathbb{R}_j</math> to minimize classification error, therefore leading to a high percent correct classified in the classification table.Hardle, W., Simar, L. (2007). ''Applied Multivariate Statistical Analysis''. Springer Berlin Heidelberg. pp.&nbsp;289–303.

Each function is given a discriminant score to determine how well it predicts group placement.

*Structure Correlation Coefficients: The correlation between each predictor and the discriminant score of each function. This is a whole{{clarify|date=April 2012}} correlation.Garson, G. D. (2008). Discriminant function analysis. http://www2.chass.ncsu.edu/garson/pa765/discrim.htm .

*Standardized Coefficients: Each predictor’s unique contribution to each function, therefore this is a [[partial correlation]]. Indicates the relative importance of each predictor in predicting group assignment from each function.

*Functions at Group Centroids: Mean discriminant scores for each grouping variable are given for each function. The farther apart the means are, the less error there will be in classification.

==Discrimination rules==

*[[Maximum likelihood]]: Assigns x to the group that maximizes population (group) density.<ref name="har">Hardle, W., Simar, L. (2007). ''Applied Multivariate Statistical Analysis''. Springer Berlin Heidelberg. pp. 289-303.</ref>

*Bayes Discriminant Rule: Assigns x to the group that maximizes <math>\pi_i f_i(x)</math>, where <math>f_i(x)</math> represents the [[prior probability]] of that classification, and ''π_i'' represents the population density.<ref name="har"/>

*[[Linear Discriminant Analysis|Fisher’s linear discriminant rule]]: Maximizes the ratio between ''SS''_between and ''SS''_within, and finds a linear combination of the predictors to predict group.<ref name="har"/>

==Eigenvalues==

An [[eigenvalues and eigenvectors|eigenvalue]] in discriminant analysis is the characteristic root of each function.{{clarify|date=April 2012}} It is an indication of how well that function differentiates the groups, where the larger the eigenvalue, the better the function differentiates.<ref name="buy"/> This however, should be interpreted with caution, as eigenvalues have no upper limit.<ref name="green"/><ref name="buy"/>

The eigenvalue can be viewed as a ratio of ''SS''_between and ''SS''_within as in ANOVA when the dependent variable is the discriminant function, and the groups are the levels of the [[Instrumental variable|IV]]{{clarify|date=April 2012}}.<ref name="green"/> This means that the largest eigenvalue is associated with the first function, the second largest with the second, etc..

==Effect size==

Some suggest the use of eigenvalues as [[effect size]] measures, however, this is generally not supported.<ref name="green"/> Instead, the [[canonical correlation]] is the preferred measure of effect size. It is similar to the eigenvalue, but is the square root of the ratio of ''SS''_between and ''SS''_total. It is the correlation between groups and the function.<ref name="green"/>

Another popular measure of effect size is the percent of variance{{clarify|date=April 2012}} for each function. This is calculated by: (''λ_x/Σλ_i'') X 100 where ''λ_x'' is the eigenvalue for the function and Σ''λ_i'' is the sum of all eigenvalues. This tells us how strong the prediction is for that particular function compared to the others.<ref name="green"/>

Percent correctly classified can also be analyzed as an effect size. The kappa value{{clarify|date=April 2012}} can describe this while correcting for chance agreement.<ref name="green"/>

==Variations==

*[[Linear Discriminant Analysis#Multiclass LDA|Multiple discriminant analysis (MDA)]]: related to MANOVA. Has more than two groups, and uses multiple dummy variables.<ref name="garson">Garson, G. D. (2008). Discriminant function analysis. http://www2.chass.ncsu.edu/garson/pa765/discrim.htm .</ref>

*Sequential discriminant analysis: assesses the importance of a set of IVs over and above a set of controls. In this case, the controls are entered first, and then the IVs.<ref name="garson"/>

*Stepwise discriminant analysis: Selects the most correlated predictor first, removes that variance in the grouping variable then adds the next most correlated and continues until the change in canonical correlation is not significant. Of course, both forward and backward stepwise procedures may be performed.<ref name="garson"/>

==Comparison to logistic regression==

Discriminant function analysis is very similar to [[logistic regression]], and both can be used to answer the same research questions.<ref name="green"/> Logistic regression does not have as many assumptions and restrictions as discriminant analysis. However, when discriminant analysis’ assumptions are met, it is more powerful than logistic regression.{{citation needed|date=April 2012}} Unlike logistic regression, discriminant analysis can be used with small sample sizes. It has been shown that when sample sizes are equal, and homogeneity of variance/covariance holds, discriminant analysis is more accurate.<ref name="buy"/> With all this being considered, logistic regression has become the common choice, since the assumptions of discriminant analysis are rarely met.<ref name="cohen"/><ref name="buy"/>

==See also==

*[[Statistical classification]]

*[[Linear discriminant analysis]].

*[[Multiple discriminant analysis]]

==References==

{{Reflist}}

==External links==

{{wikiversity}}

* [http://www2.chass.ncsu.edu/garson/pa765/discrim.htm Course notes, Discriminant function analysis by G. David Garson, NC State University]

* [http://people.revoledu.com/kardi/tutorial/LDA/ Discriminant analysis tutorial in Microsoft Excel by Kardi Teknomo]

* [http://www.psychstat.missouristate.edu/multibook/mlt03m.html Course notes, Discriminant function analysis by David W. Stockburger, Missouri State University]

* [http://userwww.sfsu.edu/~efc/classes/biol710/discrim/discrim.pdf Discriminant function analysis (DA) by John Poulsen and Aaron French, San Francisco State University]

[[Category:Multivariate statistics]]

[[Category:Statistical classification]]