Content deleted Content added
m math formatting |
m →Probability calibration: HTTP to HTTPS for Cornell University |
||
| (44 intermediate revisions by 33 users not shown) | |||
Line 1:
{{Short description|Machine learning problem}}
{{machine learning bar}}
In [[machine learning]], a '''probabilistic classifier''' is a [[statistical classification|classifier]] that is able to predict, given
==Types of classification==
Formally, an "ordinary" classifier is some rule, or [[function (mathematics)|function]], that assigns to a sample {{mvar|x}} a class label {{mvar|ŷ}}:
:<math>\hat{y} = f(x)</math>
The samples come from some set {{mvar|X}} (e.g., the set of all [[document classification|documents]], or the set of all [[Computer vision#
Probabilistic classifiers generalize this notion of classifiers: instead of functions, they are [[conditional probability|conditional]] distributions <math>\Pr(Y \vert X)</math>, meaning that for a given <math>x \in X</math>, they assign probabilities to all <math>y \in Y</math> (and these probabilities sum to one). "Hard" classification can then be done using the [[Bayes estimator|optimal decision rule]]<ref name="bishop">{{cite book |first=Christopher M. |last=Bishop |year=2006 |title=Pattern Recognition and Machine Learning |publisher=Springer}}</ref>{{rp|39–40}}
Line 14 ⟶ 16:
or, in English, the predicted class is that which has the highest probability.
Binary probabilistic classifiers are also called [[
Some classification models, such as [[naive Bayes classifier|naive Bayes]], [[logistic regression]] and [[multilayer perceptron]]s (when trained under an appropriate [[loss function]]) are naturally probabilistic. Other models such as [[support vector machine]]s are not, but [[#Probability calibration|methods exist]] to turn them into probabilistic classifiers.
==Generative and conditional training==
Some models, such as [[logistic regression]], are conditionally trained: they optimize the conditional probability <math>\Pr(Y \vert X)</math> directly on a training set (see [[empirical risk minimization]]). Other classifiers, such as [[naive Bayes]], are trained [[Generative model|generatively]]: at training time, the class-conditional distribution <math>\Pr(X \vert Y)</math> and the class [[Prior probability|prior]] <math>\Pr(Y)</math> are found, and the conditional distribution <math>\Pr (Y \vert X)</math> is derived using [[Bayes' theorem|Bayes' rule]].<ref name="bishop"/>{{rp|43}}
==Probability calibration==
{{Main article|Calibration (statistics)}}
Not all classification models are naturally probabilistic, and some that are, notably naive Bayes classifiers, [[decision tree learning|decision trees]] and [[Boosting (machine learning)|boosting]] methods, produce distorted class probability distributions.<ref name="Niculescu">{{cite
For the [[binary classification|binary]] case, a common approach is to apply [[Platt scaling]], which learns a [[logistic regression]] model on the scores.<ref name="platt99">{{cite journal |last=Platt |first=John |
An alternative method using [[isotonic regression]]<ref>{{
In the [[multiclass classification|multiclass]] case, one can use a reduction to binary tasks, followed by univariate calibration with an algorithm as described above and further application of the pairwise coupling algorithm by Hastie and Tibshirani.<ref>{{
==Evaluating probabilistic classification==
Commonly used
Calibration errors metrics aim to quantify the extent to which a probabilistic classifier's outputs are ''well-calibrated''. As [[Philip Dawid]] put it, "a forecaster is well-calibrated if, for example, of those events to which he assigns a probability 30 percent, the long-run proportion that actually occurs turns out to be 30 percent".<ref>{{cite journal |doi=10.1080/01621459.1982.10477856 |title=The Well-Calibrated Bayesian |journal=Journal of the American Statistical Association |volume=77 |issue=379 |pages=605–610 |year=1982 |last1=Dawid |first1=A. P}}</ref> Foundational work in the ___domain of measuring calibration error is the Expected Calibration Error (ECE) metric.<ref>{{cite book | first1 = M.P. | last1= Naeini | first2 = G. | last2 = Cooper| first3 = M. | last3 = Hauskrecht| chapter = Obtaining well calibrated probabilities using bayesian binning |title = Proceedings of the AAAI Conference on Artificial Intelligence | year = 2015 | chapter-url = https://www.dbmi.pitt.edu/wp-content/uploads/2022/10/Obtaining-well-calibrated-probabilities-using-Bayesian-binning.pdf}}</ref> More recent works propose variants to ECE that address limitations of the ECE metric that may arise when classifier scores concentrate on narrow subset of the [0,1], including the Adaptive Calibration Error (ACE) <ref>{{cite book | first1 = J. | last1= Nixon | first2 = M.W. | last2 = Dusenberry| first3 = L. | last3 = Zhang| first4= G. | last4 = Jerfel | first5 = D. | last5 = Tran | chapter = Measuring Calibration in Deep Learning |title = CVPR workshops | year = 2019 | chapter-url = https://openaccess.thecvf.com/content_CVPRW_2019/papers/Uncertainty%20and%20Robustness%20in%20Deep%20Visual%20Learning/Nixon_Measuring_Calibration_in_Deep_Learning_CVPRW_2019_paper.pdf}}</ref> and Test-based Calibration Error (TCE).<ref>{{cite book | first1 = T. | last1= Matsubara | first2 = N. | last2 = Tax| first3 = R. | last3 = Mudd| first4= I. | last4 = Guy | chapter = TCE: A Test-Based Approach to Measuring Calibration Error |title = Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence (UAI) | year = 2023 | arxiv= 2306.14343 }}</ref>
A method used to assign scores to pairs of predicted probabilities and actual discrete outcomes, so that different predictive methods can be compared, is called a [[scoring rule]].
==Software Implementations==
* MoRPE<ref>{{cite web |title=MoRPE |url=https://github.com/adaviding/morpe |website=GitHub |access-date=17 February 2023}}</ref> is a trainable probabilistic classifier that uses [[isotonic regression]] for probability calibration. It solves the [[multiclass classification|multiclass]] case by reduction to binary tasks. It is a type of kernel machine that uses an inhomogeneous polynomial kernel.
==References==
{{reflist|30em}}
[[Category:Probabilistic models]]
| |||