Loss functions for classification

This sandbox is in the article namespace. Either move this page into your userspace, or remove the {{User sandbox}} template.

Plot of hinge loss (blue) vs. zero-one loss (misclassification, green: y < 0) for t = 1 and variable y. Note that the hinge loss penalizes predictions y < 1, corresponding to the notion of a margin in a support vector machine.

Loss function surrogates for classification are computationally feasible loss functions representing the price we will pay for inaccuracy in our predictions in classification problems. ^[1] Specifically, if ${\displaystyle g:X\mapsto }$ {-1,1} represents the mapping of a vector ${\displaystyle {\vec {x}}\in X}$ to a class label ${\displaystyle y\in }$ {-1,1}, we wish to find a function ${\displaystyle f:X\mapsto \mathbb {R} }$ which best approximates the true mapping ${\displaystyle g}$. (citation needed) Given that loss functions are always true functions of only one variable, it is standard practice to define loss functions for classification solely in terms of the product of the true classifier ${\displaystyle y}$ and the predicted value ${\displaystyle f({\vec {x}})}$. (citation needed) Selection of a loss function in this manner

${\displaystyle V(f({\vec {x}}),y)=\phi (yf({\vec {x}}))}$

impacts the optimal ${\displaystyle f^{*}}$ which minimizes empirical risk, as well as the computational complexity of the learning algorithm.

Given the binary nature of classification, a natural selection for a loss function (assuming equal cost for false positives and false negatives) would be the 0-1 indicator function which takes the value of 0 if the predicted classification equals that of the true class or a 1 if the predicted classification does not match the true class. Consequently, we could choose the loss function:

${\displaystyle V(f({\vec {x}}),y)=\mathbf {\theta } (-yf({\vec {x}}))}$

where ${\displaystyle \mathbf {\theta } }$ indicates the Heaviside step function. However, this loss function is non-convex and non-smooth, and solving for the optimal solution is an NP-hard combinatorial optimization problem. (cite utah) As a result, we seek continuous, convex loss function surrogates which are tractable for our learning algorithms. In addition to their computational tractability, the convexity of these functions allows us to provide probabilistic bounds on their estimation error from the true distribution (cite uci).Some of these surrogates are described below.

Square Loss

Hinge Loss

Main article: Hinge loss

Hinge loss

${\displaystyle V(f({\vec {x}}),y)=\max(0,1-yf({\vec {x}}))=|1-yf({\vec {x}})|_{+}}$ 

Logistic Loss

References

1. Attention: This template ({{cite doi}}) is deprecated. To cite the publication identified by doi:10.1162/089976604773135104, please use {{cite journal}} (if it was published in a bona fide academic journal, otherwise {{cite report}} with |doi=10.1162/089976604773135104 instead.