Loss functions for classification

This sandbox is in the article namespace. Either move this page into your userspace, or remove the {{User sandbox}} template.

Plot of hinge loss (blue) vs. zero-one loss (misclassification, green: y < 0) for t = 1 and variable y. Note that the hinge loss penalizes predictions y < 1, corresponding to the notion of a margin in a support vector machine.

Loss function surrogates for classification are computationally feasible loss functions representing the price we will pay for inaccuracy in our predictions in classification problems. ^[1] Specifically, if ${\displaystyle g:X\mapsto }$ {-1,1} represents the mapping of a vector ${\displaystyle {\vec {x}}\in X}$ to a class label ${\displaystyle y\in }$ {-1,1}, we wish to find a function ${\displaystyle f:X\mapsto \mathbb {R} }$ which best approximates the true mapping ${\displaystyle g}$. (citation needed) Given that loss functions are always true functions of only one variable, it is standard practice to define loss functions for classification solely in terms of the product of the true classifier ${\displaystyle y}$ and the predicted value ${\displaystyle f({\vec {x}})}$. (citation needed) Selection of a loss function within this framework

${\displaystyle V(f({\vec {x}}),y)=\phi (yf({\vec {x}}))}$

impacts the optimal ${\displaystyle f^{*}}$ which minimizes empirical risk, as well as the computational complexity of the learning algorithm.

Given the binary nature of classification, a natural selection for a loss function (assuming equal cost for false positives and false negatives) would be the 0-1 indicator function which takes the value of 0 if the predicted classification equals that of the true class or a 1 if the predicted classification does not match the true class. Consequently, we could choose the loss function:

${\displaystyle V(f({\vec {x}}),y)=\mathbf {\theta } (-yf({\vec {x}}))}$

where ${\displaystyle \mathbf {\theta } }$ indicates the Heaviside step function. However, this loss function is non-convex and non-smooth, and solving for the optimal solution is an NP-hard combinatorial optimization problem. (cite utah) As a result, we seek continuous, convex loss function surrogates which are tractable for our learning algorithms. In addition to their computational tractability, the convexity of these functions allows us to provide probabilistic bounds on their estimation error from the true distribution. (cite uci) Some of these surrogates are described below.

Square Loss

While more commonly used in regression, the square loss function can be re-written as a function ${\displaystyle \phi (yf({\vec {x}}))}$  and utilized for classification. Defined as

${\displaystyle V(f({\vec {x}}),y)=(1-yf({\vec {x}}))^{2}}$ 

the square loss function is both convex and smooth and matches the 0-1 indicator function when ${\displaystyle yf({\vec {x}})=0}$  and when ${\displaystyle yf({\vec {x}})=1}$ .

Hinge Loss

Main article: Hinge loss

Hinge loss

${\displaystyle V(f({\vec {x}}),y)=\max(0,1-yf({\vec {x}}))=|1-yf({\vec {x}})|_{+}}$ 

Logistic Loss

${\displaystyle V(f({\vec {x}}),y)={\frac {1}{ln(2)}}ln(1+e^{-yf({\vec {x}})})}$ 

References

1. Attention: This template ({{cite doi}}) is deprecated. To cite the publication identified by doi:10.1162/089976604773135104, please use {{cite journal}} (if it was published in a bona fide academic journal, otherwise {{cite report}} with |doi=10.1162/089976604773135104 instead.