Loss functions for classification

This sandbox is in the article namespace. Either move this page into your userspace, or remove the {{User sandbox}} template.

Plot of hinge loss (blue) vs. zero-one loss (misclassification, green: y < 0) for t = 1 and variable y. Note that the hinge loss penalizes predictions y < 1, corresponding to the notion of a margin in a support vector machine.

Loss function surrogates for classification are computationally feasible loss functions representing the price we will pay for inaccuracy in our predictions in classification problems. ^[1] Specifically, if ${\displaystyle g:X\mapsto }$ {-1,1} represents the mapping of a vector ${\displaystyle {\vec {x}}\in X}$ to a class label ${\displaystyle y\in }$ {-1,1}, we wish to find a function ${\displaystyle f:X\mapsto \mathbb {R} }$ which best approximates the true mapping ${\displaystyle g}$. (citation needed) Given that loss functions are always true functions of only one variable, it is standard practice to define loss functions for classification solely in terms of the product of the true classifier ${\displaystyle y}$ and the predicted value ${\displaystyle f({\vec {x}})}$. (citation needed) Selection of a loss function within this framework

${\displaystyle V(f({\vec {x}}),y)=\phi (yf({\vec {x}}))}$

impacts the optimal ${\displaystyle f^{*}}$ which minimizes empirical risk, as well as the computational complexity of the learning algorithm.

Given the binary nature of classification, a natural selection for a loss function (assuming equal cost for false positives and false negatives) would be the 0-1 indicator function which takes the value of 0 if the predicted classification equals that of the true class or a 1 if the predicted classification does not match the true class. Consequently, we could choose the loss function:

${\displaystyle V(f({\vec {x}}),y)=\mathbf {\theta } (-yf({\vec {x}}))}$

where ${\displaystyle \mathbf {\theta } }$ indicates the Heaviside step function. However, this loss function is non-convex and non-smooth, and solving for the optimal solution is an NP-hard combinatorial optimization problem. (cite utah) As a result, we seek continuous, convex loss function surrogates which are tractable for our learning algorithms. In addition to their computational tractability, the convexity of these functions allows us to provide probabilistic bounds on their estimation error from the true distribution. (cite uci) Some of these surrogates are described below.

Square Loss

While more commonly used in regression, the square loss function can be re-written as a function ${\displaystyle \phi (yf({\vec {x}}))}$  and utilized for classification. Defined as

${\displaystyle V(f({\vec {x}}),y)=(1-yf({\vec {x}}))^{2}}$ 

the square loss function is both convex and smooth and matches the 0-1 indicator function when ${\displaystyle yf({\vec {x}})=0}$  and when ${\displaystyle yf({\vec {x}})=1}$ . However, the square loss function tends to penalize outliers excessively leading to slower convergence rates than for the logistic loss or hinge loss functions. In addition, functions which yield high values of ${\displaystyle f({\vec {x}})}$  for some ${\displaystyle x\in X}$  will perform poorly with the square loss function, since high values of ${\displaystyle yf({\vec {x}})}$  will be penalized severely, regardless of whether the signs of ${\displaystyle yf({\vec {x}})}$  match.

Hinge Loss

Main article: Hinge loss

The hinge loss function is defined as

${\displaystyle V(f({\vec {x}}),y)=\max(0,1-yf({\vec {x}}))=|1-yf({\vec {x}})|_{+}}$ 

The hinge loss provides a relatively tight, convex upper bound on the 0-1 indicator function. Specifically, the hinge loss equals the 0-1 indicator function when ${\displaystyle sgn(f({\vec {x}}))=y}$  and ${\displaystyle yf({\vec {x}})>=1}$ . In addition, the structure provides a "maximum-margin" classification for support vector machines (SVMs).

While the hinge loss function is both convex and continuous, it is not smooth (that is not differentiable) at ${\displaystyle yf({\vec {x}})=1}$ . Consequently, the hinge loss function cannot be used with gradient descent methods or stochastic gradient descent methods which rely on differentiability over the entire ___domain. However, the hinge loss does have a subgradient at ${\displaystyle yf({\vec {x}})=1}$ , which allows for the utilization of subgradient descent methods. (cite Utah)

Logistic Loss

The logistic loss function is defined as

${\displaystyle V(f({\vec {x}}),y)={\frac {1}{ln(2)}}ln(1+e^{-yf({\vec {x}})})}$ 

This function displays a similar convergence rate to the

References

1. Attention: This template ({{cite doi}}) is deprecated. To cite the publication identified by doi:10.1162/089976604773135104, please use {{cite journal}} (if it was published in a bona fide academic journal, otherwise {{cite report}} with |doi=10.1162/089976604773135104 instead.