Normalization (machine learning): Difference between revisions

Revision as of 20:29, 9 September 2024

Revision as of 17:07, 10 September 2024

Line 155:

The numbers <math>k, n, \alpha, \beta</math> are hyperparameters picked by using a validation set.

It was a variant of the earlier '''local contrast normalization'''.<ref>{{Cite journal |last=Jarrett |first=Kevin |last2=Kavukcuoglu |first2=Koray |last3=Ranzato |first3=Marc' Aurelio |last4=LeCun |first4=Yann |date=September 2009-09 |title=What is the best multi-stage architecture for object recognition? |url=http://dx.doi.org/10.1109/iccv.2009.5459469 |journal=2009 IEEE 12th International Conference on Computer Vision |publisher=IEEE |doi=10.1109/iccv.2009.5459469}}</ref><math display="block">b_{x, y}^i=\frac{a_{x, y}^i}{\left(k+\alpha \sum_{j=\max (0, i-n / 2)}^{\min (N-1, i+n / 2)}\left(a_{x, y}^j - \bar a_{x, y}^j\right)^2\right)^\beta}</math>where <math>\bar a_{x, y}^j</math> is the average activation in a small window centered on ___location <math>(x,y)</math> and channel <math>i</math>. The numbers <math>k, n, \alpha, \beta</math>, and the size of the small window, are hyperparameters picked by using a validation set.

Similar methods were called '''divisive normalization''', as they divide activations by a number depending on the activations. They were originally inspired by biology, where it was used to explain nonlinear responses of cortical neurons and nonlinear masking in visual perception.<ref>{{Cite journal |last=Lyu |first=Siwei |last2=Simoncelli |first2=Eero P. |date=2008 |title=Nonlinear Image Representation Using Divisive Normalization |url=https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4207373/ |journal=Proceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition |volume=2008 |pages=1–8 |doi=10.1109/CVPR.2008.4587821 |issn=1063-6919 |pmc=4207373 |pmid=25346590}}</ref>