Content deleted Content added
→Local response normalization: divisive |
|||
Line 156:
It was a variant of the earlier '''local contrast normalization'''.<ref>{{Cite journal |last=Jarrett |first=Kevin |last2=Kavukcuoglu |first2=Koray |last3=Ranzato |first3=Marc' Aurelio |last4=LeCun |first4=Yann |date=2009-09 |title=What is the best multi-stage architecture for object recognition? |url=http://dx.doi.org/10.1109/iccv.2009.5459469 |journal=2009 IEEE 12th International Conference on Computer Vision |publisher=IEEE |doi=10.1109/iccv.2009.5459469}}</ref><math display="block">b_{x, y}^i=\frac{a_{x, y}^i}{\left(k+\alpha \sum_{j=\max (0, i-n / 2)}^{\min (N-1, i+n / 2)}\left(a_{x, y}^j - \bar a_{x, y}^j\right)^2\right)^\beta}</math>where <math>\bar a_{x, y}^j</math> is the average activation in a small window centered on ___location <math>(x,y)</math> and channel <math>i</math>. The numbers <math>k, n, \alpha, \beta</math>, and the size of the small window, are hyperparameters picked by using a validation set.
Similar methods were called '''divisive normalization''', as they divide activations by a number depending on the activations. They were originally inspired by biology, where it was used to explain nonlinear responses of cortical neurons and nonlinear masking in visual perception.<ref>{{Cite journal |last=Lyu |first=Siwei |last2=Simoncelli |first2=Eero P. |date=2008 |title=Nonlinear Image Representation Using Divisive Normalization |url=https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4207373/ |journal=Proceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition |volume=2008 |pages=1–8 |doi=10.1109/CVPR.2008.4587821 |issn=1063-6919 |pmc=4207373 |pmid=25346590}}</ref>
Both kinds of local normalization were obsoleted by batch normalization, which is a more global form of normalization.<ref>{{Cite journal |last=Ortiz |first=Anthony |last2=Robinson |first2=Caleb |last3=Morris |first3=Dan |last4=Fuentes |first4=Olac |last5=Kiekintveld |first5=Christopher |last6=Hassan |first6=Md Mahmudulla |last7=Jojic |first7=Nebojsa |date=2020 |title=Local Context Normalization: Revisiting Local Normalization |url=https://openaccess.thecvf.com/content_CVPR_2020/html/Ortiz_Local_Context_Normalization_Revisiting_Local_Normalization_CVPR_2020_paper.html |pages=11276–11285}}</ref>
| |||