Content deleted Content added
Citation bot (talk | contribs) Add: pmc, pmid. | Use this bot. Report bugs. | Suggested by Лисан аль-Гаиб | #UCB_webform 14/454 |
Citation bot (talk | contribs) Removed URL that duplicated identifier. Removed access-date with no URL. Removed parameters. | Use this bot. Report bugs. | Suggested by Abductive | Category:Orphaned articles from June 2025 | #UCB_Category 462/875 |
||
| (16 intermediate revisions by 10 users not shown) | |||
Line 1:
{{Short description|
{{Multiple issues|
{{technical|date=June 2025}}▼
{{AI-generated|date=June 2025}}
{{Orphan|date=June 2025}}
}}
'''Sharpness Aware Minimization''' ('''SAM''') is an [[optimization algorithm]] used in [[machine learning]] that aims to improve model [[generalization (machine learning)|generalization]]. The method seeks to find model parameters that are located in regions of the loss landscape with uniformly low loss values, rather than parameters that only achieve a minimal loss value at a single point. This approach is described as finding "flat" minima instead of "sharp" ones. The rationale is that models trained this way are less sensitive to variations between training and test [[data set|data]], which can lead to better performance on unseen data.<ref name="Foret2021">{{cite conference |last1=Foret |first1=Pierre |last2=Kleiner |first2=Ariel |last3=Mobahi |first3=Hossein |last4=Neyshabur |first4=Behnam |year=2021 |title=Sharpness-Aware Minimization for Efficiently Improving Generalization |url=https://openreview.net/forum?id=6Tm1mposlrM |conference= |arxiv=2010.01412 |book-title=International Conference on Learning Representations (ICLR) 2021
▲{{technical}}
▲'''Sharpness Aware Minimization''' ('''SAM''') is an [[optimization algorithm]] used in [[machine learning]] that aims to improve model [[generalization (machine learning)|generalization]]. The method seeks to find model parameters that are located in regions of the loss landscape with uniformly low loss values, rather than parameters that only achieve a minimal loss value at a single point. This approach is described as finding "flat" minima instead of "sharp" ones. The rationale is that models trained this way are less sensitive to variations between training and test [[data set|data]], which can lead to better performance on unseen data.<ref name="Foret2021">{{cite conference |last1=Foret |first1=Pierre |last2=Kleiner |first2=Ariel |last3=Mobahi |first3=Hossein |last4=Neyshabur |first4=Behnam |title=Sharpness-Aware Minimization for Efficiently Improving Generalization |book-title=International Conference on Learning Representations (ICLR) 2021 |year=2021 |arxiv=2010.01412 |url=https://openreview.net/forum?id=6Tm1m_rRrwY}}</ref>
The algorithm was introduced in a 2020 paper by a team of researchers including Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.<ref name="Foret2021"/>
Line 36 ⟶ 31:
SAM has been applied in various machine learning contexts, primarily in [[computer vision]]. Research has shown it can improve generalization performance in models such as [[Convolutional Neural Network|Convolutional Neural Networks (CNNs)]] and [[Transformer (machine learning model)|Vision Transformers (ViTs)]] on image datasets including [[ImageNet]], [[CIFAR-10]], and [[CIFAR-100]].<ref name="Foret2021"/>
The algorithm has also been found to be effective in training models with [[Label noise|noisy labels]], where it performs comparably to methods designed specifically for this problem.<ref name="
== Limitations ==
A primary limitation of SAM is its computational cost. By requiring two gradient computations (one for the ascent and one for the descent) per optimization step, it approximately doubles the training time compared to standard optimizers.<ref name="Foret2021"/>
The theoretical [[Convergence of an algorithm|convergence properties]] of SAM are still under investigation. Some research suggests that with a constant step size, SAM may not converge to a stationary point.<ref name="Andriushchenko2022Understanding">{{cite conference |last1=Andriushchenko |first1=Maksym |last2=Flammarion |first2=Nicolas |title=Towards Understanding Sharpness-Aware Minimization |book-title=International Conference on Machine Learning (ICML) 2022 |year=2022 |pages=612–639 |publisher=PMLR |url=https://proceedings.mlr.press/v162/andriushchenko22a.html}}</ref> The accuracy of the single gradient step approximation for finding the worst-case perturbation may also decrease during the training process.<ref name="Kwon2021ASAM">{{cite conference |last1=Kwon |first1=Jungmin |last2=Kim |first2=Jeongseop |last3=Park |first3=Hyunseo |last4=Choi |first4=Il-Chul |year=2021 |title=ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks |url=https://proceedings.mlr.press/v139/kwon21b.html |conference= |publisher=PMLR |pages=5919–5929 |book-title=International Conference on Machine Learning (ICML) 2021
The effectiveness of SAM can also be ___domain-dependent. While it has shown benefits for computer vision tasks, its impact on other areas, such as [[GPT model|GPT-style language models]] where each training example is seen only once, has been reported as limited in some studies.
== Research, Variants, and Enhancements ==
Active research on SAM focuses on reducing its computational overhead and improving its performance. Several variants have been proposed to make the algorithm more efficient. These include methods that attempt to parallelize the two gradient computations, apply the perturbation to only a subset of parameters, or reduce the number of computation steps required.<ref name="Dou2022SAMPa">{{cite arXiv |
To improve performance and robustness, variants have been developed that adapt the neighborhood size based on model parameter scales (Adaptive SAM or ASAM)<ref name="Kwon2021ASAM"/> or incorporate information about the curvature of the loss landscape (Curvature Regularized SAM or CR-SAM).
Theoretical work continues to analyze the algorithm's behavior, including its implicit bias towards flatter minima and the development of broader frameworks for sharpness-aware optimization that use different measures of sharpness.
== References ==
{{reflist}}
[[Category:Machine learning algorithms]]
[[Category:Optimization algorithms and methods]]
| |||