Sharpness aware minimization: Difference between revisions

Revision as of 05:30, 22 May 2025

Latest revision as of 09:00, 27 July 2025

(23 intermediate revisions by 13 users not shown)

Line 1:

{{Short description|Machine learning optimization algorithm}}

{{AfC submission|t||ts=20250522052743|u=2600:1700:3EC7:2CC0:F69C:2A90:F461:223D|ns=118|demo=}}

{{Multiple issues|

<!-- Important, do not remove anything above this line before article has been created. -->

{{technical|date=June 2025}}

{{AI-generated|date=June 2025}}

{{Orphan|date=June 2025}}

}}

'''Sharpness Aware Minimization''' ('''SAM''') is an [[optimization algorithm]] used in [[machine learning]] that aims to improve model [[generalization (machine learning)|generalization]]. The method seeks to find model parameters that are located in regions of the loss landscape with uniformly low loss values, rather than parameters that only achieve a minimal loss value at a single point. This approach is described as finding "flat" minima instead of "sharp" ones. The rationale is that models trained this way are less sensitive to variations between training and test [[data set|data]], which can lead to better performance on unseen data.<ref name="Foret2021">{{cite conference |last1=Foret |first1=Pierre |last2=Kleiner |first2=Ariel |last3=Mobahi |first3=Hossein |last4=Neyshabur |first4=Behnam |year=2021 |title=Sharpness-Aware Minimization for Efficiently Improving Generalization |url=https://openreview.net/forum?id=6Tm1mposlrM |conference= |arxiv=2010.01412 |book-title=International Conference on Learning Representations (ICLR) 2021}}</ref>

The algorithm was introduced in a 2020 paper by a team of researchers including Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.<ref name="Foret2021"/>

{{short description|Optimization algorithm for improving generalization in machine learning models}}

{{technical}}

== Underlying Principle ==

'''Sharpness Aware Minimization''' ('''SAM''') is an [[optimization algorithm]] designed to improve the [[generalization (machine learning)|generalization performance]] of [[machine learning]] models, particularly [[deep neural network]]s. Instead of merely seeking parameters that achieve low training [[loss function|loss]], SAM aims to find parameters that reside in ''neighborhoods'' of uniformly low loss, effectively favoring "flat" minima in the loss landscape over "sharp" ones. The intuition is that models converging to flatter minima are more robust to variations between training and test [[data set|data distributions]], leading to better generalization.<ref name="Foret2021">{{cite conference |last1=Foret |first1=Pierre |last2=Kleiner |first2=Ariel |last3=Mobahi |first3=Hossein |last4=Neyshabur |first4=Behnam |title=Sharpness-Aware Minimization for Efficiently Improving Generalization |booktitle=International Conference on Learning Representations (ICLR) 2021 |year=2021 |arxiv=2010.01412 |url=https://openreview.net/forum?id=6Tm1m_rRrwY}}</ref>

SAM modifies the standard training objective by minimizing a "sharpness-aware" loss. This is formulated as a minimax problem where the inner objective seeks to find the highest loss value in the immediate neighborhood of the current model weights, and the outer objective minimizes this value:<ref name="Foret2021"/>

SAM was introduced by Foret et al. in 2020 in the paper "Sharpness-Aware Minimization for Efficiently Improving Generalization".<ref name="Foret2021"/>

==Core Idea and Mechanism==

The core idea of SAM is to minimize a "sharpness-aware" loss function. This is typically formulated as a minimax problem:<ref name="Foret2021"/>

<math>\min_{w} \max_{\|\epsilon\|_p \le \rho} L_{\text{train}}(w + \epsilon) + \lambda \|w\|_2^2</math>

where:

* <math>w</math> are the model parameters.

* <math>L_{\text{train}}</math> is the training loss.

* <math>\epsilon</math> is an adversarial perturbation.

* <math>\rho</math> is a [[hyperparameter (machine learning)|hyperparameter]] defining the size of the neighborhood (<math>L_p</math> ball) around <math>w</math>.

* The inner maximization finds the perturbation <math>\epsilon</math> that maximizes the loss within the <math>\rho</math>-neighborhood.

* The outer minimization updates the weights <math>w</math> to minimize this maximized loss.

* An optional [[Regularization (mathematics)|L2 regularization]] term can also be included.

In this formulation:

In practice, solving the inner maximization problem exactly is often intractable. SAM approximates the solution by performing a single [[gradient ascent]] step to find the adversarial perturbation <math>\epsilon</math>:<ref name="Foret2021"/>

* <math>w</math> represents the model's parameters (weights).

<math>\epsilon(w) = \rho \frac{\nabla L_{\text{train}}(w)}{\|\nabla L_{\text{train}}(w)\|_2}</math>

* <math>L_{\text{train}}</math> is the [[loss function|loss]] calculated on the training data.

* <math>\epsilon</math> is a perturbation applied to the weights.

* <math>\rho</math> is a [[hyperparameter (machine learning)|hyperparameter]] that defines the radius of the neighborhood (an <math>L_p</math> ball) to search for the highest loss.

* An optional [[Regularization (mathematics)|L2 regularization]] term, scaled by <math>\lambda</math>, can be included.

A direct solution to the inner maximization problem is computationally expensive. SAM approximates it by taking a single [[gradient ascent]] step to find the perturbation <math>\epsilon</math>. This is calculated as:<ref name="Foret2021"/>

The SAM optimizer then typically performs two steps per iteration:<ref name="Foret2021"/>

# '''Ascent Step (Finding "Sharp" Weights):''' Calculate the [[gradient]] <math>\nabla L_{\text{train}}(w)</math> and compute the adversarial weights <math>w_{\text{adv}} = w + \epsilon(w)</math>.

# '''Descent Step (Updating Original Weights):''' Compute the gradient <math>\nabla L_{\text{train}}(w_{\text{adv}})</math> using the adversarial weights and update the original weights <math>w</math> using this gradient, typically with a base optimizer like [[Stochastic gradient descent|SGD]] or [[Adam (optimization algorithm)|Adam]].

<math>\epsilon(w) = \rho \frac{\nabla L_{\text{train}}(w)}{\|\nabla L_{\text{train}}(w)\|_2}</math>

This process encourages the model to converge to regions where the loss remains low even when small perturbations are applied to the weights.

The optimization process for each training step involves two stages. First, an "ascent step" computes a perturbed set of weights, <math>w_{\text{adv}} = w + \epsilon(w)</math>, by moving towards the direction of the highest local loss. Second, a "descent step" updates the original weights <math>w</math> using the gradient calculated at these perturbed weights, <math>\nabla L_{\text{train}}(w_{\text{adv}})</math>. This update is typically performed using a standard optimizer like [[Stochastic gradient descent|SGD]] or [[Adam (optimization algorithm)|Adam]].<ref name="Foret2021"/>

==Scenarios Where SAM Works Well==

SAM has demonstrated significant success in various scenarios:

* '''Improved Generalization:''' SAM consistently leads to better generalization performance across a wide range of deep learning models (especially [[Convolutional Neural Network|Convolutional Neural Networks (CNNs)]] and [[Transformer (machine learning model)|Vision Transformers (ViTs)]]) and datasets (e.g., [[ImageNet]], [[CIFAR-10]], [[CIFAR-100]] dataset|CIFAR-100]]]).<ref name="Foret2021"/>

* '''State-of-the-Art Results:''' It has helped achieve [[state-of-the-art]] or near state-of-the-art performance on several benchmark image classification tasks.<ref name="Foret2021"/>

* '''Robustness to Label Noise:''' SAM inherently provides robustness to [[Label noise|noisy labels]] in training data, performing comparably to methods specifically designed for this purpose.<ref name="Wen2021Mitigating">{{cite arXiv |last1=Wen |first1=Yulei |last2=Liu |first2=Zhen |last3=Zhang |first3=Zhe |last4=Zhang |first4=Yilong |last5=Wang |first5=Linmi |last6=Zhang |first6=Tiantian |title=Mitigating Memorization in Sample Selection for Learning with Noisy Labels |eprint=2110.08529 |year=2021 |class=cs.LG}}</ref><ref name="Zhuang2022Surrogate">{{cite conference |last1=Zhuang |first1=Juntang |last2=Gong |first2=Ming |last3=Liu |first3=Tong |title=Surrogate Gap Minimization Improves Sharpness-Aware Training |booktitle=International Conference on Machine Learning (ICML) 2022 |year=2022 |pages=27098-27115 |publisher=PMLR |url=https://proceedings.mlr.press/v162/zhuang22d.html}}</ref>

* '''[[Out-of-distribution generalization|Out-of-Distribution (OOD) Generalization]]:''' Studies have shown that SAM and its variants can improve a model's ability to generalize to data distributions different from the training distribution.<ref name="Croce2021SAMBayes">{{cite arXiv |last1=Croce |first1=Francesco |last2=Hein |first2=Matthias |title=SAM as an Optimal Relaxation of Bayes |eprint=2110.11214 |year=2021 |class=cs.LG}}</ref><ref name="Kim2022Slicing">{{cite conference |last1=Kim |first1=Daehyeon |last2=Kim |first2=Seungone |last3=Kim |first3=Kwangrok |last4=Kim |first4=Sejun |last5=Kim |first5=Jangho |title=Slicing Aided Hyper-dimensional Inference and Fine-tuning for Improved OOD Generalization |booktitle=Conference on Neural Information Processing Systems (NeurIPS) 2022 |year=2022 |url=https://openreview.net/forum?id=fN0K3jtnQG_}}</ref>

* '''Gradual [[Domain adaptation|Domain Adaptation]]:''' SAM has shown benefits in settings where models are adapted incrementally across changing data domains.<ref name="Liu2021Delving">{{cite arXiv |last1=Liu |first1=Sitong |last2=Zhou |first2=Pan |last3=Zhang |first3=Xingchao |last4=Xu |first4=Zhi |last5=Wang |first5=Guang |last6=Zhao |first6=Hao |title=Delving into SAM: An Analytical Study of Sharpness Aware Minimization |eprint=2111.00905 |year=2021 |class=cs.LG}}</ref>

* '''[[Overfitting]] Mitigation:''' It is particularly effective in scenarios where models might overfit due to seeing training examples multiple times.<ref name="Foret2021"/>

== Application and Performance ==

==Scenarios Where SAM May Not Work Well or Has Limitations==

SAM has been applied in various machine learning contexts, primarily in [[computer vision]]. Research has shown it can improve generalization performance in models such as [[Convolutional Neural Network|Convolutional Neural Networks (CNNs)]] and [[Transformer (machine learning model)|Vision Transformers (ViTs)]] on image datasets including [[ImageNet]], [[CIFAR-10]], and [[CIFAR-100]].<ref name="Foret2021"/>

Despite its strengths, SAM also has limitations:

* '''Increased Computational Cost:''' The most significant drawback of SAM is its computational overhead. Since it requires two forward and backward passes per optimization step, it roughly doubles the training time compared to standard optimizers.<ref name="Foret2021"/>

* '''Convergence Guarantees:''' While empirically successful, theoretical understanding of SAM's [[Convergence of an algorithm|convergence properties]] is still evolving. Some works suggest SAM might have limited capability to converge to global minima or precise stationary points with constant step sizes.<ref name="Andriushchenko2022Understanding">{{cite conference |last1=Andriushchenko |first1=Maksym |last2=Flammarion |first2=Nicolas |title=Towards Understanding Sharpness-Aware Minimization |booktitle=International Conference on Machine Learning (ICML) 2022 |year=2022 |pages=612-639 |publisher=PMLR |url=https://proceedings.mlr.press/v162/andriushchenko22a.html}}</ref>

* '''Effectiveness of Sharpness Approximation:''' The one-step gradient ascent used to approximate the worst-case perturbation <math>\epsilon</math> might become less accurate as training progresses.<ref name="Kwon2021ASAM">{{cite conference |last1=Kwon |first1=Jungmin |last2=Kim |first2=Jeongseop |last3=Park |first3=Hyunseo |last4=Choi |first4=Il-Chul |title=ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks |booktitle=International Conference on Machine Learning (ICML) 2021 |year=2021 |pages=5919-5929 |publisher=PMLR |url=https://proceedings.mlr.press/v139/kwon21a.html}}</ref> Multi-step ascent could be more accurate but would further increase computational costs.

* '''Domain-Specific Efficacy:''' While highly effective in [[computer vision]], its benefits might be less pronounced or require careful tuning in other domains. For instance, some studies found limited or no improvement for [[GPT model|GPT-style language models]] that process each training example only once.<ref name="Chen2023SAMLLM">{{cite arXiv |last1=Chen |first1=Xian |last2=Zhai |first2=Saining |last3=Chan |first3=Crucian |last4=Le |first4=Quoc V. |last5=Houlsby |first5=Graham |title=When is Sharpness-Aware Minimization (SAM) Effective for Large Language Models? |eprint=2308.04932 |year=2023 |class=cs.LG}}</ref>

* '''Potential for Finding "Poor" Flat Minima:''' While the goal is to find generalizing flat minima, some research indicates that in specific settings, sharpness minimization algorithms might converge to flat minima that do not generalize well.<ref name="Liu2023SAMOOD">{{cite conference |last1=Liu |first1=Kai |last2=Li |first2=Yifan |last3=Wang |first3=Hao |last4=Liu |first4=Zhen |last5=Zhao |first5=Jindong |title=When Sharpness-Aware Minimization Meets Data Augmentation: Connect the Dots for OOD Generalization |booktitle=International Conference on Learning Representations (ICLR) 2023 |year=2023 |url=https://openreview.net/forum?id=Nc0e196NhF}}</ref>

* '''Hyperparameter Sensitivity:''' SAM introduces new hyperparameters, such as the neighborhood size <math>\rho</math>, which may require careful tuning for optimal performance.<ref name="Foret2021"/>

The algorithm has also been found to be effective in training models with [[Label noise|noisy labels]], where it performs comparably to methods designed specifically for this problem.<ref name="Zhuang2022Surrogate">{{cite conference |last1=Zhuang |first1=Juntang |last2=Gong |first2=Ming |last3=Liu |first3=Tong |year=2022 |title=Surrogate Gap Minimization Improves Sharpness-Aware Training |url=https://openreview.net/forum?id=edONMAnhLu- |conference= |publisher=PMLR |pages=27098–27115 |book-title=International Conference on Machine Learning (ICML) 2022}}</ref> Some studies indicate that SAM and its variants can improve [[Out-of-distribution generalization|out-of-distribution (OOD) generalization]], which is a model's ability to perform well on data from distributions not seen during training. Other areas where it has been applied include gradual [[___domain adaptation]] and mitigating [[overfitting]] in scenarios with repeated exposure to training examples.<ref name="Foret2021"/>

==Recent Progress and Variants==

Research on SAM is highly active, focusing on improving its efficiency, understanding its mechanisms, and extending its applicability. Key areas of progress include:

* '''Efficiency Enhancements:'''

** '''SAMPa (SAM Parallelized):''' Modifies SAM to allow the two gradient computations to be performed in parallel.<ref name="Dou2022SAMPa">{{cite arXiv |last1=Dou |first1=Yong |last2=Zhou |first2=Cong |last3=Zhao |first3=Peng |last4=Zhang |first4=Tong |title=SAMPa: A Parallelized Version of Sharpness-Aware Minimization |eprint=2202.02081 |year=2022 |class=cs.LG}}</ref>

** '''Sparse SAM (SSAM):''' Applies the adversarial perturbation to only a subset of the model parameters.<ref name="Chen2022SSAM">{{cite arXiv |last1=Chen |first1=Wenlong |last2=Liu |first2=Xiaoyu |last3=Yin |first3=Huan |last4=Yang |first4=Tianlong |title=Sparse SAM: Squeezing Sharpness-aware Minimization into a Single Forward-backward Pass |eprint=2205.13516 |year=2022 |class=cs.LG}}</ref>

** '''Single-Step/Reduced-Step SAM:''' Variants that approximate the sharpness-aware update with fewer computations, sometimes using historical gradient information (e.g., S2-SAM,<ref name="Zhuang2022S2SAM">{{cite arXiv |last1=Zhuang |first1=Juntang |last2=Liu |first2=Tong |last3=Tao |first3=Dacheng |title=S2-SAM: A Single-Step, Zero-Extra-Cost Approach to Sharpness-Aware Training |eprint=2206.08307 |year=2022 |class=cs.LG}}</ref> Momentum-SAM<ref name="He2021MomentumSAM">{{cite arXiv |last1=He |first1=Zequn |last2=Liu |first2=Sitong |last3=Zhang |first3=Xingchao |last4=Zhou |first4=Pan |last5=Zhang |first5=Cong |last6=Xu |first6=Zhi |last7=Zhao |first7=Hao |title=Momentum Sharpness-Aware Minimization |eprint=2110.03265 |year=2021 |class=cs.LG}}</ref>) or applying SAM steps intermittently. Lookahead SAM<ref name="Liu2022LookaheadSAM">{{cite conference |last1=Liu |first1=Sitong |last2=He |first2=Zequn |last3=Zhang |first3=Xingchao |last4=Zhou |first4=Pan |last5=Xu |first5=Zhi |last6=Zhang |first6=Cong |last7=Zhao |first7=Hao |title=Lookahead Sharpness-aware Minimization |booktitle=International Conference on Learning Representations (ICLR) 2022 |year=2022 |url=https://openreview.net/forum?id=7s38W2293F}}</ref> also aims to reduce overhead.

* '''Understanding SAM's Behavior:'''

** '''Implicit Bias Studies:''' Research has shown that SAM has an implicit bias towards flatter minima, and even applying SAM for only a few epochs late in training can yield significant generalization benefits.<ref name="Wen2022SAMLandscape">{{cite arXiv |last1=Wen |first1=Yulei |last2=Zhang |first2=Zhe |last3=Liu |first3=Zhen |last4=Li |first4=Yue |last5=Zhang |first5=Tiantian |title=How Does SAM Influence the Loss Landscape? |eprint=2203.08065 |year=2022 |class=cs.LG}}</ref>

** '''Component Analysis:''' Investigations into which components of the gradient contribute most to SAM's effectiveness in the perturbation step.<ref name="Liu2023FriendlySAM">{{cite conference |last1=Liu |first1=Kai |last2=Wang |first2=Hao |last3=Li |first3=Yifan |last4=Liu |first4=Zhen |last5=Zhang |first5=Runpeng |last6=Zhao |first6=Jindong |title=Friendly Sharpness-Aware Minimization |booktitle=International Conference on Learning Representations (ICLR) 2023 |year=2023 |url=https://openreview.net/forum?id=RndGzfJl4y}}</ref>

* '''Performance and Robustness Enhancements:'''

** '''Adaptive SAM (ASAM):''' Introduces adaptive neighborhood sizes, making the method scale-invariant with respect to the parameters.<ref name="Kwon2021ASAM"/>

** '''Curvature Regularized SAM (CR-SAM):''' Incorporates measures like the normalized [[Hessian matrix|Hessian]] trace to get a more accurate representation of the loss landscape's curvature.<ref name="Kim2022CRSAM">{{cite arXiv |last1=Kim |first1=Minhwan |last2=Lee |first2=Suyeon |last3=Shin |first3=Jonghyun |title=CR-SAM: Curvature Regularized Sharpness-Aware Minimization |eprint=2210.01011 |year=2022 |class=cs.LG}}</ref>

** '''Random SAM (R-SAM):''' Employs random smoothing techniques in conjunction with SAM.<ref name="Singh2021RSAM">{{cite arXiv |last1=Singh |first1=Sandeep Kumar |last2=Ahn |first2=Kyungsu |last3=Oh |first3=Songhwai |title=R-SAM: Random Structure-Aware Minimization for Generalization and Robustness |eprint=2110.07486 |year=2021 |class=cs.LG}}</ref>

** '''Friendly SAM (F-SAM):''' Aims to refine the perturbation by focusing on the stochastic gradient noise component.<ref name="Liu2023FriendlySAM"/>

** '''Delta-SAM:''' This term has been used to describe approaches that use dynamic reweighting or other techniques to approximate per-instance adversarial perturbations more efficiently. Specific implementations and papers may vary.<ref name="Du2022SWA-SAM">{{cite arXiv |last1=Du |first1=Yong |last2=Li |first2=Chang |last3=Kar |first3=Purvak |last4=Krishnapriyan |first4=Adarsh |last5=Xiao |first5=Li |last6=Anil |first6=Rohan |title=An Efficient Way to Improve Generalization: Stochastic Weight Averaging Meets SAM |eprint=2203.04151 |year=2022 |class=cs.LG}}</ref> ** '''μP² (Maximal Update and Perturbation Parameterization):''' Proposes layerwise perturbation scaling to ensure SAM's effectiveness in very wide neural networks.<ref name="Zhang2022Ensemble">{{cite arXiv |last1=Zhang |first1=Jerry |last2=Chen |first2=Tianle |last3=Du |first3=Simon S. |title=Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning |eprint=2202.01074 |year=2022 |class=cs.LG}}</ref> * '''Broader Theoretical Frameworks:'''

** Development of universal classes of sharpness-aware minimization algorithms that can utilize different measures of sharpness beyond the one used in the original SAM (e.g., Frob-SAM using [[Frobenius norm]] of the Hessian, Det-SAM using the [[determinant]] of the Hessian).<ref name="Zhou2023SAMUnified">{{cite arXiv |last1=Zhou |first1=Kaizheng |last2=Zhang |first2=Yulai |last3=Tao |first3=Dacheng |title=Sharpness-Aware Minimization: A Unified View and A New Theory |eprint=2305.10276 |year=2023 |class=cs.LG}}</ref>

== Limitations ==

==Current Open Problems and Future Directions==

A primary limitation of SAM is its computational cost. By requiring two gradient computations (one for the ascent and one for the descent) per optimization step, it approximately doubles the training time compared to standard optimizers.<ref name="Foret2021"/>

Despite significant advancements, several open questions and challenges remain:

* '''Bridging the Efficiency Gap:''' Developing SAM variants that achieve comparable generalization improvements with computational costs close to standard optimizers remains a primary goal.<ref name="Dou2022SAMPa"/><ref name="Chen2022SSAM"/><ref name="Zhuang2022S2SAM"/><ref name="He2021MomentumSAM"/><ref name="Liu2022LookaheadSAM"/>

* '''Deepening Theoretical Understanding:'''

** Providing tighter generalization bounds that fully explain SAM's empirical success.<ref name="Foret2021"/><ref name="Andriushchenko2022Understanding"/><ref name="Neyshabur2021WhatTransferred">{{cite arXiv |last1=Neyshabur |first1=Behnam |last2=Sedghi |first2=Hanie |last3=Zhang |first3=Chiyuan |title=What is being Transferred in Transfer Learning? |eprint=2008.11687 |year=2020 |class=cs.LG}}</ref> ** Establishing more comprehensive convergence guarantees for SAM and its variants under diverse conditions.<ref name="Andriushchenko2022Understanding"/><ref name="Mi2022ConvergenceSAM">{{cite arXiv |last1=Mi |first1=Guanlong |last2=Lyu |first2=Lijun |last3=Wang |first3=Yuan |last4=Wang |first4=Lili |title=On the Convergence of Sharpness-Aware Minimization: A Trajectory and Landscape Analysis |eprint=2206.03046 |year=2022 |class=cs.LG}}</ref>

** Understanding the interplay between sharpness, flatness, and generalization, and why SAM-found minima often generalize well.<ref name="Liu2023SAMOOD"/><ref name="Jiang2020FantasticMeasures">{{cite conference |last1=Jiang |first1=Yiding |last2=Neyshabur |first2=Behnam |last3=Mobahi |first3=Hossein |last4=Krishnan |first4=Dilip |last5=Bengio |first5=Samy |title=Fantastic Generalization Measures and Where to Find Them |booktitle=International Conference on Learning Representations (ICLR) 2020 |year=2020 |url=https://openreview.net/forum?id=SJgMfnR9Y7}}</ref>

* '''Improved Sharpness Approximation:''' Designing more sophisticated and computationally feasible methods to find or approximate the "worst-case" loss in a neighborhood.<ref name="Kwon2021ASAM"/>

* '''Hyperparameter Optimization and Robustness:''' Developing adaptive methods for setting SAM's hyperparameters (like <math>\rho</math>) or reducing its sensitivity to them.<ref name="Kwon2021ASAM"/>

* '''Applicability Across Diverse Domains:''' Further exploring and optimizing SAM for a wider range of machine learning tasks and model architectures beyond computer vision, including [[large language model]]s,<ref name="Chen2023SAMLLM"/> [[reinforcement learning]], and [[graph neural network]]s.

* '''Distinguishing Generalizing vs. Non-Generalizing Flat Minima:''' Investigating how SAM navigates the loss landscape to select flat minima that are genuinely good for generalization, and avoiding those that might be flat but still lead to poor out-of-sample performance.<ref name="Liu2023SAMOOD"/><ref name="Jiang2020FantasticMeasures"/>

* '''Interaction with Other Techniques:''' Understanding how SAM interacts with other regularization techniques, [[Data augmentation|data augmentation]] methods, and architectural choices.

The theoretical [[Convergence of an algorithm|convergence properties]] of SAM are still under investigation. Some research suggests that with a constant step size, SAM may not converge to a stationary point.<ref name="Andriushchenko2022Understanding">{{cite conference |last1=Andriushchenko |first1=Maksym |last2=Flammarion |first2=Nicolas |title=Towards Understanding Sharpness-Aware Minimization |book-title=International Conference on Machine Learning (ICML) 2022 |year=2022 |pages=612–639 |publisher=PMLR |url=https://proceedings.mlr.press/v162/andriushchenko22a.html}}</ref> The accuracy of the single gradient step approximation for finding the worst-case perturbation may also decrease during the training process.<ref name="Kwon2021ASAM">{{cite conference |last1=Kwon |first1=Jungmin |last2=Kim |first2=Jeongseop |last3=Park |first3=Hyunseo |last4=Choi |first4=Il-Chul |year=2021 |title=ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks |url=https://proceedings.mlr.press/v139/kwon21b.html |conference= |publisher=PMLR |pages=5919–5929 |book-title=International Conference on Machine Learning (ICML) 2021}}</ref>

SAM represents a significant step towards building more robust and generalizable deep learning models by explicitly considering the geometry of the loss landscape. Ongoing research continues to refine its efficiency, theoretical underpinnings, and practical applications.

The effectiveness of SAM can also be ___domain-dependent. While it has shown benefits for computer vision tasks, its impact on other areas, such as [[GPT model|GPT-style language models]] where each training example is seen only once, has been reported as limited in some studies. Furthermore, while SAM seeks flat minima, some research suggests that not all flat minima necessarily lead to good generalization. The algorithm also introduces the neighborhood size <math>\rho</math> as a new hyperparameter, which requires tuning.<ref name="Foret2021"/>

==References==

{{reflist}}

== Research, Variants, and Enhancements ==

Active research on SAM focuses on reducing its computational overhead and improving its performance. Several variants have been proposed to make the algorithm more efficient. These include methods that attempt to parallelize the two gradient computations, apply the perturbation to only a subset of parameters, or reduce the number of computation steps required.<ref name="Dou2022SAMPa">{{cite arXiv |eprint=2410.10683 |class=cs.LG |first1=Wanyun |last1=Xie |first2=Thomas |last2=Pethick |title=SAMPa: Sharpness-aware Minimization Parallelized |last3=Cevher |first3=Volkan |year=2022}}</ref><ref name="u277">{{citation |last1=Mi |first1=Peng |title=Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach |date=2022 |page= |arxiv=2210.05177 |last2=Shen |first2=Li |last3=Ren |first3=Tianhe |last4=Zhou |first4=Yiyi |last5=Sun |first5=Xiaoshuai |last6=Ji |first6=Rongrong |last7=Tao |first7=Dacheng }}</ref><ref name="k651">{{cite conference |last1=Ji |first1=Jie |last2=Li |first2=Gen |last3=Fu |first3=Jingjing |last4=Afghah |first4=Fatemeh |last5=Guo |first5=Linke |last6=Yuan |first6=Xiaoyong |last7=Ma |first7=Xiaolong |date=2025-06-05 |title=Proceedings of the 38th International Conference on Neural Information Processing Systems |url=https://dl.acm.org/doi/10.5555/3737916.3739321 |publisher=Curran Associates Inc. |publication-place=Red Hook, NY, USA |volume=37 |page= |pages=44269–44290 |isbn=979-8--33131438-5 |access-date=2025-06-26}}</ref> Other approaches use historical gradient information or apply SAM steps intermittently to lower the computational burden.<ref name="Liu2022LookaheadSAM">{{cite conference |last1=Yu |first1=Runsheng |last2=Zhang |first2=Youzhi |last3=Kwok |first3=James |year=2024 |title=Improving Sharpness-Aware Minimization by Lookahead |url=https://proceedings.mlr.press/v235/yu24q.html |conference= |book-title=International Conference on Learning Representations (ICLR) 2022}}</ref>

To improve performance and robustness, variants have been developed that adapt the neighborhood size based on model parameter scales (Adaptive SAM or ASAM)<ref name="Kwon2021ASAM"/> or incorporate information about the curvature of the loss landscape (Curvature Regularized SAM or CR-SAM). Other research explores refining the perturbation step by focusing on specific components of the gradient or combining SAM with techniques like random smoothing.<ref name="m141">{{cite conference |last1=Li |first1=Tao |last2=Zhou |first2=Pan |last3=He |first3=Zhengbao |last4=Cheng |first4=Xinwen |last5=Huang |first5=Xiaolin |title=2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) |date=2024-06-16 |chapter=Friendly Sharpness-Aware Minimization |page= |publisher=IEEE |pages=5631–5640 |doi=10.1109/CVPR52733.2024.00538 |isbn=979-8-3503-5300-6 }}</ref><ref name="t248">{{cite journal |last1=Liu |first1=Yong |last2=Mai |first2=Siqi |last3=Cheng |first3=Minhao |last4=Chen |first4=Xiangning |last5=Hsieh |first5=Cho-Jui |last6=You |first6=Yang |date=2022-12-06 |title=Random Sharpness-Aware Minimization |url=https://papers.nips.cc/paper_files/paper/2022/hash/9b79416c0dc4b09feaa169ed5cdd63d4-Abstract-Conference.html |journal=Advances in Neural Information Processing Systems |volume=35 |pages=24543–24556 |access-date=2025-06-26}}</ref>

Theoretical work continues to analyze the algorithm's behavior, including its implicit bias towards flatter minima and the development of broader frameworks for sharpness-aware optimization that use different measures of sharpness.

== References ==

<!-- Inline citations added to your article will automatically display here. See en.wikipedia.org/wiki/WP:REFB for instructions on how to add citations. -->

{{reflist}}

[[Category:Machine learning algorithms]]

[[Category:Optimization algorithms and methods]]