LSAM optimizes deep learning training with improved efficiency.
problem Inefficiency in distributed large-batch training with Sharpness-Aware Minimization (SAM).
method Integrates SAM's adversarial steps with an asynchronous distributed sampling strategy.
result Higher final accuracy compared to data-parallel SAM.
Sharp-MAML improves MAML by reducing saddle points in few-shot learning.
problem Challenges in optimizing MAML due to complex loss landscape.
method Sharpness-aware minimization applied to MAML.
result Sharp-MAML and its variant outperform plain MAML on few-shot learning tasks.
Monge SAM improves deep learning by making sharpness-aware minimization invariant to reparametrizations.
problem Non-invariance of sharpness-aware minimization (SAM) to reparametrizations.
method Introduces Monge SAM, a reparametrization-invariant version of SAM using a Riemannian metric.
result Monge SAM enhances robustness and generalization compared to previous methods.
This work connects SAM to variational inference and evaluates its performance.
problem Improving generalization of gradient-based learning by finding flat minima.
method Establishes connections between SAM and Mean-Field Variational Inference (MFVI), and evaluates variational algorithms combining or interpolating between SAM and MFVI.
result SAM-like updates can be used as a drop-in replacement for the reparametrisation trick.
SAM improves generalization by operating near the edge of stability.
problem Improving generalization in neural networks.
method Sharpness-Aware Minimization (SAM) approach to training neural networks.
result SAM operates near the 'edge of stability' identified by the analysis.
DGSAM improves domain generalization by minimizing individual sharpness.
problem Improving domain generalization models that perform well on unseen target domains.
method Shifts DG paradigm toward minimizing individual sharpness across source domains.
result DGSAM reduces performance variance across domains with less computational overhead.
SAM optimizes deep networks by oscillating between sides of the minimum.
problem Improving performance of deep networks.
method Gradient-based optimization method that oscillates between sides of the minimum.
result SAM effectively performs gradient descent on the spectral norm of the Hessian, encouraging drift towards wider minima.
SAM optimizer benefits from normalization, stabilizing and guiding optimization.
problem Improving deep neural network performance with SAM optimizer.
method Theoretical and empirical study of normalization in SAM for convex and non-convex functions.
result Normalization helps SAM in stabilizing and guiding optimization along a continuum of minima.
Noise in linear networks minimizes sharpness and leads to shrinkage-thresholding.
problem Minimizing sharpness in diagonal linear networks.
method Stochastic sharpness-aware minimization (SAM) with isotropic noise.
result Noise forces shrinkage-thresholding of true parameters.
SAMPa speeds up SAM by parallelizing its computations.
problem Improving neural network generalization through SAM.
method Parallelizing the two gradient computations in SAM.
result Achieves a twofold speedup of SAM.
SAMformer improves transformer performance in time series forecasting.
problem Transformers struggle with multivariate long-term forecasting.
method Sharpness-aware minimization and channel-wise attention.
result SAMformer surpasses state-of-the-art methods in multivariate time series forecasting.
Paper proposes a method to estimate scientific parameters in hybrid models without relying on model architecture.
problem Estimating unknown parameters in hybrid models combining machine learning and scientific models.
method Sharpness-aware minimization adapted for hybrid modeling, focusing on model simplicity.
result Demonstrates effectiveness of SAM-based hybrid model learning for scientific parameter estimation.
SAM improves deep learning tasks by promoting balancedness, reducing outlier impact.
problem Improving generalization in deep learning tasks, especially with scale-invariant problems.
method Introduces balancedness as a new concept to depict global behaviors of SAM, focusing on the difference between squared norms of two variables.
result SAM promotes balancedness and is data-responsive, outperforming SGD in outlier scenarios.
SAM improves neural network generalization better than SGD, especially in noisy data.
problem Overfitting in large neural networks with label noise.
method Sharpness-Aware Minimization (SAM) compared to Stochastic Gradient Descent (SGD).
result SAM prevents noise learning and facilitates feature learning better than SGD.
New optimization method improves generalization across various tasks.
problem Improving zeroth-order optimization for better generalization.
method Exponential tilting objective to connect zeroth-order optimization with sharpness-aware minimization.
result Achieves better generalization compared to vanilla zeroth-order baselines.
This work analyzes statistical properties of SAM, showing it outperforms GD.
problem Improving deep neural network generalization through flatter solutions.
method Directly studies statistical performance of Sharpness-Aware Minimization (SAM).
result SAM has smaller prediction error than Gradient Descent (GD) under certain conditions.
New adaptive scheduler improves SAM for better model training.
problem Training machine learning models requires selecting a learning rate, which is often difficult and time-consuming.
method Derive Polyak schedulers tailored to SAM-style updates, proving linear convergence for strongly convex objectives and an O(1/T) rate for convex objectives.
result Polyak schedulers achieve comparable or better performance than tuned SAM baselines, reducing the need for learning-rate tuning.
SAM improves generalization in overparameterized models, but its behavior in tensorized models is less understood.
problem Understanding the implicit regularization of SAM in tensorized models.
method Scale-invariance analysis and gradient flow analysis to derive Norm Deviation as a measure of core norm imbalance, and propose Deviation-Aware Scaling (DAS).
result DAS achieves competitive or improved performance over SAM, while offering reduced computational overhead.
New method μP2 improves neural network training by scaling perturbations layerwise.
problem Improving neural network performance as models scale up.
method Layerwise perturbation scaling in the infinite-width limit of neural networks.
result Layerwise perturbation scaling ensures all layers are effectively perturbed in the limit.
SAM selects flatter minima late in training, improving generalization.
problem Improving neural network generalization under various settings.
method Sharpness-Aware Minimization (SAM) applied late in training.
result SAM efficiently selects flatter minima late in training, improving generalization.
Overparameterization enhances SAM's effectiveness in minimizing sharpness.
problem Improving generalization in deep neural networks.
method Analysis of Sharpness-Aware Minimization (SAM) under varying degrees of overparameterization.
result Overparameterization significantly improves SAM's performance, particularly in noisy and sparse settings.
SALR improves deep learning generalization by dynamically adjusting learning rates.
problem Improving generalization in deep learning models.
method Sharpness-aware learning rate scheduling based on local loss function sharpness.
result SALR drives solutions to flatter regions, improving generalization and convergence.
SAM improves neural network generalization by penalizing sharpness, clarifying its exact notion and mechanism.
problem Improving deep neural network generalization for various settings.
method Sharpness-Aware Minimization (SAM) technique that penalizes a notion of sharpness of the model.
result SAM regularizes the third notion of sharpness, most likely preferred for practical performance.
Develops new methods to evaluate data influence in SAM for improved model training.
problem Challenges in mislabeled noisy data and privacy concerns in SAM.
method Two innovative data valuation methods based on influence functions (IF) for SAM.
result Demonstrates effectiveness in identifying mislabeled data and enhancing interpretability.
Unified analysis improves SAM for non-convex optimization.
problem Improving generalization in machine learning models.
method Sharpness-aware minimization (SAM) and Unified SAM.
result Unified SAM provides convergence guarantees under relaxed assumptions.
ASAM improves deep neural network generalization by adapting sharpness to scale.
problem Fixed-radius sharpness measure is sensitive to parameter scaling, weakening its connection to generalization.
method Introduces adaptive sharpness, a scale-invariant measure, and proposes ASAM for deep learning.
result ASAM significantly improves model generalization performance across various datasets.
SAM optimizer struggles to converge to global minima or stationary points in practical settings.
problem Limited convergence of SAM optimizer to global minima or stationary points in practical scenarios.
method Deterministic and stochastic versions of SAM with constant perturbation size and gradient normalization were studied.
result SAM has limited capability to converge to global minima or stationary points in many scenarios.
This paper improves SAM by reformulating it as a bilevel optimization problem.
problem Improving Sharpness-Aware Minimization (SAM) for better performance.
method Reformulate SAM as a bilevel optimization problem using a 0-1 loss surrogate.
result BiSAM consistently results in improved performance compared to SAM and its variants.
New proof shows D-SGD and SAM are equivalent, revealing advantages of decentralization.
problem The generalization benefits of decentralized learning.
method Proved D-SGD implicitly minimizes SAM's loss function.
result Decentralized SGD and Average-direction SAM are asymptotically equivalent.
mSAM improves generalization by making models flatter.
problem Over-parameterized models can have varying generalization performance.
method mSAM modifies loss function to favor flatter minima through adversarial perturbations.
result mSAM achieves flatter minima than SAM and SGD, improving generalization.
Improves model generalization by minimizing loss sharpness.
problem Overparameterized models often fail to generalize well despite low training loss.
method Sharpness-Aware Minimization (SAM) minimizes both loss value and sharpness.
result SAM improves model generalization across various datasets and models.
New algorithm avoids spurious sharpness minimization for NLP models.
problem SAM fails in NLP, leading to performance degradation.
method Developed Functional-SAM, which modifies logit statistics instead of function geometry.
result Functional-SAM and combined methods outperform AdamW and SAM in NLP tasks.
SAM improves deep learning by relaxing Bayes objective.
problem Improving generalization in deep learning models.
method SAM as a relaxation of Bayes objective using Fenchel biconjugate.
result SAM connects adversarial and Bayesian methods for robustness.
Flat-minima optimizers improve neural network generalization.
problem Improving neural network generalization performance.
method Stochastic Weight Averaging (SWA) and Sharpness-Aware Minimization (SAM).
result Surprising findings from loss surface analysis and broad benchmarking.
Enhances deep learning by boosting generalization and convergence.
problem Improving generalization and convergence in deep learning models.
method Implicit Regularization Enhancement (IRE) framework that decouples flat and sharp directions.
result IRE consistently improves generalization performance across various deep learning tasks and models.
New Stein identity for q-Gaussians reduces gradient variance in machine learning.
problem Improving gradient estimators for non-Gaussian distributions.
method Deriving a new Stein identity for bounded-support q-Gaussians and simplifying previous results.
result Gradient estimators for q-Gaussians have nearly identical forms to Gaussian ones, reducing variance.
New framework learns sufficient invariant features robustly across distribution shifts.
problem Learning robust models under distribution shifts between training and test datasets.
method Sufficient Invariant Learning (SIL) framework and Adaptive Sharpness-aware Group Distributionally Robust Optimization (ASGDRO) algorithm.
result Empirical evaluations confirm ASGDRO's robustness against distribution shifts.
SRO optimizes decisions against worst-case sampler induced by generative models.
problem Operational uncertainty shifts from explicit probability law to sampler induced by learned generators.
method SRO optimizes decisions against the worst-case sampler induced by perturbing the learned generator.
result Empirical worst-case objective provides high-probability upper certificate for true population objective.
SAM minimizes loss sharpness, improving adversarial transferability.
problem Improving adversarial transferability of deep neural networks.
method Evaluating surrogate models trained with seven minimizers, focusing on loss sharpness and flat neighborhoods.
result SAM minimizes loss sharpness, leading to better adversarial transferability.
SGD updates align with a low-rank subspace but do not lead to further loss reduction.
problem Understanding the training dynamics of deep neural networks, particularly the role of the dominant subspace.
method Exploring whether neural networks can be trained within the dominant subspace of the loss Hessian.
result SGD updates, when projected onto the dominant subspace, do not decrease the training loss further, suggesting spurious alignment.
LORENZA improves LLM fine-tuning efficiency and generalization.
problem Improving robustness and generalization of LLMs under hardware constraints.
method AdaZo-SAM and LORENZA, combining Adam and SAM with zeroth-order estimation and randomized SVD.
result LORENZA achieves better generalization and reduced memory consumption compared to existing methods.
A model predicts how hyperparameters affect pruning performance.
problem Predicting the impact of hyperparameters on pruning performance.
method Phenomenological model using temperature-like and load-like parameters.
result A sharp transition phenomenon in pruning performance.
This paper improves HNNs by learning optimal curvature for better generalization.
problem Inappropriate curvatures in HNNs lead to suboptimal performance.
method Sharpness-aware curvature learning method to smooth loss landscape.
result Proposed method improves HNNs' generalization across various settings.
New findings cast doubt on the role of λmax in generalizing neural networks.
problem The role of λmax in neural network generalization remains unclear. method Experiments with various training interventions and batch sizes.
result Generalization benefits can vanish at larger batch sizes, challenging the role of λmax. Outliers with opposing signals significantly affect neural network optimization.
problem Understanding and mitigating the impact of outliers with opposing signals on neural network training.
method Identifying and analyzing pairs of outliers with strong opposing signals in training data.
result Outliers with opposing signals can cause optimization to enter a narrow valley, leading to oscillatory behavior and eventual loss spikes.