Unified analysis improves SAM for non-convex optimization.
problem Improving generalization in machine learning models.
method Sharpness-aware minimization (SAM) and Unified SAM.
result Unified SAM provides convergence guarantees under relaxed assumptions.
Monge SAM improves deep learning by making sharpness-aware minimization invariant to reparametrizations.
problem Non-invariance of sharpness-aware minimization (SAM) to reparametrizations.
method Introduces Monge SAM, a reparametrization-invariant version of SAM using a Riemannian metric.
result Monge SAM enhances robustness and generalization compared to previous methods.
This work analyzes statistical properties of SAM, showing it outperforms GD.
problem Improving deep neural network generalization through flatter solutions.
method Directly studies statistical performance of Sharpness-Aware Minimization (SAM).
result SAM has smaller prediction error than Gradient Descent (GD) under certain conditions.
SAM selects flatter minima late in training, improving generalization.
problem Improving neural network generalization under various settings.
method Sharpness-Aware Minimization (SAM) applied late in training.
result SAM efficiently selects flatter minima late in training, improving generalization.
SAM improves deep learning tasks by promoting balancedness, reducing outlier impact.
problem Improving generalization in deep learning tasks, especially with scale-invariant problems.
method Introduces balancedness as a new concept to depict global behaviors of SAM, focusing on the difference between squared norms of two variables.
result SAM promotes balancedness and is data-responsive, outperforming SGD in outlier scenarios.
SAMPa speeds up SAM by parallelizing its computations.
problem Improving neural network generalization through SAM.
method Parallelizing the two gradient computations in SAM.
result Achieves a twofold speedup of SAM.
New algorithm avoids spurious sharpness minimization for NLP models.
problem SAM fails in NLP, leading to performance degradation.
method Developed Functional-SAM, which modifies logit statistics instead of function geometry.
result Functional-SAM and combined methods outperform AdamW and SAM in NLP tasks.
SAM optimizer struggles to converge to global minima or stationary points in practical settings.
problem Limited convergence of SAM optimizer to global minima or stationary points in practical scenarios.
method Deterministic and stochastic versions of SAM with constant perturbation size and gradient normalization were studied.
result SAM has limited capability to converge to global minima or stationary points in many scenarios.
SAM improves deep learning by relaxing Bayes objective.
problem Improving generalization in deep learning models.
method SAM as a relaxation of Bayes objective using Fenchel biconjugate.
result SAM connects adversarial and Bayesian methods for robustness.
Overparameterization enhances SAM's effectiveness in minimizing sharpness.
problem Improving generalization in deep neural networks.
method Analysis of Sharpness-Aware Minimization (SAM) under varying degrees of overparameterization.
result Overparameterization significantly improves SAM's performance, particularly in noisy and sparse settings.
SAM improves neural network generalization by penalizing sharpness, clarifying its exact notion and mechanism.
problem Improving deep neural network generalization for various settings.
method Sharpness-Aware Minimization (SAM) technique that penalizes a notion of sharpness of the model.
result SAM regularizes the third notion of sharpness, most likely preferred for practical performance.
LSAM optimizes deep learning training with improved efficiency.
problem Inefficiency in distributed large-batch training with Sharpness-Aware Minimization (SAM).
method Integrates SAM's adversarial steps with an asynchronous distributed sampling strategy.
result Higher final accuracy compared to data-parallel SAM.
This work connects SAM to variational inference and evaluates its performance.
problem Improving generalization of gradient-based learning by finding flat minima.
method Establishes connections between SAM and Mean-Field Variational Inference (MFVI), and evaluates variational algorithms combining or interpolating between SAM and MFVI.
result SAM-like updates can be used as a drop-in replacement for the reparametrisation trick.
This paper improves SAM by reformulating it as a bilevel optimization problem.
problem Improving Sharpness-Aware Minimization (SAM) for better performance.
method Reformulate SAM as a bilevel optimization problem using a 0-1 loss surrogate.
result BiSAM consistently results in improved performance compared to SAM and its variants.
SAM optimizer benefits from normalization, stabilizing and guiding optimization.
problem Improving deep neural network performance with SAM optimizer.
method Theoretical and empirical study of normalization in SAM for convex and non-convex functions.
result Normalization helps SAM in stabilizing and guiding optimization along a continuum of minima.
SAM improves generalization in overparameterized models, but its behavior in tensorized models is less understood.
problem Understanding the implicit regularization of SAM in tensorized models.
method Scale-invariance analysis and gradient flow analysis to derive Norm Deviation as a measure of core norm imbalance, and propose Deviation-Aware Scaling (DAS).
result DAS achieves competitive or improved performance over SAM, while offering reduced computational overhead.
New method μP2 improves neural network training by scaling perturbations layerwise.
problem Improving neural network performance as models scale up.
method Layerwise perturbation scaling in the infinite-width limit of neural networks.
result Layerwise perturbation scaling ensures all layers are effectively perturbed in the limit.
LEARN-SAM improves RL from sub-optimal demonstrations by localizing expert policies and selectively using demonstrations.
problem Improving RL from sub-optimal or sparse demonstrations.
method Local Ensemble and Reparameterization with Split and Merge of expert policies (LEARN-SAM).
result LEARN-SAM boosts learning speed and accuracy by selectively using demonstrations.
mSAM improves generalization by making models flatter.
problem Over-parameterized models can have varying generalization performance.
method mSAM modifies loss function to favor flatter minima through adversarial perturbations.
result mSAM achieves flatter minima than SAM and SGD, improving generalization.
SAM improves neural network generalization better than SGD, especially in noisy data.
problem Overfitting in large neural networks with label noise.
method Sharpness-Aware Minimization (SAM) compared to Stochastic Gradient Descent (SGD).
result SAM prevents noise learning and facilitates feature learning better than SGD.
This paper presents a Semantic Attribute Modulation (SAM) for language modeling and style variation. The semantic attribute modulation includes various document attributes, such as titles, authors, and document categories. We consider two types of attributes, (title attributes and category attributes), and a flexible a…
New adaptive scheduler improves SAM for better model training.
problem Training machine learning models requires selecting a learning rate, which is often difficult and time-consuming.
method Derive Polyak schedulers tailored to SAM-style updates, proving linear convergence for strongly convex objectives and an O(1/T) rate for convex objectives.
result Polyak schedulers achieve comparable or better performance than tuned SAM baselines, reducing the need for learning-rate tuning.
SAMS-VAE models cellular perturbations using sparse additive mechanisms.
problem Modeling effects of diverse interventions on cells.
method Sparse Additive Mechanism Shift Variational Autoencoder (SAMS-VAE).
result SAMS-VAE identifies disentangled, perturbation-specific latent subspaces.
Develops new methods to evaluate data influence in SAM for improved model training.
problem Challenges in mislabeled noisy data and privacy concerns in SAM.
method Two innovative data valuation methods based on influence functions (IF) for SAM.
result Demonstrates effectiveness in identifying mislabeled data and enhancing interpretability.
LORENZA improves LLM fine-tuning efficiency and generalization.
problem Improving robustness and generalization of LLMs under hardware constraints.
method AdaZo-SAM and LORENZA, combining Adam and SAM with zeroth-order estimation and randomized SVD.
result LORENZA achieves better generalization and reduced memory consumption compared to existing methods.
New optimization method improves generalization across various tasks.
problem Improving zeroth-order optimization for better generalization.
method Exponential tilting objective to connect zeroth-order optimization with sharpness-aware minimization.
result Achieves better generalization compared to vanilla zeroth-order baselines.
SAM improves generalization by operating near the edge of stability.
problem Improving generalization in neural networks.
method Sharpness-Aware Minimization (SAM) approach to training neural networks.
result SAM operates near the 'edge of stability' identified by the analysis.
SAM optimizes deep networks by oscillating between sides of the minimum.
problem Improving performance of deep networks.
method Gradient-based optimization method that oscillates between sides of the minimum.
result SAM effectively performs gradient descent on the spectral norm of the Hessian, encouraging drift towards wider minima.
A new causal discovery method, Structural Agnostic Modeling (SAM), is presented in this paper. Leveraging both conditional independencies and distributional asymmetries, SAM aims to find the underlying causal structure from observational data. The approach is based on a game between different players estimating each va…
New proof shows D-SGD and SAM are equivalent, revealing advantages of decentralization.
problem The generalization benefits of decentralized learning.
method Proved D-SGD implicitly minimizes SAM's loss function.
result Decentralized SGD and Average-direction SAM are asymptotically equivalent.
Improves model generalization by minimizing loss sharpness.
problem Overparameterized models often fail to generalize well despite low training loss.
method Sharpness-Aware Minimization (SAM) minimizes both loss value and sharpness.
result SAM improves model generalization across various datasets and models.
A new spectrum attention mechanism improves time series classification.
problem Improving robustness and classification accuracy in time series classification.
method Proposes a spectrum attention mechanism (SAM) to filter and highlight important frequency components, using L1 regularization and a tumbling window for segmentation.
result Experimental results show that the proposed SSAM method produces better feature representations and improves classification accuracy.
DGSAM improves domain generalization by minimizing individual sharpness.
problem Improving domain generalization models that perform well on unseen target domains.
method Shifts DG paradigm toward minimizing individual sharpness across source domains.
result DGSAM reduces performance variance across domains with less computational overhead.
Proposes a novel SAM operator for separate item and relational memories.
problem Limited memory interactions in neural networks.
method Introduces a Self-attentive Associative Memory (SAM) operator to separate item and relational memories.
result Achieves competitive results in various tasks, including geometry, graph, reinforcement learning, and question answering.
Agent-based model uses SAM to create realistic economic system.
problem Lack of tools to understand and predict economic crises.
method Agent-based modeling (ABM) with Social Accounting Matrix (SAM) calibration.
result ABM can produce economic systems close to real-world data.
SAM minimizes loss sharpness, improving adversarial transferability.
problem Improving adversarial transferability of deep neural networks.
method Evaluating surrogate models trained with seven minimizers, focusing on loss sharpness and flat neighborhoods.
result SAM minimizes loss sharpness, leading to better adversarial transferability.
We show that the classification performance of graph convolutional networks (GCNs) is related to the alignment between features, graph, and ground truth, which we quantify using a subspace alignment measure (SAM) corresponding to the Frobenius norm of the matrix of pairwise chordal distances between three subspaces ass…
New findings cast doubt on the role of λmax in generalizing neural networks.
problem The role of λmax in neural network generalization remains unclear. method Experiments with various training interventions and batch sizes.
result Generalization benefits can vanish at larger batch sizes, challenging the role of λmax. Conference compiles problems on foliations and diffeomorphisms.
problem Challenges in foliations and diffeomorphism groups.
method Compilation of problems from conference participants.
result Compilation of 20+ problems on foliations and diffeomorphisms.
Corrects a flawed proof of the Kropholler Conjecture.
problem Kropholler Conjecture in relative group theory.
method Explains the flaws and proposes possible ways to correct the proof.
result Provides a clearer understanding of the conjecture's proof difficulties.
MSPM uses modular agents to manage financial portfolios efficiently.
problem Scalability and reusability issues in RL-based financial portfolio management.
method Modular design with Evolving Agent Module (EAM) and Strategic Agent Module (SAM).
result MSPM improves profit accumulation by at least 186.5% compared to CRP.
Study on quandle coloring quivers for (p, 2)-torus knots and links.
problem Understanding quandle colorings of (p, 2)-torus knots and links.
method Introduced quandle coloring quivers and studied them for dihedral quandles.
result Characterized quandle coloring quivers for (p, 2)-torus knots and links.
We prove a representation stability result for the second homology groups of Torelli subgroups of mapping class groups and automorphism groups of free groups. This strengthens the results of Boldsen-Hauge Dollerup and Day-Putman. We also prove a new representation stability result for the homology of certain congruence…
A new framework for brain mapping using statistical agnostic methods.
problem Estimating brain connectivity with limited data and controlling false positives.
method Statistical Agnostic Mapping (SAM) based on concentration inequalities.
result Relieves instability and provides less conservative p-value correction.
New axioms for singquandles simplify applications and reveal algebraic aspects.
problem Axiomatizing singular knots and links.
method Presented new axioms for singquandles, simplified existing ones, and reformulated for affine singquandles.
result Simplified applications and revealed new algebraic aspects of singquandles.
We introduce a technique for proving quantitative representation stability theorems for sequences of representations of certain finite linear groups over a field of characteristic zero. In particular, we prove a vanishing result for higher syzygies of VIC- and SI-modules, which can be thought of as a weaker version of …
New Langlands duality conjectures for 3-manifold skein modules.
problem Understanding Langlands duality in 3-manifold skein modules.
method Combining representation theory of double affine Hecke algebras and 1-form symmetry structure.
result Recent special cases confirmed, with detailed proofs.
Among interpretable machine learning methods, the class of Generalised Additive Neural Networks (GANNs) is referred to as Self-Explaining Neural Networks (SENN) because of the linear dependence on explicit functions of the inputs. In binary classification this shows the precise weight that each input contributes toward…