Entropy-SGD optimizes a PAC-Bayes bound, leading to improved generalization.
problem Improving generalization in machine learning models.
method Entropy-SGD optimizes a PAC-Bayes bound by adjusting the prior, which is typically chosen independently of the data.
result Entropy-SGD can yield relatively tight generalization bounds and still fit real labels.
This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenval…
Novel kernelized Renyi's entropy improves deep learning generalization bounds.
problem Improving generalization bounds for deep learning algorithms.
method Kernelized Renyi's entropy, a new information theoretical measure.
result Theoretical bounds are tighter than current SOTA results.
Proposes a new algorithm for k k k -means clustering using stochastic backward Euler.
problem Improving k k k -means clustering performance and robustness. method Implicit gradient descent with stochastic backward Euler iteration.
result The algorithm provides better clustering results compared to traditional k k k -means. New algorithms improve neural network generalization by finding flat minima.
problem Finding better generalization in neural networks through flat minima.
method Developed Entropy-SGD and Replicated-SGD algorithms to maximize flatness in the loss function.
result Consistently improved generalization error for various deep learning architectures.
The paper explores how neural networks avoid overfitting by learning from flat minima.
problem How to prevent overfitting in neural networks by learning from flat minima.
method Study of one- and two-layer neural network models, derivation of algorithms focusing on wide flat regions.
result Wide flat minima coexist with narrower minima and critical points, and are associated with good minimizers.
SGD works well with large learning rates at the edge of stability.
problem Stochasticity at the edge of stability in deep learning.
method Sharp convergence guarantees for SGD with multiclass cross-entropy loss.
result SGD self-stabilizes, ensuring convergence with large learning rates.
Simple linear relationship explains test performance differences in deep networks.
problem Understanding why two deep networks with identical training and architecture have different test performance.
method Showed that cross-entropy loss can lead to drastically different generalization performances for networks with different initialization or corrupted training.
result A linear relationship emerges between training and test losses, revealing the intrinsic problem of measuring test performance with cross-entropy loss.
Proposes mutual information for regression without prior knowledge.
problem Regression tasks without prior model knowledge.
method Mutual information learning formulation, SGD convergence analysis.
result High dimensionality can be beneficial with a threshold.
Paper studies SGD stability and optimization error in pairwise learning.
problem Stability and optimization error of SGD for pairwise learning.
method Established stability and optimization error trade-offs for SGD in convex, strongly convex, and non-convex settings.
result Lower bounds for SGD optimization error and excess expected risk.
New analysis of SGD with MCMC gradient estimator shows convergence rate and saddle point escape.
problem Analyzing SGD with MCMC gradient estimator under complex conditions.
method Introduced MCMC-SGD, analyzed convergence rate and saddle point escape using Bernstein inequality.
result Proven first order convergence rate O ( log K / n K ) O(\log K/\sqrt{n K}) O ( log K / n K ) and saddle point escape at least O ( ε − 11 / 2 log 2 ( 1 / ε ) ) O(ε^{-11/2}\log^{2}(1/ε) ) O ( ε − 11/2 log 2 ( 1/ ε )) steps. New study shows deep networks generalize well due to loss surface geometry.
problem Why deep networks generalize well despite many parameters.
method Analyzed local geometry of loss surface and its effect on SGD.
result SGD stays close to low-dimensional subspace, leading to better generalization bounds.
HSIC bottleneck trains deep networks without backpropagation.
problem Training deep neural networks with exploding and vanishing gradients.
method HSIC bottleneck, alternative to cross-entropy loss and backpropagation.
result HSIC bottleneck achieves comparable performance to backpropagation.
This paper explores how adding noise to neural network representations can improve generalization.
problem Improving generalization of neural network encoders.
method Presented a bound on the generalization gap of cross-entropy loss that scales with information complexity (IC) of representations.
result SGD implicitly minimizes information complexity, suggesting that dropout improves generalization capacity.
New research challenges the flatness-generalization link in deep neural networks.
problem The correlation between flatness of the loss landscape and generalization in deep neural networks is questioned.
method The study examines various flatness measures and popular SGD variants, finding some break the flatness-generalization link. It proposes using log P ( f ) \log P(f) log P ( f ) , a global quantity, as a predictor of generalization. result The log of Bayesian prior upon initialization, log P ( f ) \log P(f) log P ( f ) , is a significantly more robust predictor of generalization than flatness measures. Local SGD outperforms minibatch SGD for quadratic objectives.
problem Theoretical foundations of local SGD are lacking.
method Proved local SGD strictly dominates minibatch SGD for quadratic objectives and accelerated local SGD is minimax optimal.
result Local SGD does not dominate minibatch SGD in general convex objectives.
New analysis shows how cross-entropy training shapes attention in transformers.
problem Understanding how gradient-based learning creates the required internal geometry in transformers.
method Developed a first-order analysis of cross-entropy training effects on attention scores and values in a transformer attention head.
result Introduced an advantage-based routing law and responsibility-weighted update for attention scores and values, respectively.
Minibatch SGD outperforms Local SGD in heterogeneous distributed learning.
problem Optimizing a combined convex objective with stochastic gradient estimates from different machines.
method Analysis of Minibatch SGD and Local SGD in a heterogeneous distributed setting.
result Minibatch SGD dominates Local SGD in the heterogeneous distributed setting.
We show that unconverged stochastic gradient descent can be interpreted as a procedure that samples from a nonparametric variational approximate posterior distribution. This distribution is implicitly defined as the transformation of an initial distribution by a sequence of optimization updates. By tracking the change …
WP-SGD optimizes SGD for unevenly distributed data in distributed systems.
problem Inequalities in node performance and data consumption in parallel SGD.
method Combines weighted model parameters from different nodes to compensate for performance inconsistencies.
result WP-SGD significantly outperforms traditional parallel SGD in systems with uneven workloads.
SQuARM-SGD improves decentralized SGD efficiency with momentum.
problem Efficient decentralized training of large-scale models over networks.
method Fixed local SGD steps with Nesterov's momentum, sparsified and quantized updates, locally computed triggering criterion.
result Convergence rate matches vanilla SGD, momentum improves test performance.
Unified framework for efficient distributed SGD algorithms.
problem Lack of rigorous convergence analysis and comparative study of communication-reduction strategies.
method Cooperative SGD framework that unifies and analyzes existing communication-efficient SGD algorithms.
result Novel convergence guarantees for existing algorithms and design of new efficient algorithms.
Improved convergence rates for MFLD in various gradient estimators.
problem Proving convergence rates for mean-field Langevin dynamics with stochastic gradient updates.
method General framework for propagation of chaos, including finite-particle approximation, time-discretization, and stochastic gradient approximation.
result Improved convergence rates for SGD and SVRG settings.
STL-SGD accelerates Local SGD by gradually increasing communication periods.
problem Reducing communication complexity in distributed SGD.
method Gradually increasing communication periods while decreasing learning rate.
result Significant reduction in communication complexity with improved convergence rates.
Nesterov SGD doesn't accelerate over SGD in over-parameterized learning.
problem Theoretical and practical acceleration of SGD with momentum in over-parameterized learning.
method Introducing a compensation term to Nesterov SGD, resulting in MaSS algorithm.
result MaSS converges for same step sizes as SGD and achieves accelerated convergence rates over SGD.
Paper shows local SGD outperforms mini-batch SGD under certain conditions.
problem Proving local SGD's superiority in distributed learning with heterogeneous data.
method New lower and upper bounds for local SGD under first-order heterogeneity assumptions.
result Local SGD is min-max optimal under certain conditions, resolving understanding of distributed optimization.
DeepMaxent uses neural networks to improve species distribution models.
problem Sampling biases and lack of absence data in presence-only observations.
method DeepMaxent employs neural networks to learn shared features among species using the maximum entropy principle.
result DeepMaxent outperforms traditional methods in predicting species distributions, especially in unevenly sampled regions.
LSGD improves deep learning training efficiency by synchronizing and decentralizing SGD.
problem Asynchronous SGD's accuracy issues and synchronous SGD's communication inefficiency.
method LSGD divides nodes into subgroups with centralized communication and decentralized computation.
result LSGD achieves better accuracy and efficiency than synchronous and asynchronous SGD.
Homogenized SGD explains SGD dynamics in high dimensions.
problem Understanding SGD dynamics in high-dimensional settings.
method Developed a homogenized SGD model to analyze high-dimensional SGD.
result Convergent high-dimensional SGD to homogenized SGD for quadratic statistics.
MindFlayer SGD improves parallel SGD for heterogeneous, random compute times.
problem Minimizing nonconvex functions with heterogeneous, random compute times.
method MindFlayer SGD, designed for stochastic and heterogeneous delays.
result MindFlayer SGD outperforms existing methods in environments with heavy-tailed noise.
Large batch training with DP-SGD reduces model performance due to implicit bias.
problem Large batch training with DP-SGD reduces model performance.
method The study analyzes the phenomenon of implicit bias in Noisy-SGD (DP-SGD without clipping) and its theoretical solutions for linear models.
result The implicit bias in large batch training with DP-SGD is amplified by additional noise, similar to SGD.
New proof shows D-SGD and SAM are equivalent, revealing advantages of decentralization.
problem The generalization benefits of decentralized learning.
method Proved D-SGD implicitly minimizes SAM's loss function.
result Decentralized SGD and Average-direction SAM are asymptotically equivalent.
LAGS-SGD optimizes deep learning training by sparsifying gradients layer-wise.
problem Reduces long training times in large deep neural networks with distributed S-SGD.
method Layer-wise adaptive gradient sparsification combined with S-SGD.
result LAGS-SGD achieves convergence guarantees and outperforms vanilla S-SGD.
SGD with large learning rates can converge to local maxima.
problem Understanding the behavior of SGD with large learning rates.
method Constructing worst-case optimization problems.
result SGD can converge to local maxima under certain conditions.
SGD dynamics influence neural network training speed and generalization.
problem Understanding SGD dynamics in relation to neural network loss.
method Empirical investigation and analysis of SGD dynamics in relation to the sharpest directions of the loss.
result Reducing learning rate along the sharpest directions improves training speed and generalization.
Study reveals convergence properties of SGD with random learning rate.
problem Analyzing convergence of SGD with random learning rate in non-convex optimization.
method Introduced Poisson SGD with random learning rate and used stationary distribution analysis.
result Poisson SGD converges to a stationary distribution and finds global minima in non-convex optimization.
VRL-SGD reduces communication complexity in non-identical data settings.
problem Training machine learning models with non-identical data distribution.
method VRL-SGD, which eliminates gradient variance dependency and achieves linear speedup with lower communication complexity.
result VRL-SGD reduces communication complexity from $O(T^{ rac{3}{4}} N^{ rac{3}{4}})$ to $O(T^{ rac{1}{2}} N^{ rac{3}{2}})$ .
New bounds for M-SGD show its error distribution is nearly Gaussian.
problem Understanding the error distribution of M-SGD.
method Proved non-asymptotic bounds for M-SGD in Wasserstein distance.
result Error distribution of M-SGD is approximately Gaussian.
Improved Local SGD convergence for general convex objectives with bounded second-order heterogeneity.
problem Understanding when and why Local SGD outperforms alternatives in distributed optimization.
method Established improved convergence guarantees for Local SGD on general convex objectives under bounded second-order heterogeneity.
result Upper bounds for Local SGD are nearly tight, providing a sharper convergence theory.
Study on the noise in SGD minibatches near local minima.
problem Understanding the noise in SGD minibatches near local minima.
method Detailed analysis of SGD noise in linear regression and derivation of a general formula for different types of minima.
result Provides insight into the stability of training neural networks and suggests large learning rates can help generalization.
Proposes a new method to improve deep neural network performance.
problem Improving generalization in deep neural networks on unseen data.
method Post-synaptic potential regularization (PSP).
result PSP achieves comparable performance to sophisticated learning strategies and improves generalization over ℓ 2 \ell_2 ℓ 2 regularization. The paper improves SGD's generalization error bounds for nonconvex optimization.
problem Improving generalization error bounds for SGD in nonconvex optimization.
method Characterizing the on-average stability of SGD iterates and using it to derive probabilistic generalization error bounds.
result Improved generalization error bounds for SGD in both nonconvex and gradient dominant loss functions.
AdaSGD combines SGD and Adam benefits, eliminating the need for transition.
problem Understanding when to transition from Adam to SGD for optimal performance.
method Adapting a single global learning rate for SGD (AdaSGD).
result AdaSGD combines the benefits of both SGD and Adam, improving convergence and generalization.
SGD outperforms GD in high dimensions via implicit conditioning, revealed by asymptotic analysis.
problem Understanding why SGD outperforms GD in high-dimensional convex problems.
method Asymptotic analysis of multi-pass SGD on high-dimensional convex quadratics, establishing an equivalence to HSGD.
result SGD's efficiency is explained by implicit conditioning, not regularization.
New insights into SGD and SGD-M in high dimensions.
problem Understanding and comparing SGD and SGD-M in high-dimensional settings.
method Developed high-dimensional scaling limits for SGD-M and online SGD, examining their dynamics and performance.
result SGD-M amplifies high-dimensional effects, potentially degrading performance compared to online SGD.
SGD without replacement decouples into curvature-following and flatness-regularizing steps.
problem Theoretical analysis of SGD without replacement for large-scale neural networks.
method Analysis of SGD without replacement in a realistic regime, considering high curvature and flatness.
result Optimizing with SGD without replacement is locally equivalent to an additional regularizer step.
This paper analyzes multi-pass SGD for least squares, improving generalization bounds.
problem Improving generalization bounds for multi-pass SGD in the least squares problem.
method Develops an instance-dependent excess risk bound for least squares in the interpolation regime.
result SGD performs worse than GD instance-wise but saves computational time.
Stochastic Gradient Descent (SGD) is widely used in machine learning problems to efficiently perform empirical risk minimization, yet, in practice, SGD is known to stall before reaching the actual minimizer of the empirical risk. SGD stalling has often been attributed to its sensitivity to the conditioning of the probl…