Large batch training with DP-SGD reduces model performance due to implicit bias.
problem Large batch training with DP-SGD reduces model performance.
method The study analyzes the phenomenon of implicit bias in Noisy-SGD (DP-SGD without clipping) and its theoretical solutions for linear models.
result The implicit bias in large batch training with DP-SGD is amplified by additional noise, similar to SGD.
Resetting from checkpoints improves DNN training with noisy labels.
problem Latent gradient bias induced by noisy labels causes overfitting.
method Stochastic resetting applied to SGD to mitigate latent gradient bias.
result Resetting significantly improves DNN generalization performance.
The gradient noise of SGD is considered to play a central role in the observed strong generalization abilities of deep learning. While past studies confirm that the magnitude and the covariance structure of gradient noise are critical for regularization, it remains unclear whether or not the class of noise distribution…
This research tackles data deletion in linear regression with noisy SGD, finding perfect deleted points.
problem Finding points to delete from a dataset without significantly affecting the training result.
method Signal-to-noise ratio and an algorithm based on it.
result The perfect deleted point is crucial for maintaining model performance and privacy budget.
Projective DP-SGD reduces privacy error by identifying low-dimensional gradient subspaces.
problem Differentially private SGD's error rate scales with model's dimensionality, problematic for over-parameterized models.
method Projective DP-SGD, projecting noisy gradients to a low-dimensional subspace identified from a public dataset.
result The method reduces the dependence on model dimensionality, improving accuracy in high privacy regimes.
Paper relaxes SGD privacy and generalization guarantees for non-smooth convex losses.
problem Privacy and generalization in SGD for non-smooth convex losses.
method Relaxes Lipschitz and strong smoothness assumptions to Hölder smoothness, proving ( ε , δ ) (ε,δ) ( ε , δ ) -DP and optimal excess risk. result Noisy SGD with α α α -Hölder smooth losses achieves optimal excess risk with linear gradient complexity for α ≥ 1 / 2 α \geq 1/2 α ≥ 1/2 . SGD noise has no bias advantage in online learning, contrary to offline learning.
problem The role of SGD noise in online learning.
method Extensive empirical analysis of image and language data.
result Small batch sizes do not confer any implicit bias advantages in online learning.
New bounds for mixing time and privacy in projected Langevin algorithm and noisy SGD.
problem Analyzing mixing times and privacy in projected Langevin algorithm and noisy SGD.
method New bounds derived using PABI framework and optimization problems.
result New bounds for mixing time and privacy in projected Langevin algorithm and noisy SGD, showing dependency on gradient regularity.
Stochastic Gradient Descent (SGD) has become one of the most popular optimization methods for training machine learning models on massive datasets. However, SGD suffers from two main drawbacks: (i) The noisy gradient updates have high variance, which slows down convergence as the iterates approach the optimum, and (ii)…
Paper improves generalization bounds for noisy stochastic algorithms.
problem Improving generalization bounds for noisy stochastic algorithms.
method Introduces Exponential Family Langevin Dynamics (EFLD) and establishes data-dependent expected stability based generalization bounds.
result Sharp generalization bounds with O(1/n) sample dependence and gradient discrepancy.
SGD transitions between maxima and minima with varying time scales.
problem Understanding SGD's behavior near critical points in noisy landscapes.
method Analyzing SGD convergence and escape dynamics in 1D landscapes with infinite- and finite-variance noise.
result SGD reliably moves to the basin's minimum unless close to a local maximum, where it can linger.
Label noise in SGD helps converge to flatter minima.
problem Improving generalization in overparametrized models.
method Analyzes SGD with label noise, showing convergence to regularized minima.
result SGD with label noise converges to flatter minima, improving generalization.
Three DP variants linked, improving SGD privacy bounds.
problem Relating different DP variants for tighter privacy bounds.
method Developed machinery to relate approximate DP to RDP and hypothesis test DP.
result Improved privacy guarantees for noisy SGD.
New framework limits SGD for multi-index models, addressing SQ framework shortcomings.
problem Limitations of SGD for multi-index models beyond SQ framework.
method Developed a new non-SQ framework to study SGD limitations for single-index and multi-index models.
result Applies to broad settings and architectures, including neural networks.
We consider the problem of minimizing a convex risk with stochastic subgradients guaranteeing ε ε ε -locally differentially private ( ε ε ε -LDP). While it has been shown that stochastic optimization is possible with ε ε ε -LDP via the standard SGD (Song et al., 2013), its convergence rate largely depends on the learning rate, w…
Multi-layer neural networks are among the most powerful models in machine learning, yet the fundamental reasons for this success defy mathematical understanding. Learning a neural network requires to optimize a non-convex high-dimensional objective (risk function), a problem which is usually attacked using stochastic g…
The paper analyzes generalization of noisy iterative algorithms using communication theory.
problem Generalization of models trained by noisy iterative algorithms under different distributions.
method Connecting noisy iterative algorithms to additive noise channels in communication theory.
result Distribution-dependent generalization bounds for noisy iterative algorithms.
New algorithms help machines forget old data efficiently.
problem Machine learning models can retain old data, hindering new learning.
method Developed TV-stable algorithms based on noisy SGD for convex and non-convex functions.
result Achieved efficient unlearning with upper and lower bounds on risk.
Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We prop…
FedSel uses local differential privacy to protect data privacy in federated SGD.
problem Privacy leakage from gradients in federated SGD.
method Two-stage framework with top-k dimension selection and gradient accumulation.
result FedSel reduces privacy leakage by privately selecting important dimensions.
Study on gradient clipping in SGD for high-dimensional problems.
problem Understanding and optimizing gradient clipping in high-dimensional machine learning models.
method Theoretical analysis of streaming SGD with gradient clipping in a least squares problem, focusing on large intrinsic dimensionality.
result Developed a deterministic equation to describe the loss evolution in clipped SGD, showing benefits under certain conditions.
FGD reduces noisy gradient variance in SGD for neural networks.
problem Noisy and unreliable gradient estimation in SGD for deep learning.
method Solves an adaptive filtering problem to consistently estimate the local gradient.
result Significantly reduces gradient variance and accelerates convergence.
Privacy-preserving SGD with heavy-tailed noise achieves differential privacy guarantees.
problem Privacy preservation in noisy SGD with heavy-tailed noise.
method Differential privacy guarantees for SGD with heavy-tailed noise.
result SGD with heavy-tailed perturbations achieves ( 0 , O ( 1 / n ) ) (0, O(1/n)) ( 0 , O ( 1/ n )) -DP. Seesaw optimizes training by balancing learning rate and batch size, accelerating model pretraining.
problem Optimizing training efficiency for large language models with adaptive optimizers.
method Develops a principled framework for batch-size scheduling, introducing Seesaw which multiplies learning rate by 1/√2 and doubles batch size.
result Empirically, Seesaw reduces wall-clock time by approximately 36% compared to cosine decay, matching theoretical limits.
Stochastic gradient descent (SGD) still is the workhorse for many practical problems. However, it converges slow, and can be difficult to tune. It is possible to precondition SGD to accelerate its convergence remarkably. But many attempts in this direction either aim at solving specialized problems, or result in signif…
Improves deep learning training by matching mini-batch distributions.
problem Overfitting and noise in mini-batch training.
method ITDM, which matches the moments of mini-batch distributions to reduce overfitting.
result ITDM reduces overfitting and improves DNN training.
SGD trains NNs to learn low-dimensional representations efficiently.
problem Training two-layer NNs with SGD for multiple-index models.
method Online SGD with weight decay for training.
result First-layer weights converge to the principal subspace of the true model.
Improved private AdaGrad achieves faster convergence rates for convex functions.
problem Private empirical risk minimization with differential privacy.
method Noisy AdaGrad with knowledge of gradient subspace geometry.
result Faster convergence rates for convex functions, bypassing traditional bounds.
SAM improves neural network generalization better than SGD, especially in noisy data.
problem Overfitting in large neural networks with label noise.
method Sharpness-Aware Minimization (SAM) compared to Stochastic Gradient Descent (SGD).
result SAM prevents noise learning and facilitates feature learning better than SGD.
Byrd-SAGA reduces variance to robustify SGD against Byzantine attacks.
problem Learning over networks with malicious Byzantine attacks.
method Byrd-SAGA uses geometric median for robust aggregation of corrected stochastic gradients.
result Byrd-SAGA achieves provably linear convergence to optimal solution in the presence of Byzantine workers.
Improved privacy analysis for stochastic gradient descent.
problem Analyzing privacy leakage in noisy stochastic gradient descent.
method Modeling Rényi divergence dynamics with Langevin diffusions, proving exponential privacy loss convergence for smooth and strongly convex objectives.
result Privacy loss converges exponentially fast for smooth and strongly convex objectives under constant step size.
Unified stability bounds for noisy SGD across convex and non-convex losses.
problem Deriving generalization bounds for noisy stochastic gradient descent.
method Unified approach using Lyapunov functions and applied probability.
result Time-uniform stability bounds for SGD on various loss functions.
New method for asynchronous stochastic approximation converges in reinforcement learning.
problem Finding solutions to equations with noisy measurements in reinforcement learning.
method Batch Asynchronous Stochastic Approximation (BASA) with conditions for convergence and rate of convergence.
result Sufficient conditions for convergence and rate of convergence of BASA.
New method estimates SDE parameters efficiently using WCE and SGD.
problem Parameter estimation for stochastic differential equations.
method Wiener Chaos Expansion and Stochastic Gradient Descent.
result Accurate parameter recovery from noisy observations.
New method shows hidden state can significantly improve differential privacy in SGD.
problem Differential privacy in SGD with hidden state.
method Proves converging privacy bounds for hidden state SGD, using privacy amplification techniques.
result Privacy bound converges exponentially fast and is smaller than composition bounds.
Faster weak supervision framework using triplet methods.
problem Computational inefficiency in weak supervision models.
method Closed-form solution for latent variable models, avoiding iterative methods.
result Orders of magnitude faster than previous approaches.
New SGD method uses adaptive sampling to converge faster in non-convex problems.
problem Non-convex optimization problems with noisy gradients.
method Adaptive coordinate sampling in stochastic gradient descent (SGD).
result Almost sure convergence and non-asymptotic bounds established.
Stochastic Gradient Descent (SGD) is a fundamental algorithm in machine learning, representing the optimization backbone for training several classic models, from regression to neural networks. Given the recent practical focus on distributed machine learning, significant work has been dedicated to the convergence prope…
Algorithm generates private continuous-time data for sensitive domains.
problem Private generation of continuous-time data for sensitive domains.
method Mean-field Langevin dynamics and noisy particle gradient descent.
result Strong privacy guarantees for one-time data contributions.
New method optimizes privacy and compute trade-offs for deep learning.
problem Privacy and compute trade-offs in deep learning training.
method Decoupling privacy analysis and experimental behavior, using TAN and scaling laws for DP-SGD.
result Stronger privacy guarantees with significant reduction in computational budget.
AdaBelief optimizes deep learning models with faster convergence and better stability.
problem Combining fast convergence and stability in deep learning models.
method Adapts stepsize based on the belief in observed gradients using exponential moving average (EMA) of noisy gradients.
result AdaBelief outperforms other methods in image classification and GAN training, achieving comparable accuracy to SGD on ImageNet.
Improved neural networks by averaging late-stage weights.
problem Improving the performance of neural networks.
method Ensemble late-stage weights and average them.
result Augmenting standard models with late-phase weights improves generalization.
This paper analyzes shallow ViTs, providing sample complexity and SGD behavior insights.
problem Theoretical understanding of shallow ViTs, especially their sample complexity and SGD behavior.
method Data model with label-relevant and label-irrelevant tokens, theoretical analysis of shallow ViT training.
result Characterization of sample complexity for zero generalization error in shallow ViTs.
New algorithm learns halfspaces with near-optimal sample complexity in noisy conditions.
problem Learning margin halfspaces with Massart noise.
method Computational efficient algorithm using online SGD on carefully selected convex losses.
result Sample complexity of Θ ~ ( 1 / ( γ 2 ε 2 ) ) \widetilde{\Theta}(1/(γ^2 ε^2)) Θ ( 1/ ( γ 2 ε 2 )) , nearly matching lower bound. Differentially private learning avoids iterative optimization in parameter space.
problem Differentially private training of neural networks
method Hypernetworks trained on public datasets
result Significantly reduces noise in parameter space
Improved averaging method for noisy observations converges strongly.
problem Noisy observations from random dynamical systems require stable estimates.
method Introduced p p p -EMA, a modified exponential moving average with subharmonic weight decay. result Stochastic convergence guarantees for p p p -EMA under mild assumptions. Deep neural networks are usually trained with stochastic gradient descent (SGD), which minimizes objective function using very rough approximations of gradient, only averaging to the real gradient. Standard approaches like momentum or ADAM only consider a single direction, and do not try to model distance from extremum…
Last-iterate guarantees for learning in co-coercive games under noisy feedback.
problem Learning in co-coercive games with noisy feedback.
method Vanilla stochastic gradient descent with a new noise model.
result Last-iterate bound of order O ( log ( t ) / t 1 / 3 ) O(\log(t)/t^{1/3}) O ( log ( t ) / t 1/3 ) for co-coercive games.