Is AdamW effective under heavy-tailed noise?
problem Stochastic gradient noise in LLM pretraining is typically heavy-tailed.
method Formulate as an open problem, prove a positive weighted-metric benchmark, and give a corridor lower-bound mechanism.
result No rigorous convergence theory for AdamW established in heavy-tailed regime.
Optimized method tackles convex optimization with heavy-tailed noise.
problem Convex optimization problems with noisy gradients.
method Vanilla stochastic proximal subgradient method without gradient clipping or normalization.
result Achieves optimal complexity for various convex optimization types under heavy-tailed noise.
Efficiently estimates sparse linear regression with heavy-tailed and outlier-contaminated data.
problem Estimating sparse linear regression coefficients with heavy-tailed and outlier-contaminated data.
method Efficient computation of estimators with sharp error bounds.
result Sharp error bounds for efficient estimators.
Work on SGDm under heavy-tailed noise, revealing its generalization properties.
problem Understanding generalization of SGDm under heavy-tailed noise.
method Analysis of continuous-time limit (SDE) and discrete-time SGDm, establishing generalization bounds.
result SGDm can have worse generalization in the presence of heavy-tailed noise for quadratic loss functions.
Study on SA with heavy-tailed and LRD noise, establishing finite-time bounds.
problem Analyzing stochastic approximation under heavy-tailed and LRD noise.
method Noise-averaging argument to regularize impact of non-classical noise.
result Established first finite-time moment bounds for SA under heavy-tailed and LRD noise.
Privacy-preserving SGD with heavy-tailed noise achieves differential privacy guarantees.
problem Privacy preservation in noisy SGD with heavy-tailed noise.
method Differential privacy guarantees for SGD with heavy-tailed noise.
result SGD with heavy-tailed perturbations achieves ( 0 , O ( 1 / n ) ) (0, O(1/n)) ( 0 , O ( 1/ n )) -DP. Generative models learn better with data-adaptive noise.
problem Learning heavy-tailed distributions in flow-based models.
method Data-adaptive latent noise using 1D quantile functions optimized via Wasserstein distance.
result Flexibility and effectiveness in learning heavy-tailed and compactly supported distributions.
Efficiently estimates sparse linear regression with heavy-tailed data and outliers.
problem Sparse estimation of linear regression coefficients with heavy-tailed covariates and noises, including outliers.
method Efficient computation of robust estimator with nearly optimal error bound.
result Nearly optimal error bound for robust sparse estimation.
New method bounds stochastic subgradient methods with heavy-tailed noise.
problem Bounding stochastic subgradient methods under heavy-tailed noise.
method Clipped version of projected stochastic subgradient method.
result Near optimal any-time and finite horizon bounds for averaging schemes.
Truncated SGD with heavy-tailed noise eliminates sharp local minima.
problem Avoiding sharp local minima in deep learning models.
method Truncated SGD with heavy-tailed gradient noise.
result Truncated SGD can eliminate sharp local minima entirely from its training trajectory.
GNIs induce asymmetric heavy-tailed noise in SGD, affecting network performance.
problem The effect of Gaussian noise injections on SGD dynamics and network performance.
method Developed a Langevin-like SDE driven by asymmetric heavy-tailed noise to model the modified SGD dynamics.
result GNIs induce an implicit bias that varies with noise heaviness and asymmetry, affecting network performance.
RS-NSGD improves SGD convergence for heavy-tailed noise.
problem Nonconvex optimization with heavy-tailed noise.
method Integrates direction normalization into subspace updates.
result Achieves better oracle complexity than full-dimensional normalized SGD.
Ridge regression performs optimally in noisy environments with heavy-tailed distributions.
problem Performance of ridge regression in noisy environments with heavy-tailed noise.
method Established excess risk bounds using integral operator framework and Fuk-Nagaev inequality.
result Ridge regression achieves optimal convergence rates under heavy-tailed noise, demonstrating robustness.
DLPM replaces Gaussian noise with α-stable noise in DDPM, improving data distribution coverage and robustness.
problem Handling mode collapse and class imbalance in datasets with heavy-tailed noise.
method Extending DDPM to use α-stable noise, simplifying the process with elementary proof techniques.
result DLPM yields better coverage of data distribution tails, improved robustness to unbalanced datasets, and faster computation times.
Stochastic gradient methods can converge in expectation under heavy-tailed noise.
problem Convergence of stochastic gradient methods under heavy-tailed noise.
method Comprehensive study of stochastic optimization under heavy-tailed noise for e x t s f S G D extsf{SGD} e x t s f S G D , e x t s f S M D extsf{SMD} e x t s f S M D , e x t s f A S M D extsf{ASMD} e x t s f A S M D , e x t s f S G D M extsf{SGDM} e x t s f S G D M in convex and nonconvex optimization. result Established in-expectation convergence results for various stochastic gradient methods.
Paper explores whether gradient normalization can replace clipping for SGD in heavy-tailed noise.
problem Ensuring convergence of SGD in heavy-tailed noise.
method Revisits gradient clipping and normalization, proving their sufficiency and effectiveness.
result Gradient normalization alone is sufficient for nonconvex SGD convergence under smoothness assumptions.
Study improves robustness and sparsity in linear regression with adversarial outliers and heavy-tailed noise.
problem Outliers and heavy-tailed noise in linear regression coefficients.
method Sharp concentration inequalities and generic chaining.
result Sharper error bounds under weaker assumptions.
This paper analyzes bias-variance trade-off for clipped SFOMs, improving complexity guarantees for heavy-tailed noise.
problem Improving complexity guarantees for stochastic optimization methods with heavy-tailed noise.
method Novel analysis of bias-variance trade-off in gradient clipping for clipped SFOMs.
result Improved complexity guarantees for clipped SFOMs across various tail indices, including infinite mean noise.
New methods solve optimization problems with heavy-tailed noise, improving upon existing complexity bounds.
problem Optimization problems with heavy-tailed noise and weakly average smoothness.
method Normalized stochastic first-order methods with Polyak, multi-extrapolated, and recursive momentum.
result First-order oracle complexity results for finding approximate stochastic stationary points under heavy-tailed noise.
Estimates rank-one spikes from heavy-tailed noise using self-avoiding walks.
problem Estimating rank-one spikes from heavy-tailed noise.
method Self-avoiding walks to count and estimate the spikes.
result Optimal estimation up to the BBP threshold for heavy-tailed noise.
Paper tackles robust matrix completion with heavy-tailed noise.
problem Estimating a low-rank matrix from noisy incomplete data.
method Adaptive Huber loss for robustness, nonconvex algorithm with spectral initialization.
result Achieves minimax-optimal statistical estimation error under bounded second moment condition.
New convergence rates for SGD under heavy-tailed noise with infinite variance.
problem Convergence analysis of SGD under heavy-tailed noise with infinite variance.
method Identifying a condition on the Hessian and providing a convergence rate for the distance to the global optimum.
result SGD can converge to the global optimum under heavy-tailed noise with infinite variance.
Stochastic optimization's success linked to heavy-tailed noise.
problem Understanding stochastic optimization's success mechanisms.
method Modeling stochastic optimization as random recurrence relations, analyzing multiplicative noise and heavy-tailed behavior.
result Multiplicative noise leads to heavy-tailed stationary behavior in optimization parameters.
AdaGrad converges under heavy-tailed noise without extra operations.
problem Optimizing with heavy-tailed noise in machine learning.
method Investigation of AdaGrad convergence under heavy-tailed noise.
result First provable convergence rate for AdaGrad in non-convex optimization.
Improves generative models by using heavy-tailed noise in score matching.
problem High-dimensional limitations of Gaussian noise in generative models.
method Extended DSM to generalised normal distribution, relaxed key assumptions, developed iterative noise scaling algorithm.
result Heavy-tailed DSM leads to improved generative performance.
Study on estimating rank-one tensors in noisy data with heavy tails.
problem Estimating rank-one spiked tensors in the presence of heavy tailed errors.
method Analysis of spectral norm of random tensors with iid entries.
result Signal strength requirements for optimal estimation are similar for heavy tailed and Gaussian noise, but vanish for noise with finite fourth moment.
Proposes a new distribution for robust time series modeling with heavy tails.
problem Robust modeling of time series with heavy-tailed noise.
method Spliced Binned-Pareto distribution for non-stationary time series.
result Accurately models extreme events and captures time dependencies in higher moments.
Paper solves robust convex problems with heavy-tailed noise.
problem Solving convex compositional problems with heavy-tailed noise.
method Sub-Gaussian confidence bounds under weak heavy-tailed noise assumptions, using boosting strategy.
result Achieves nearly optimal high probability convergence result.
We propose a new heavy-tailed distribution --- Gaussian-Chain (GC) distribution, which is inspirited by the hierarchical structures prevailing in social organizations. We determine the mean, variance and kurtosis of the Gaussian-Chain distribution to show its heavy-tailed property, and compute the tail distribution tab…
Nonlinear SGD achieves high-probability rates in non-convex optimization with heavy-tailed noise.
problem Optimization in non-convex problems with heavy-tailed noise.
method General nonlinear framework for SGD, including symmetrization techniques.
result Achieves O ~ ( t − 1 / 2 ) \widetilde{\mathcal{O}}(t^{-1/2}) O ( t − 1/2 ) rate for heavy-tailed noise. New RDP guarantees for heavy-tailed SDEs and SGD.
problem Characterizing differential privacy for heavy-tailed noise in learning algorithms.
method Rényi flow computations and fractional Poincaré inequalities.
result First RDP guarantees for heavy-tailed SDEs with weaker dependence on dimension.
New theory explains why normalization is preferred in SGD under heavy-tailed noise.
problem Understanding why normalization is preferred in stochastic gradient descent (SGD) under heavy-tailed noise.
method Developed a worst-case complexity theory for stochastically preconditioned SGD and its variants.
result Normalization guarantees convergence at optimal rates, while clipping may fail in the worst case.
Paper provides tail bounds for stochastic mirror descent in heavy-tailed noise.
problem Optimizing convex and Lipschitz functions with heavy-tailed noise.
method Develops tail bounds for optimization error of Stochastic Mirror Descent.
result Tail bounds extend to heavier-tailed noise regimes without diameter constraints.
Paper develops a TR-SSQP method for noisy optimization with heavy-tailed noise.
problem Optimization problems with stochastic objectives and heavy-tailed noise.
method Trust-Region Stochastic Sequential Quadratic Programming (TR-SSQP) method.
result Achieves high-probability first-order and second-order stationarity bounds for heavy-tailed noise.
New algorithm reduces heavy-tailed linear bandits' computational cost.
problem Stochastic linear bandits with heavy-tailed noise.
method One-pass online mirror descent with adaptive Huber regression.
result Near-optimal regret bound with reduced computational cost.
Paper examines the structure of stochastic gradients in deep learning.
problem Exploring the structure and heavy tails of stochastic gradients in deep learning.
method Conducted formal statistical tests on stochastic gradients and gradient noise.
result Stochastic gradients and gradient noise do not exhibit power-law heavy tails, but their covariance spectra do.
Exponential Lasso improves Lasso's robustness to outliers and heavy-tailed noise.
problem Lasso's sensitivity to outliers and heavy-tailed noise in high-dimensional statistics.
method Integrates an exponential-type loss function into the Lasso framework.
result Achieves strong statistical convergence rates robust to heavy-tailed contamination.
New convergence bounds for online learning with heavy-tailed noise.
problem Learning on streaming data with heavy-tailed noise.
method Nonlinear stochastic gradient descent (SGD) for non-convex and strongly convex costs.
result Strong convergence rates for various nonlinearities and noise distributions.
New method improves convergence of SGD for heavy-tailed noise.
problem Heavy-tailed noise in machine learning applications.
method Normalized SGD (NSGD) to handle heavy-tailed noise.
result Improved sample complexity and high-probability convergence.
New method targets deep learning optima with heavy-tailed noise.
problem Heavy-tailed gradient noise biases SGDm optima.
method Fractional Underdamped Langevin Dynamics (FULD) for heavy-tailed noise.
result FULD targets Gibbs distribution matching optima of cost function.
Paper proposes a self-supervised method to denoise autoregressive signals with heavy-tailed noise.
problem Denoising autoregressive signals corrupted by heavy-tailed noise.
method Self-supervised learning approach without requiring full noise distribution knowledge.
result Strong denoising performance compared to baseline methods, especially for impulsive noise.
New method handles complex systems with discontinuous, heavy-tailed noise.
problem Handling discontinuous, heavy-tailed Lévy noise in stochastic systems.
method Developed nonlocal Kramers-Moyal formulas for SDEs with multiplicative Lévy noise.
result Validated framework for discovering interpretable SDE models from data.
Paper analyzes convergence of stochastic methods under heavy-tailed noise.
problem Analyzing convergence of stochastic methods under heavy-tailed noise.
method Investigates vanilla and clipped stochastic subgradient descent methods.
result Demonstrates convergence properties under sub-Weibull and p-BCM noise assumptions.
Heavy Lasso improves robustness in high-dimensional linear regression with heavy-tailed errors.
problem Challenges of classical Lasso in handling heavy-tailed noise and outliers.
method Data-augmented soft-thresholding with Student's t-distribution loss.
result Heavy Lasso achieves comparable rates to Huber loss under theoretical bounds.
Study improves robust nonparametric regression in heavy-tailed noise.
problem Robust nonparametric regression with heavy-tailed noise and unbounded functions.
method Huber regression in reproducing kernel Hilbert spaces (RKHS), probabilistic effective hypothesis space, new comparison theorems.
result Explicit finite-sample error bounds and convergence rates for Huber regression in RKHS under heavy-tailed noise.
Robust state-space radio interferometric imaging using Stochastic Approximation Expectation Maximization
problem Improving state-space radio interferometric imaging in the presence of heavy-tailed noise
method Stochastic Approximation Expectation Maximization
result Significant improvement in reconstruction fidelity and robustness to radio-frequency interference
Paper develops sparse learning for heavy-tailed time series with locally stationary dynamics.
problem Sparse learning for high-dimensional heavy-tailed locally stationary time series.
method Additive modeling with kernel smoothing, sparsity-inducing penalized estimation.
result Prediction-error bounds and convergence rates for different sparsity structures.
Maximal concentration bounds for stochastic approximation with heavy-tailed noise.
problem Analyzing the convergence of stochastic approximation algorithms under heavy-tailed Markovian noise.
method Novel Lyapunov function and black-box truncation argument.
result Tail behavior of the error can be sub-Gaussian, sub-Weibull, or lighter than any Pareto but heavier than any Weibull.