A new method improves stochastic gradient descent for faster and more efficient estimation.
problem Efficient and fast parametric estimation methods.
method Projected stochastic gradient descent corrected by Fisher scoring.
result The method is faster and more efficient than traditional methods.
Improved SGD algorithm with faster convergence.
problem Optimization of machine learning models.
method Conditional accelerated lazy stochastic gradient descent.
result Convergence rate of $O\left(\frac{1}{\varepsilon^2}
ight)$, faster than previous methods.
Improves SGD for convex functions with mini-batches, proving linear convergence.
problem Minimizing convex functions with constraints.
method Projected semi-stochastic gradient descent with mini-batches.
result Linear convergence under weak strong convexity assumption.
Stochastic approximation algorithms show exponential progress bounds.
problem Analyzing the convergence of stochastic approximation algorithms.
method Developed geometric ergodicity proofs to establish exponential concentration bounds.
result Proved faster convergence rates for specific algorithms.
Paper studies PSGD for constrained optimization problems and its statistical properties.
problem Online inference for constrained optimization problems.
method Stochastic gradient descent with projection (PSGD) for constrained optimization.
result Limiting distribution of PSGD-based estimates under linear-equality constraints.
New algorithms for convex optimization with many constraints using random projections and stochastic gradient descent.
problem Optimization problems with a large number of constraints.
method Stochastic gradient descent combined with random feasibility updates on randomly selected constraint subsets.
result The proposed algorithms converge almost surely and achieve efficient performance.
A parameter-free PGD algorithm for convex optimization.
problem Minimizing convex functions over convex sets.
method A fully adaptive AdaGrad variant of PGD without parameters or restarts.
result Optimal convergence rates for cumulative regret.
We consider stochastic strongly convex optimization with a complex inequality constraint. This complex inequality constraint may lead to computationally expensive projections in algorithmic iterations of the stochastic gradient descent~(SGD) methods. To reduce the computation costs pertaining to the projections, we pro…
Develops first and second-order pseudo-mirror descent methods for nonnegative function estimation.
problem Nonnegative function estimation in settings like MLE and trajectory optimization.
method First and second-order pseudo-mirror descent with pseudo-gradients and projections.
result Establishes tradeoffs and non-asymptotic bounds on model complexity.
Algorithm improves online canonical correlation analysis.
problem Online canonical correlation analysis.
method Stochastic Scaled-Gradient Descent (SSGD) for minimizing expectation over Riemannian manifolds.
result Achieved optimal one-time-scale algorithm with explicit rate of local asymptotic convergence.
New method speeds up training of large kernel models.
problem Scaling kernel machines to large datasets and model sizes.
method Delayed projections in Preconditioned Stochastic Gradient Descent (PSGD).
result Significant training speed up over existing methods.
We model how Lipschitz continuity changes during neural network training.
problem Understanding how Lipschitz continuity evolves during training.
method We use a system of stochastic differential equations to capture the dynamics of Lipschitz continuity under SGD.
result We identify three factors driving the evolution of Lipschitz continuity: gradient flow projection, gradient noise, and Hessian projection.
New algorithm speeds up recovery of low-rank matrices.
problem Nonconvex low-rank matrix recovery problems.
method Stochastic variance-reduced gradient descent with semi-stochastic gradient.
result Linear convergence rate to unknown low-rank matrix.
New projection techniques reduce the frequency of projections in solving LCPs.
problem Solving linearly constrained problems efficiently with reduced projection frequency.
method Delayed projection technique to call a projection less frequently.
result Theoretical and practical improvements in convergence rates and efficiency.
Algorithm selects public datasets for private machine learning.
problem Choosing the most suitable public dataset for private machine learning.
method Measures gradient subspace distance between public and private datasets.
result Excess risk scales with the subspace distance between gradients.
New method estimates SDE parameters efficiently using WCE and SGD.
problem Parameter estimation for stochastic differential equations.
method Wiener Chaos Expansion and Stochastic Gradient Descent.
result Accurate parameter recovery from noisy observations.
Stochastic gradient descent on manifolds improves low-rank approximation.
problem Efficiently approximate large matrices with lower rank.
method Stochastic gradient descent on a manifold.
result Algorithm outperforms Euclidean space methods on Netflix Prize data.
New method optimizes discrete submodular problems using stochastic gradient descent.
problem Optimizing submodular functions in discrete settings with stochastic elements.
method Projected stochastic gradient ascent for continuous optimization, followed by rounding.
result Achieves optimal approximation guarantees with significantly reduced computational cost.
New analysis shows SNG's effectiveness in small samples.
problem Limited understanding of SNG in small data settings.
method Sketch-and-project analysis of SNG.
result Global convergence and rate characterization for SNG.
The paper analyzes how noise geometry influences the performance of SGD in machine learning.
problem Understanding how noise geometry affects the performance of stochastic gradient descent.
method Developed two metrics to quantify noise alignment strength and analyzed their effects on loss and subspace projection dynamics.
result Noise geometry can be used to guarantee alignment under certain conditions, aiding SGD's ability to escape from sharp minima.
Study mini-batch SGD noise and its limits, proving complexity guarantees.
problem Analyzing the noise in mini-batch SGD and its impact on optimization.
method Examined the conditional covariance and diffusion limits of SGD under different sampling designs.
result Proved mean-square upper bounds and Fisher van Trees lower bounds for SGD, linking them to effective dimension and condition number.
The paper analyzes parameter estimation from nonlinear observations.
problem Recovering a structured but unknown parameter from nonlinear observations.
method Develops a framework for characterizing time-data tradeoffs for various parameter estimation algorithms.
result Projected gradient descent schemes converge at a linear rate with near minimal number of samples.
New adaptive step-size method for convex optimization without tuning.
problem Optimizing convex functions efficiently with stochastic gradients.
method Adapted Adaptive Gradient Descent Without Descent to stochastic setting.
result Stochastic gradient descent converges under various assumptions.
Gradient descent optimizes deep ReLU networks with proper initialization.
problem Training deep neural networks with ReLU activation.
method Gradient descent and stochastic gradient descent with proper random weight initialization.
result Gradient descent finds global minima for over-parameterized deep ReLU networks.
Paper introduces P-FGD for online quantile regression models.
problem Training nonparametric additive quantile regression models in online settings.
method Projected functional gradient descent algorithm (P-FGD) for pinball loss.
result P-FGD achieves minimax optimal consistency rate O(t−2s+12s). Stochastic gradient descent converges to universal limits in high dimensions.
problem Statistical tasks in high dimensions with specific data projections.
method Stochastic gradient descent applied to mixture distributions, proving universality of limits.
result The ODE limits are universal for mixtures of arbitrary product distributions.
Kalman Gradient Descent optimizes machine learning models by reducing variance in stochastic optimization.
problem Reducing variance in stochastic gradient descent to improve optimization performance.
method Uses Kalman filtering to adaptively reduce gradient variance in stochastic gradient descent.
result Improved performance on various machine learning tasks including neural networks and black box variational inference.
SGD and stochastic gradient descent converge at optimal rates for certain non-convex functions.
problem Optimal convergence rates for non-convex functions under gradient noise.
method Geometric interpretation of the PL-condition to analyze convergence rates.
result Convergence rates of SGD and stochastic gradient descent match those of strongly convex quadratics.
The paper studies stochastic gradient descent with infinite variance gradients.
problem Theoretical properties of SGD with infinite variance gradients.
method Establish asymptotic behavior of SGD with infinite variance gradients.
result Asymptotic distribution of SGD is characterized as a stationary distribution of an Ornstein-Uhlenbeck process driven by a stable Lévy process.
This work shows neural networks can solve non-convex constraints problems.
problem Training neural networks under non-convex constraints.
method Project stochastic gradient descent with no-regret analysis of online learning.
result Overparameterized neural networks achieve near-optimal and near-feasible solutions.
New guarantees for black-box variational inference methods.
problem Insufficient theoretical guarantees for black-box variational inference.
method Novel convergence guarantees for stochastic optimization of variational inference.
result Provable convergence of proximal and projected stochastic gradient descent for variational inference.
Gradient descent with delayed updates converges faster with noise, even when delays are significant.
problem Analyzing convergence of gradient descent with delayed gradients and stochastic noise.
method Novel technique using generating functions for convergence analysis.
result Convergence bounds show that stochastic noise mitigates the negative effects of delays, improving performance.
Proof given for SGD convergence in a concise manner.
problem Convergence of Stochastic Gradient Descent (SGD)
method Self-contained proof
result SGD convergence proven
SGD reduces test error by decorrelating updates.
problem Improving generalization error in machine learning models.
method Derive a formula for generalization gap change due to SGD updates, compare to GD, and show decorrelation effect.
result SGD implicitly regularizes generalization error by decorrelating updates.
New dynamics for SGD in small learning rate regime.
problem Improving stochastic gradient descent in small learning rate regime.
method Introducing stochastic modified flows and distribution dependent stochastic modified flows.
result Captures fluctuating dynamics of SGD in small learning rate - infinite width scaling regime.
Stochastic gradient descent procedures have gained popularity for parameter estimation from large data sets. However, their statistical properties are not well understood, in theory. And in practice, avoiding numerical instability requires careful tuning of key parameters. Here, we introduce implicit stochastic gradien…
Gradient descent variants improve phase retrieval accuracy.
problem Phase retrieval problem in high-dimensional spaces.
method Gradient descent, stochastic gradient descent, Langevin algorithm, dynamical mean-field theory.
result Stochastic variants of gradient descent achieve better generalization in phase retrieval.
Paper tackles non-convex tensor regression with gradient descent.
problem Learning high-dimensional tensor regression with low-rank structure.
method Projected gradient descent on non-convex constraint set.
result Non-convex projected gradient descent provides superior statistical error and run-time.
Stochastic Gradient Descent prefers minimizers with flat basins in nonconvex problems.
problem Understanding why SGD prefers minimizers with flat basins in nonconvex problems.
method Detailed analysis of a generic stochastic quadratic problem, deriving a deterministic mechanism.
result Derives a deterministic mechanism explaining why SGD prefers flat minimizers.
A new pruning method reduces neural network computation without retraining.
problem Efficiently reduce neural network computation while maintaining accuracy.
method Structured directional pruning via perturbation orthogonal projection.
result Achieves state-of-the-art pruned accuracy without retraining.
A new method for Bayesian inference in high dimensions using projected Stein variational gradient descent.
problem Bayesian inference challenges in high-dimensional data.
method Adapting Stein variational gradient descent to exploit intrinsic low dimensionality of data.
result pSVGD is more accurate and efficient than SVGD, especially in high-dimensional settings.
SAGD uses Langevin algorithm for efficient gradient descent.
problem Efficiently approximating gradients in complex models.
method Langevin algorithm for biased but asymptotically accurate gradients.
result Theoretical convergence guarantee for SAGD.
New adaptive and accelerated SGD methods achieve optimal convergence rates.
problem Optimizing convergence rates of stochastic gradient descent methods.
method Integrates diagonal scaling and momentum into accelerated SGD.
result Achieves optimal sampling and iteration complexity for smooth stochastic optimization.
Langevin algorithms enhance training of deep neural networks for stochastic control problems.
problem Training acceleration for deep neural networks in stochastic control problems.
method Application of Langevin algorithms to minimize the loss of deep neural networks in stochastic control problems.
result Langevin algorithms improve training on various stochastic control problems.
Stochastic gradient methods converge for training wide PINNs.
problem Convergence of stochastic gradient descent in training over-parameterized PINNs.
method Established linear convergence of stochastic gradient descent/flow in training over-parameterized two-layer PINNs.
result Linear convergence with high probability for general activation functions.
This paper uses antithetic sampling to reduce variance in stochastic gradient descent.
problem High variance in stochastic gradient descent slows down convergence.
method Antithetic sampling to make gradients negatively correlated.
result The proposed method accelerates convergence in machine learning applications.
Improved neural network training in low-dimensional random bases.
problem Inefficient optimization in large-scale neural networks.
method Re-draw random subspace at each training step, apply independent projections to different network parts.
result Significantly better optimization performance and efficiency.
The paper analyzes stability and generalization of decentralized SGD.
problem Stability and generalization of decentralized stochastic gradient descent.
method Novel formulation of decentralized stochastic gradient descent combined with non/convex optimization theory.
result First stability and generalization guarantees for decentralized stochastic gradient descent.