A new algorithm SSAG reduces gradient variance to achieve linear convergence in large-scale optimization.
problem Achieving linear convergence rate in large-scale optimization problems due to gradient variance.
method Introduces SSAG, a novel algorithm that combines stratified sampling and averaging over iterations to reduce gradient variance.
result SSAG achieves linear convergence rate of O((1-μ/(8CL))^k) with smaller storage and iterative costs, depending mainly on class variance.
Study shows gradient variance increases during deep learning training, contrary to common belief.
problem Understanding and minimizing gradient variance in deep learning models.
method Gradient Clustering method using stratified sampling to minimize gradient variance.
result Gradient variance increases during training, and smaller learning rates coincide with higher variance.
Stochastic Gradient Descent (SGD) is a popular optimization method which has been applied to many important machine learning tasks such as Support Vector Machines and Deep Neural Networks. In order to parallelize SGD, minibatch training is often employed. The standard approach is to uniformly sample a minibatch at each…
MSTGD optimizes gradient descent with stratified sampling for faster convergence.
problem Fluctuation in gradient expectation and variance between iterations.
method Memory Stochastic Stratified Gradient Descent (MSTGD) with stratified sampling and variance reduction.
result MSTGD achieves an exponential convergence rate independent of dataset size and batch size.
Paper proposes SCott optimizer to reduce forecasting model training variance.
problem Large variance in gradient estimation for forecasting models.
method Stratified sampling and control variate to reduce gradient variance.
result SCott optimizer converges faster on time series forecasting problems.
A new averaging scheme improves stochastic gradient methods.
problem Improving the performance of stochastic gradient descent.
method Proposes a weighted averaging scheme with decaying weights.
result The method has a regularizing effect equivalent to ridge regression.
DM-SGD uses DPP to diversify mini-batches for SGD, improving model performance.
problem Improving mini-batch diversity in SGD to reduce variance and improve model interpretability.
method DM-SGD uses Determinantal Point Processes (DPP) to select mini-batches with diverse data points.
result DM-SGD outperforms regular SGD and stratified sampling in various setups.
Efficiently decomposes large tensors using stochastic gradients.
problem Efficiently decomposing large tensors for multiway data analysis.
method Stochastic gradients computed via MTTKRP kernel for efficient computation.
result Advantages and scalability demonstrated for large-scale problems.
A new method for faster optimization of machine learning problems.
problem Minimization of composition of expected value functions.
method C-SAG, a novel extension of SAG for FS-CEVF problems.
result C-SAG achieves lower oracle query complexity per iteration than C-SVRG and converges faster.
Analysis of a stochastic system showing convergence to an averaged model with Gaussian deviations.
problem Convergence analysis of a perturbed compositional gradient flow system.
method Separation of scales and averaging principle applied to stochastic differential equations.
result The slow motion of the system can be approximated by a standard perturbed gradient flow or SCGD algorithm.
Our objective is to develop a stratified Morse theory with tangential conditions. We define a continuous strata-wise smooth Morse function on an abstract stratified space by using control conditions and radiality assumptions on the gradient vector field. For critical points of a Morse function one can show that the loc…
Paper explores weighted averaging schemes for SGD, achieving asymptotic normality and optimality.
problem Improving convergence of SGD in various settings.
method Develops a general weighted averaging scheme for SGD and establishes asymptotic normality.
result Establishes asymptotic normality and optimality of weighted averaged SGD solutions.
Paper introduces a new gradient statistic to improve deep learning convergence.
problem Fluctuation effect of gradient updates between iterations.
method Introduces an unbiased stratified statistic \(\bar{G}_{mst}\) and a new algorithm MSSG.
result MSSG algorithm outperforms other sgd-like algorithms in training deep models.
We apply stochastic average gradient (SAG) algorithms for training conditional random fields (CRFs). We describe a practical implementation that uses structure in the CRF gradient to reduce the memory requirement of this linearly-convergent stochastic gradient method, propose a non-uniform sampling scheme that substant…
Averaged SGD achieves optimal convergence rate for neural networks in the NTK regime.
problem Convergence analysis of averaged stochastic gradient descent for neural networks.
method Analyzed convergence of averaged stochastic gradient descent for overparameterized two-layer neural networks.
result Achieved minimax optimal convergence rate with global convergence guarantee.
Paper proposes a new approach for stochastic gradient descent in probabilistic modeling.
problem Finding optimal predictions in probabilistic models with large step sizes.
method Averaging moment parameters instead of natural parameters for constant-step-size stochastic gradient descent.
result Constant-step-size SGD can lead to better predictions in some cases and always converges in infinite-dimensional models.
This work analyzes mini-batching and averaging in SGD for least squares regression.
problem Improving parallelization and reducing variance in SGD for least squares regression.
method Characterizes mini-batching and tail-averaging for least squares regression, providing non-asymptotic excess risk bounds.
result Proves near-linear parallelization speedups and minimax risk for SGD with nearly the same number of updates as batch gradient descent.
New methods reduce variance in stochastic dual averaging for sparse solutions.
problem Regularized empirical risk minimization problems in machine learning.
method Stochastic dual averaging with variance reduction for sparser solutions.
result Achieve best known convergence rates for both strongly and non-strongly convex regularizers.
We improve SGD convergence on manifolds using averaging.
problem Minimizing functions on Riemannian manifolds with noisy gradients.
method Developed a geometric framework to transform SGD iterates into an averaged sequence with robust and fast convergence.
result Averaged SGD iterates converge at O(1/n) rate, improving on slow convergence of SGD. New averaging strategy achieves optimal convergence rate with high probability.
problem Optimizing convergence rate for strongly-convex functions.
method Simple non-uniform averaging strategy combined with Freedman's inequality.
result Achieves optimal O(1/T) convergence rate with high probability. Stochastic variational inference (SVI) lets us scale up Bayesian computation to massive data. It uses stochastic optimization to fit a variational distribution, following easy-to-compute noisy natural gradients. As with most traditional stochastic optimization methods, SVI takes precautions to use unbiased stochastic g…
The paper analyzes time-dependent streaming data with biased gradient estimates and proposes improved stochastic optimization methods.
problem Stochastic optimization in a streaming setting with time-dependent and biased gradient estimates.
method Analysis of several first-order methods including SGD, mini-batch SGD, and time-varying mini-batch SGD, along with their Polyak-Ruppert averages.
result Time-varying mini-batch SGD methods can break long- and short-range dependence structures, and biased SGD methods can achieve comparable performance to their unbiased counterparts.
STORM-PG uses momentum for faster policy gradient updates.
problem Improving policy gradient methods for reinforcement learning.
method Introduces STORM-PG, a SARAH-based algorithm with exponential moving average.
result Achieves O(1/ε3) sample complexity, matching best-known rate. Paper approximates risk measures using SGD with Langevin dynamics.
problem Approximating arbitrary law invariant risk measures.
method Stochastic Gradient Langevin Dynamics (SGD-Langevin) for general risk measures.
result Non-asymptotic convergence rates of the approximation algorithm.
Optimizes stochastic convex optimization with local minimax theory and adaptive methods.
problem Optimizing stochastic convex optimization problems with local complexity measures.
method Local minimax theory, adaptive fully online methods, Nesterov's dual averaging, Riemannian stochastic gradient methods.
result Achieves optimal convergence guarantees for stochastic convex optimization problems.
Paper analyzes stochastic gradient for PCA in streaming data.
problem Complexity analysis of stochastic gradient for PCA in online settings.
method Studied stochastic gradient algorithm with online learning rate selection.
result Practical relevance of plain stochastic gradient confirmed; learning rate improvement possible.
A new method improves stochastic gradient descent for faster and more efficient estimation.
problem Efficient and fast parametric estimation methods.
method Projected stochastic gradient descent corrected by Fisher scoring.
result The method is faster and more efficient than traditional methods.
Improved averaging method for noisy observations converges strongly.
problem Noisy observations from random dynamical systems require stable estimates.
method Introduced p-EMA, a modified exponential moving average with subharmonic weight decay. result Stochastic convergence guarantees for p-EMA under mild assumptions. New method for unbiased regression reduces excess risk.
problem Least squares regression with optimal solution and Hessian matrix.
method Averaged stochastic gradient descent with time-average estimator.
result Unbiased estimator with O(1/k) expected excess risk.
Study of accelerated dynamics for convex function minimization with noisy gradients.
problem Minimizing smooth convex functions with noisy gradients.
method Formulate and study continuous-time stochastic dynamics, prove convergence rates.
result Derive estimates of convergence rates for function values, both persistent and asymptotic.
SGP combines PushSum with stochastic gradient updates for robust distributed deep learning.
problem Synchronization issues in distributed deep learning.
method Stochastic Gradient Push (SGP) using PushSum for approximate distributed averaging.
result SGP converges to a stationary point at the same rate as SGD and achieves consensus.
Given a complex analytic function f on a Whitney stratified complex analytic variety of complex dimension n, whose real part Re(f) is Morse, we prove the existence of a stratified gradient-like vector field for Re(f) such that the unstable set of a critical point p on a stratum S of complex dimension s has real dimensi…
New method for zeroth-order stochastic gradient algorithms provides confidence intervals.
problem Lack of inferential capabilities for zeroth-order stochastic gradient algorithms.
method Established central limit theorem and provided online estimators for asymptotic covariance matrix.
result Asymptotically valid confidence sets for parameter estimation and prediction.
Stochastic Gradient Descent approximates Bayesian inference.
problem Approximating Bayesian posterior distributions efficiently.
method Constant SGD with parameter tuning and momentum.
result Constant SGD can approximate posterior distributions with minimized KL divergence.
TANGO optimizes models with small learning rates, converging to natural gradient.
problem Optimizing models with small learning rates to converge to natural gradient.
method TANGO, a simple algorithm that converges to natural gradient descent.
result TANGO achieves natural gradient descent with small learning rates.
New streaming methods improve convergence rates for optimization problems.
problem Optimizing large-scale, sequential data problems.
method Time-varying mini-batches and Polyak-Ruppert averaging for gradient-based algorithms.
result Time-varying mini-batches and averaging achieve optimal convergence and variance reduction.
A2SGD reduces distributed SGD communication to O(1) per worker.
problem Heavy communication costs in distributed SGD for large models.
method Two-level gradient averaging to consolidate gradients to two local averages.
result Achieves O(1) communication complexity per worker, significantly reducing traffic and training time.
K-AVG improves convergence for nonconvex optimization problems.
problem Improving the convergence of ASGD for nonconvex optimization.
method K-step averaging stochastic gradient descent (K-AVG) for nonconvex objectives.
result K-AVG converges faster and achieves better accuracies than ASGD.
Algorithm improves online canonical correlation analysis.
problem Online canonical correlation analysis.
method Stochastic Scaled-Gradient Descent (SSGD) for minimizing expectation over Riemannian manifolds.
result Achieved optimal one-time-scale algorithm with explicit rate of local asymptotic convergence.
Paper explores how combining tail-averaging and minibatching improves SGD convergence.
problem Understanding and optimizing learning properties of SGD variants.
method Least squares learning in a nonparametric setting, focusing on multiple passes, mini-batching, and averaging.
result Tail averaging allows faster convergence rates than uniform averaging in nonparametric settings.
Optimizes SGD for text classification using hyper-parameter tuning.
problem Improving performance of SGD in text classification.
method Fine-tuning hyper-parameters using Grid-Search approach.
result Optimized SGD classification leads to better accuracy and execution time.
A new Riemannian algorithm reduces variance in manifold optimization.
problem Optimizing functions on manifolds with stochastic gradient descent.
method Riemannian stochastic variance reduction with retraction and vector transport.
result The proposed algorithm outperforms standard methods on SPD and Grassmann manifolds.
This paper shows faster convergence rates for stochastic gradient descent in binary classification.
problem Achieving faster convergence rates for stochastic gradient descent in binary classification.
method Stochastic gradient descent and averaging variant, focusing on exponential convergence rates under strong low-noise conditions.
result Exponential convergence of the expected classification error in the final phase of stochastic gradient descent and averaged stochastic gradient descent for differentiable convex loss functions.
SGDM accelerates faster than SGD with large batch sizes and permits broader learning rates.
problem Understanding the role of momentum in SGDM and its convergence rates.
method Analysis of SGDM convergence rates under strongly convex settings, including finite-sample rates and asymptotic normality of the averaged estimator.
result SGDM converges faster than SGD with large batch sizes and permits broader learning rates.
A novel Riemannian extension of stochastic variance reduction for manifold optimization.
problem Optimization on the Grassmann manifold for large-scale problems.
method Riemannian stochastic variance reduced gradient (R-SVRG) on the Grassmann manifold.
result The proposed algorithm outperforms standard Riemannian SGD on various problems.
Paper develops methods for statistical inference in SGD with infinite variance.
problem Challenges in statistical inference for SGD with infinite variance.
method Model-agnostic methodology based on weak convergence and subsampling calibration.
result Asymptotically valid confidence regions for SGD in both finite and infinite variance regimes.
Stochastic conditional gradient methods improve optimization for convex and submodular functions.
problem Optimization of large-scale stochastic problems with high-dimensional constraints.
method Proposes averaging technique for gradient approximations and linear programming for descent/ascent directions.
result Achieves optimal or near-optimal guarantees for various submodular maximization problems.
StratPPI improves prediction-powered inference with stratified sampling.
problem Improving statistical estimates with limited human-labeled data.
method Combining small human-labeled data with large automatic-labeled data, stratifying data for tighter confidence intervals.
result StratPPI provides substantially tighter confidence intervals than unstratified approaches.