Stochastic gradient methods can converge in expectation under heavy-tailed noise.
problem Convergence of stochastic gradient methods under heavy-tailed noise.
method Comprehensive study of stochastic optimization under heavy-tailed noise for extsfSGD, extsfSMD, extsfASMD, extsfSGDM in convex and nonconvex optimization. result Established in-expectation convergence results for various stochastic gradient methods.
New adaptive and accelerated SGD methods achieve optimal convergence rates.
problem Optimizing convergence rates of stochastic gradient descent methods.
method Integrates diagonal scaling and momentum into accelerated SGD.
result Achieves optimal sampling and iteration complexity for smooth stochastic optimization.
Kalman Gradient Descent optimizes machine learning models by reducing variance in stochastic optimization.
problem Reducing variance in stochastic gradient descent to improve optimization performance.
method Uses Kalman filtering to adaptively reduce gradient variance in stochastic gradient descent.
result Improved performance on various machine learning tasks including neural networks and black box variational inference.
Proposes VSGD optimizer combining probabilistic and gradient-based methods.
problem Uncertainty modeling in deep neural networks.
method Combines probabilistic and gradient-based approaches using SVI.
result VSGD outperforms Adam and SGD on image classification tasks.
Unified framework connects stochastic optimization to Bayesian inference.
problem Stochastic optimization algorithms and their theoretical underpinnings.
method Latent variational problem and Forward Backward Stochastic Differential Equations (FBSDE).
result Recovery of various adaptive stochastic gradient descent methods.
Optimizes stochastic convex optimization with local minimax theory and adaptive methods.
problem Optimizing stochastic convex optimization problems with local complexity measures.
method Local minimax theory, adaptive fully online methods, Nesterov's dual averaging, Riemannian stochastic gradient methods.
result Achieves optimal convergence guarantees for stochastic convex optimization problems.
New method uses Chebyshev expansions to compute unbiased stochastic gradients for spectral functions.
problem Computing gradients of spectral functions is expensive and challenging.
method Combining randomized trace estimators with Chebyshev expansions for unbiased stochastic gradients.
result Developed methods for optimizing objectives involving spectral-sums with fast and stable convergence.
A new method for stochastic optimization using virtual gradients.
problem Stochastic optimization challenges in computational efficiency and memory usage.
method Inspired by dynamic programming, SVGD uses a computational graph and automatic differentiation for efficient optimization.
result Experimental results show SVGD outperforms other methods on multiple datasets and network models.
A new hybrid algorithm reduces stochastic gradient evaluations for nonconvex optimization.
problem Solving stochastic composite nonconvex optimization problems efficiently.
method Proposes a new hybrid variance-reduced proximal gradient method with a stochastic gradient estimator.
result Achieves optimal stochastic oracle complexity bound with one less gradient evaluation.
Optimal algorithms for online convex optimization with missing sub-gradient observations.
problem Online convex optimization with noisy or missing sub-gradient observations.
method Adaptive algorithms using sub-gradient descent with minimax optimal regret guarantees.
result Achieves tight minimax optimal regret bounds with empirical property estimation.
New adaptive step-size method for convex optimization without tuning.
problem Optimizing convex functions efficiently with stochastic gradients.
method Adapted Adaptive Gradient Descent Without Descent to stochastic setting.
result Stochastic gradient descent converges under various assumptions.
Paper analyzes SARAH for nonconvex optimization with mini-batches.
problem Solving nonconvex optimization problems with mini-batches.
method Stochastic Recursive Gradient Algorithm (SARAH) for nonconvex losses.
result Sublinear and linear convergence rates for different types of nonconvex functions.
Paper proposes a new optimizer for faster nonconvex optimization.
problem Optimizing nonconvex objectives efficiently and quickly.
method Integrates stochastic and biased gradient estimation with a hyper-parameter.
result The hyper-parameter can be configured to improve convergence rate.
New oracles improve stochastic optimization with noisy or biased measurements.
problem Optimizing functions with noisy or biased measurements.
method Introduced biased gradient oracles for stochastic optimization, analyzed RSG and SGD algorithms with these oracles.
result Derived non-asymptotic bounds for convergence rates of algorithms with biased gradient oracles.
Optimizes reinsurance and investment strategies to minimize ruin probability.
problem Optimizing reinsurance and investment strategies to minimize ruin probability.
method Stochastic projected gradient method based on Malliavin calculus.
result Effectiveness of the proposed method demonstrated through numerical experiments.
Improved complexity for machine learning optimization methods.
problem Optimizing over-parametrized models in machine learning.
method Stochastic conditional gradient methods with interpolation-like conditions.
result Improved oracle complexities for finding optimal solutions.
Unified approach for Bayesian optimal experiment design using stochastic gradients.
problem Designing optimal experiments in high-dimensional settings.
method Stochastic gradient ascent to optimize variational lower bounds on expected information gain.
result Unified approach outperforms existing methods in higher dimensions.
Proposes SME for ASGD, revealing dynamics and optimal mini-batching.
problem Understanding and optimizing ASGD algorithms.
method Develops SME for ASGD, proving convergence and solving optimal control problem.
result ASGD converges to SME in continuous time limit and predicts ASGD trajectories.
Optimizes stochastic and online optimization methods based on problem geometry.
problem Optimizing computational and statistical outcomes in stochastic and online optimization problems.
method Characterizes optimal methods based on constraint set and gradient geometry.
result Stochastic and adaptive-gradient methods are optimal for quadratically convex constraint sets.
This work refutes the conventional wisdom and shows acceleration can be made robust for least squares regression.
problem The challenge of using fast gradient methods for stochastic optimization due to instability and error accumulation.
method Introduced an accelerated stochastic gradient method for least squares regression.
result Proves accelerated stochastic gradient descent achieves minimax optimal statistical risk faster than SGD.
A new method for faster optimization of machine learning problems.
problem Minimization of composition of expected value functions.
method C-SAG, a novel extension of SAG for FS-CEVF problems.
result C-SAG achieves lower oracle query complexity per iteration than C-SVRG and converges faster.
Omega method mitigates noise in stochastic game optimization.
problem Noise sensitivity and convergence issues in stochastic game optimization.
method Omega method incorporates EMA of historic gradients in its update rule.
result Omega method outperforms optimistic gradient method in stochastic games.
The paper proposes gradient sparsification to reduce communication costs in distributed optimization.
problem Reduction of communication overhead in distributed machine learning.
method Formulates a convex optimization problem to minimize gradient coding length, and proposes simple algorithms for approximate solution.
result The proposed sparsification techniques significantly reduce communication costs without sacrificing accuracy.
Scout-Nd optimizes parameters of stochastic simulators efficiently.
problem Optimizing parameters of stochastic, computationally expensive simulators.
method Scout-Nd algorithm, reducing gradient noise, multi-fidelity schemes.
result Demonstrates better performance compared to existing methods.
SREDA optimizes complex machine learning problems with fewer evaluations.
problem Finding an optimal point in nonconvex-strongly-concave minimax problems.
method Stochastic Recursive Gradient Descent Ascent (SREDA) with variance reduction.
result Achieves optimal stochastic gradient complexity of O(κ^3ε^-3).
New method improves zeroth-order stochastic optimization with adaptive sampling.
problem Optimization problems without gradient information.
method Adaptive sampling quasi-Newton method using finite differences.
result Significant improvement in performance with adaptive sample sizes.
Probabilistic line search improves stochastic optimization efficiency.
problem Lack of direct line search methods for stochastic optimization.
method Combines deterministic line search structure with Bayesian optimization concepts.
result Effective removal of learning rate definition for SGD.
GeoAdaLer enhances geometric understanding of Adam for stochastic optimization.
problem Understanding geometric principles behind Adam's success in stochastic optimization.
method Introduces GeoAdaLer, an adaptive learning method based on geometric properties.
result Extends interpretability and effectiveness in complex optimization scenarios.
The paper introduces ADMM methods with variance reduction for nonconvex optimization.
problem Nonconvex optimization problems.
method Stochastic ADMM with variance reduction.
result Iteration complexity bound of O(1/ε) for obtaining an ε-stationary solution. Gradient descent optimizes deep ReLU networks with proper initialization.
problem Training deep neural networks with ReLU activation.
method Gradient descent and stochastic gradient descent with proper random weight initialization.
result Gradient descent finds global minima for over-parameterized deep ReLU networks.
Improved SGD algorithm with faster convergence.
problem Optimization of machine learning models.
method Conditional accelerated lazy stochastic gradient descent.
result Convergence rate of $O\left(\frac{1}{\varepsilon^2}
ight)$, faster than previous methods.
New algorithm reduces variance in nonconvex optimization problems.
problem Finite-sum nonconvex optimization problems.
method Stochastic gradient descent with nested variance reduction.
result Converges to an ε-stationary point with improved complexity.
New algorithms improve SGD's efficiency in convex and nonconvex optimization.
problem Optimizing gradient size in stochastic optimization.
method Designing SGD3 for convex objectives and SGD5 for nonconvex objectives.
result Near-optimal rates for gradient size reduction in both convex and nonconvex settings.
Improves deep learning optimization with a new stochastic gradient method.
problem Noisy and sparse gradients in deep learning optimization.
method Proposes a family of double adaptive stochastic gradient methods (DASGrad).
result Analyzes theoretical convergence improvements and empirical validation.
Paper analyzes SGHMC for non-convex optimization with discontinuous gradients.
problem Training neural networks with ReLU activation.
method Non-asymptotic convergence analysis of SGHMC with discontinuous gradients.
result Explicit upper bounds for expected excess risk in non-convex optimization.
Stochastic multi-gradient method tackles MOO problems with uncertain data.
problem Optimizing conflicting functions in uncertain or unknown data.
method Stochastic multi-gradient (SMG) method, solving quadratic subproblems at each iteration.
result Rates to compute points in the Pareto front, similar to stochastic gradient methods.
Improved convergence for nonconvex optimization with dependent data.
problem Constrained smooth nonconvex optimization with dependent data.
method Stochastic projected gradient methods under a general dependent data sampling scheme.
result Achieved worst-case rate of convergence ildeO(t−1/4) and complexity ildeO(ε−4). New unbiased gradient estimators for complex optimization problems.
problem Unbiased and variance-limited gradient estimation for conditional stochastic optimization.
method Developed multilevel Monte Carlo gradient estimators for conditional stochastic optimization problems.
result Unbiased and finite variance gradient estimators for conditional stochastic optimization problems.
Optimal rates found for learning with Nyström stochastic gradient methods.
problem Nonparametric regression learning with improved computational efficiency.
method Combination of stochastic gradient methods with Nyström subsampling, allowing multiple passes and mini-batches.
result Derivation of optimal learning rates considering various parameters.
New algorithm optimally minimizes convex functions with noisy gradients.
problem Minimizing strongly convex, smooth functions with noisy gradient estimates.
method A multistage accelerated stochastic gradient method with restarts.
result Achieves optimal convergence rate in deterministic and stochastic cases.
Paper proposes faster method to find local minima in nonconvex optimization.
problem Escaping saddle points and finding local minima in nonconvex optimization.
method LENA (Last stEp shriNkAge) framework for faster perturbed stochastic gradient methods.
result LENA finds (ε,εH)-approximate local minima within ildeO(ε−3+εH−6) evaluations. Paper explores using action-value gradients for policy improvement in off-policy actor-critic methods.
problem Improving policies using action-value gradients in off-policy stochastic actor-critic methods.
method Discusses and analyzes the use of action-value gradients for policy improvement, and proposes an incremental approach.
result Demonstrates the feasibility and incremental approach for following the policy gradient.
Simplified proof shows SGD optimality for least squares.
problem Optimizing SGD for least squares efficiency.
method Analyzing SGD as a stochastic process, characterizing stationary covariance matrix.
result Statistical minimax optimality of SGD for least squares.
Lower bounds on queries needed for finding stationary points in non-convex optimization.
problem Finding ε-stationary points in non-convex stochastic optimization. method Proving lower bounds on the number of queries required by stochastic first-order methods.
result Lower bounds on the number of queries required to find ε-stationary points are tight and optimal. We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures. Roughly, the n…
FGD reduces noisy gradient variance in SGD for neural networks.
problem Noisy and unreliable gradient estimation in SGD for deep learning.
method Solves an adaptive filtering problem to consistently estimate the local gradient.
result Significantly reduces gradient variance and accelerates convergence.
Optimal learning rates achieved with multi-pass stochastic gradient methods.
problem Improving learning rates in stochastic gradient methods with multiple passes and mini-batches.
method Analysis of learning properties with multiple passes and mini-batches, considering step-size, number of passes, and mini-batch size.
result Optimal finite sample bounds can be achieved by early-stopping and larger step-sizes are allowed with mini-batches.
This paper analyzes adaptive gradient algorithms for better performance in ill-conditioned problems.
problem Poor performance of standard stochastic gradient algorithms in ill-conditioned problems.
method Non-asymptotic analysis of adaptive gradient algorithms (Adagrad and Stochastic Newton) for strongly convex objectives.
result Theoretical analysis and adaptation to practical applications like linear regression and regularized GLM.