This paper establishes lower bounds for smooth nonconvex finite-sum optimization.
problem Understanding the complexity of finding optimal solutions in nonconvex finite-sum optimization.
method Proving tight lower bounds for the complexity of finding ε-suboptimal points and ε-approximate stationary points.
result Existing algorithms achieve optimal IFO complexity up to logarithmic factors.
A new method speeds up optimization for sums, simpler than others.
problem Optimizing finite sums, especially in non-smooth cases.
method A novel accelerated method based on SAGA, with one parameter.
result Achieves accelerated convergence on strongly convex smooth problems.
New methods optimize sums of bivariate functions on finite domains.
problem Optimizing functions with multiple arguments that are sums of bivariate functions.
method Measure-valued extensions, ℓ 2 \ell^2 ℓ 2 -approximation, entropy-regularization, linear programming, coordinate ascent. result Tractable problem formulations solvable with various methods.
New lower bounds for gradient methods in strongly convex finite-sum optimization.
problem Developing tight lower bounds for randomized gradient methods in finite-sum optimization.
method Deriving tight lower complexity bounds for SAG, SAGA, SVRG, SARAH, and related methods.
result Tight matches between lower bounds and upper bounds for various methods under specific conditions.
New findings on optimizing finite sum problems with variance reduction and acceleration.
problem Conditions for efficient variance reduction and acceleration in finite sum optimization.
method Analysis of first-order and coordinate-descent finite sum algorithms.
result Optimal complexity bounds for minimizing L-smooth and convex finite sums.
Paper introduces variance reduction for infinite datasets with finite-sum structure.
problem Optimizing composite and strongly convex objectives with stochastic perturbations.
method Variance reduction approach for stochastic optimization with composite and strongly convex objectives.
result Convergence rate outperforms SGD with a smaller constant factor.
Paper establishes lower bounds for finite-sum optimization problems using novel construction methods.
problem Lower complexity bounds for finite-sum optimization problems with various component functions.
method Developed novel approach to construct hard instances and analyzed PIFO algorithms.
result Established lower complexity bounds for convex-concave and nonconvex-strongly-concave objectives.
Lower bounds for higher-order methods in non-convex optimization.
problem Proving lower bounds for higher-order methods in smooth non-convex finite-sum optimization.
method Analyzing deterministic and randomized algorithms, proposing a new smoothness assumption.
result Proves optimal lower bounds for simulating pth-order regularized methods on the whole function.
Paper analyzes SARAH algorithm for smooth finite-sum optimization, achieving optimal complexity.
problem Optimizing smooth finite-sum nonconvex objectives efficiently.
method Modified SARAH algorithm for convergence analysis and practical implementation.
result Achieves optimal complexity matching lower-bound for nonconvex problems.
DESTRESS optimizes decentralized nonconvex optimization with optimal IFO complexity and efficient communication.
problem Decentralized nonconvex finite-sum optimization in multi-agent systems.
method DESTRESS uses stochastic recursive gradient updates, gradient tracking, and careful hyper-parameter choices to achieve optimal IFO complexity with efficient communication.
result DESTRESS matches the optimal IFO complexity of centralized algorithms while maintaining communication efficiency.
Paper proposes a faster SPIDER-EM variant for large-scale nonconvex optimization.
problem High computational cost of EM algorithm in large-scale learning.
method Extension of SPIDER-EM for nonconvex finite-sum optimization problems.
result Achieves state-of-the-art complexity bounds and linear convergence under certain conditions.
Two algorithms find local minima faster in finite-sum and general stochastic optimization.
problem Finding local minima in finite-sum and general stochastic nonconvex optimization.
method Stochastic Nested Variance Reduction (SNVRG) + Neon2.
result Achieves better gradient complexity for convergence to ( ε , ε H ) (ε, ε_H) ( ε , ε H ) -second-order stationary points. Second-order methods for finite-sum problems are not more efficient in worst-case scenarios.
problem Optimizing finite-sum problems using second-order methods.
method Analysis of second-order methods for finite-sum problems, focusing on worst-case complexity.
result Second-order methods do not offer significant efficiency gains in worst-case scenarios.
Novel analysis of EFP for finite-sum problems in neural networks.
problem Optimization of two-layer neural networks in the mean-field regime.
method Primal-dual analysis of entropic fictitious play (EFP) for finite-sum problems.
result Established global convergence guarantees for EFP dynamics.
Katyusha accelerates stochastic gradient methods for convex finite-sum problems.
problem Counterexamples prevent Nesterov's momentum from accelerating stochastic gradient methods.
method Introduces Katyusha, a direct, primal-only stochastic gradient method.
result Katyusha achieves optimal accelerated convergence rate in convex finite-sum stochastic optimization.
This paper presents a lower bound for optimizing a finite sum of n n n functions, where each function is L L L -smooth and the sum is μ μ μ -strongly convex. We show that no algorithm can reach an error ε ε ε in minimizing all functions from this class in fewer than Ω ( n + n ( κ − 1 ) log ( 1 / ε ) ) Ω(n + \sqrt{n(κ-1)}\log(1/ε)) Ω ( n + n ( κ − 1 ) log ( 1/ ε )) iterations, where κ = L / μ κ=L/μ κ = L / μ is a …
New method reduces complexity of minimizing convex finite sums without needing individual function indices.
problem Minimizing convex finite sums efficiently without knowing which function is being addressed.
method Exploits finite noise structure to derive upper bounds and proposes a novel SVRG adaptation.
result Achieves optimal complexity bounds of O(n^2) and matches existing lower bounds.
Optimal SGD rates achieved with shuffling, covering non-convex and convex cases.
problem Optimizing finite-sum optimization problems with shuffling strategies.
method RandomShuffle and SingleShuffle algorithms for SGD, analyzing convergence rates.
result Minimax optimal convergence rates established, generalizing to non-convex costs.
Accelerates optimization in asynchronous systems with sparse updates.
problem Optimizing finite-sum objectives in asynchronous lock-free environments.
method New accelerated SVRG variant with sparse updates.
result Achieves optimal incremental gradient complexity.
Two new Frank-Wolfe algorithms improve convergence for constrained optimization.
problem Solving optimization problems with structured constraints in machine learning.
method Two new variants of the Frank-Wolfe (FW) method for stochastic finite-sum minimization.
result Best convergence guarantees for convex and non-convex objective functions.
New algorithm reduces complexity for distributed and stochastic optimization.
problem Optimization of distributed and stochastic problems with multiple agents.
method Random Gradient Extrapolation Method (RGEM)
result Achieves optimal complexity bounds for both distributed and stochastic optimization.
Katyusha X adds momentum to SVRG for faster non-convex optimization.
problem Minimizing sum-of-nonconvex functions in machine learning.
method Adding momentum to SVRG method.
result Provable accelerated stochastic algorithm for sum-of-nonconvex functions.
Freya PAGE optimizes nonconvex optimization with heterogeneous, asynchronous workers.
problem Optimizing nonconvex finite-sum problems with varying worker processing times.
method Freya PAGE, a parallel method robust to stragglers and adaptive to slow computations.
result Freya PAGE offers improved time complexity guarantees compared to previous methods.
SVRN accelerates Newton methods by reducing variance and improving performance.
problem Improving the efficiency of Newton methods for large-scale optimization problems.
method Stochastic Variance-Reduced Newton (SVRN) algorithm that accelerates Subsampled Newton and Iterative Hessian Sketch algorithms.
result SVRN accelerates Newton methods by reducing the number of passes over the data, achieving a significant improvement in performance.
Characterizes compact complex surfaces with finite homotopy rank-sum.
problem Compact complex surfaces with finite homotopy rank-sum.
method Characterization and proof of Steinness of universal cover.
result Smooth compact complex Kaehler surfaces with finite homotopy rank-sum.
A new method for faster optimization of machine learning problems.
problem Minimization of composition of expected value functions.
method C-SAG, a novel extension of SAG for FS-CEVF problems.
result C-SAG achieves lower oracle query complexity per iteration than C-SVRG and converges faster.
Study on Nesterov's method in stochastic settings, revealing divergence under certain conditions.
problem Understanding Nesterov's method in stochastic settings, especially finite-sum.
method Analysis of Nesterov's accelerated gradient method in stochastic and finite-sum settings.
result Nesterov's method may diverge in finite-sum settings without additional conditions.
New methods optimize machine learning models without sharing data.
problem Training machine learning models with distributed data.
method Decentralized stochastic optimization with gradient tracking and variance reduction.
result Improved algorithms for training machine learning models without data sharing.
Unified framework for stability and generalization of Push-Sum in decentralized learning over directed graphs.
problem Understanding stability and generalization of Push-Sum in decentralized learning over directed networks.
method Developed a unified uniform-stability framework for SGP algorithm, incorporating imbalance-aware consistency bounds.
result Established finite-iteration stability and optimization guarantees for convex and non-convex objectives.
New corks found with Stein structures and hyperbolic boundaries.
problem Existence of boundary-sum irreducible finite order corks with Stein structures.
method Proved for any positive integer n, constructed corks with Stein structures and verified hyperbolic boundaries.
result Found boundary-sum irreducible Z n {\mathbb Z}_n Z n -corks with Stein structures and verified hyperbolic boundaries. Characterizes Stein surfaces with finite homotopy rank-sum.
problem Finite homotopy rank-sum in Stein spaces.
method Rational homotopy theory, classification of Stein surfaces.
result Affine Stein surfaces with finite fundamental group are either simply connected or of order 2.
Paper develops momentum schemes with variance reduction for non-convex composition optimization.
problem Lack of convergence guarantee and efficient momentum design in existing algorithms.
method Develops various momentum schemes with SPIDER-based variance reduction.
result Achieves near-optimal sample complexity and linear convergence rate.
New nonconvex Frank-Wolfe methods for faster optimization.
problem Nonconvex optimization problems in machine learning.
method Stochastic Frank-Wolfe methods for nonconvex optimization.
result Improved convergence rates for nonconvex optimization.
We propose an optimization method for minimizing the finite sums of smooth convex functions. Our method incorporates an accelerated gradient descent (AGD) and a stochastic variance reduction gradient (SVRG) in a mini-batch setting. Unlike SVRG, our method can be directly applied to non-strongly and strongly convex prob…
Doubly SGD improves convergence for intractable objective optimization.
problem Optimizing objectives in sum of intractable expectations.
method Doubly SGD with doubly stochastic gradients and independent minibatching.
result Established convergence of doubly SGD under general conditions, including dependent component gradient estimators.
Genus 3 Heegaard groups of lens space connected sums are finitely generated.
problem Understanding the structure of mapping class groups of genus 3 Heegaard splittings.
method Proved finitely generated property through connected reducing sphere complexes.
result Mapping class groups are finitely generated and complexes are connected.
Improved variance reduction for Riemannian non-convex optimization with adaptive batch size.
problem Optimizing non-convex functions on Riemannian manifolds.
method Batch size adaptation in R-SVRG, R-SRG, and R-SPIDER.
result Achieves lower total complexities for various non-convex functions.
PAGE is a simple gradient estimator for nonconvex optimization problems.
problem Nonconvex optimization problems in machine learning.
method PAGE is a probabilistic gradient estimator that uses vanilla SGD with probability and a small adjustment with probability 1-p.
result PAGE achieves optimal convergence rates for nonconvex finite-sum and online problems.
Improved SVRC algorithm reduces complexity for nonconvex optimization.
problem Finding local minima for nonconvex finite-sum optimization with improved complexity.
method Stochastic Recursive Variance-Reduced Cubic regularization (SRVRC) using recursively updated semi-stochastic gradient and Hessian estimators.
result SRVRC achieves improved gradient and Hessian complexities to find ( ε , ε ) (ε, \sqrtε) ( ε , ε ) -approximate local minimum. New method for optimizing complex composite functions with reduced variance.
problem Optimizing multi-level composite functions with nested random and smooth mappings.
method Normalized proximal approximate gradient (NPAG) method with nested stochastic variance reduction.
result Total sample complexity of O ( ε − 3 ) O(ε^{-3}) O ( ε − 3 ) in expectation and O ( N + N ε − 2 ) O(N+\sqrt{N}ε^{-2}) O ( N + N ε − 2 ) in finite-sum cases. We propose an explicit recursive method to approximate a power-law with a finite sum of weighted exponentials. Applications to moving averages with long memory are discussed in relationship with stochastic volatility models.
Bayesian optimization for expensive integrands achieves optimal performance.
problem Optimizing functions with expensive integrands in noisy conditions.
method Bayesian optimization with discretization-free value of information optimization.
result Achieves optimal performance in noisy and smooth conditions.
The study analyzes and compares variance reduction methods for stochastic gradient Monte Carlo.
problem Improving the efficiency of Monte Carlo methods for complex log-posterior distributions.
method The study analyzes and compares SAGA Langevin diffusion, SVRG Langevin diffusion, and control-variate underdamped Langevin diffusion under uniform assumptions.
result Sharp theoretical bounds identifying the performance of each method under different conditions.
A new algorithm improves convergence rates for convex optimization problems.
problem Convex optimization problems with finite-sum structure.
method Nesterov Accelerated Shuffling Gradient (NASG) integrating Nesterov's acceleration with different shuffling schemes.
result Improved convergence rate of O(1/T) for unified shuffling schemes.
New algorithms solve non-convex optimization problems efficiently.
problem Non-convex stochastic compositional optimization problems.
method Developed two stochastic Gauss-Newton algorithms.
result Established global oracle complexity for stochastic Gauss-Newton methods.
Study on Newton-Sketch and Subsampled Newton methods for large-scale optimization.
problem Optimization of large-scale finite-sum problems with high-dimensional data.
method Hessian subsampling and randomized Hadamard transformations for dimensionality reduction in Newton's method.
result Advantages of conjugate gradient vs. stochastic gradient iterations revealed through experiments.
Paper develops probabilistic bounds for a stochastic gradient algorithm in non-convex problems.
problem Stochastic optimization in non-convex finite sum problems.
method Develops a new dimension-free Azuma-Hoeffding type bound for a martingale difference sequence.
result Empirical results show superior probabilistic performance of Prob-SARAH compared to other algorithms.
We propose the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG me…