The height function of various surfaces decomposes into finite sums of scaled and translated versions of itself.
problem Decomposing the height function of different types of surfaces into simpler components.
method Using Euler-Ramanujan identities and Weierstrass-Enneper representation to decompose height functions of minimal, maximal, timelike minimal, and Born-Infeld surfaces.
result The height function of various surfaces can be expressed as a finite sum of scaled and translated versions of itself.
A new method speeds up optimization for sums, simpler than others.
problem Optimizing finite sums, especially in non-smooth cases.
method A novel accelerated method based on SAGA, with one parameter.
result Achieves accelerated convergence on strongly convex smooth problems.
New method reduces complexity of minimizing convex finite sums without needing individual function indices.
problem Minimizing convex finite sums efficiently without knowing which function is being addressed.
method Exploits finite noise structure to derive upper bounds and proposes a novel SVRG adaptation.
result Achieves optimal complexity bounds of O(n^2) and matches existing lower bounds.
Paper introduces variance reduction for infinite datasets with finite-sum structure.
problem Optimizing composite and strongly convex objectives with stochastic perturbations.
method Variance reduction approach for stochastic optimization with composite and strongly convex objectives.
result Convergence rate outperforms SGD with a smaller constant factor.
New findings on optimizing finite sum problems with variance reduction and acceleration.
problem Conditions for efficient variance reduction and acceleration in finite sum optimization.
method Analysis of first-order and coordinate-descent finite sum algorithms.
result Optimal complexity bounds for minimizing L-smooth and convex finite sums.
SVRN accelerates Newton methods by reducing variance and improving performance.
problem Improving the efficiency of Newton methods for large-scale optimization problems.
method Stochastic Variance-Reduced Newton (SVRN) algorithm that accelerates Subsampled Newton and Iterative Hessian Sketch algorithms.
result SVRN accelerates Newton methods by reducing the number of passes over the data, achieving a significant improvement in performance.
We propose an optimization method for minimizing the finite sums of smooth convex functions. Our method incorporates an accelerated gradient descent (AGD) and a stochastic variance reduction gradient (SVRG) in a mini-batch setting. Unlike SVRG, our method can be directly applied to non-strongly and strongly convex prob…
Paper establishes lower bounds for finite-sum optimization problems using novel construction methods.
problem Lower complexity bounds for finite-sum optimization problems with various component functions.
method Developed novel approach to construct hard instances and analyzed PIFO algorithms.
result Established lower complexity bounds for convex-concave and nonconvex-strongly-concave objectives.
Novel Newton method for large-scale kernel methods using random features.
problem Efficiently solving large-scale finite-sum minimization problems in RKHS.
method Randomized feature-based Newton method for empirical risk minimization.
result Local superlinear and global linear convergence of the method.
Novel analysis of EFP for finite-sum problems in neural networks.
problem Optimization of two-layer neural networks in the mean-field regime.
method Primal-dual analysis of entropic fictitious play (EFP) for finite-sum problems.
result Established global convergence guarantees for EFP dynamics.
Two new Frank-Wolfe algorithms improve convergence for constrained optimization.
problem Solving optimization problems with structured constraints in machine learning.
method Two new variants of the Frank-Wolfe (FW) method for stochastic finite-sum minimization.
result Best convergence guarantees for convex and non-convex objective functions.
A new method for faster optimization of machine learning problems.
problem Minimization of composition of expected value functions.
method C-SAG, a novel extension of SAG for FS-CEVF problems.
result C-SAG achieves lower oracle query complexity per iteration than C-SVRG and converges faster.
New methods optimize machine learning models without sharing data.
problem Training machine learning models with distributed data.
method Decentralized stochastic optimization with gradient tracking and variance reduction.
result Improved algorithms for training machine learning models without data sharing.
Paper accelerates L-BFGS with second-order info and stochastic batches.
problem Optimizing finite-sum minimization problems.
method Proposes L-BFGS with smooth gradient difference estimates and well-scaling initial Hessians.
result Achieves acceleration in both convex and nonconvex cases.
New method reduces complexity for nonconvex optimization problems.
problem Minimizing composite functions with random or finite sum inner mappings.
method Stochastic composite gradient method with incremental variance reduction.
result Achieves complexity similar to best first-order methods for expected-value and finite-sum nonconvex functions.
SARAH uses stochastic recursive gradient for efficient machine learning.
problem Finite-sum minimization problems in machine learning.
method SARAH and SARAH+ algorithms with recursive gradient estimates.
result Linear convergence rate proven under strong convexity assumption.
DESTRESS optimizes decentralized nonconvex optimization with optimal IFO complexity and efficient communication.
problem Decentralized nonconvex finite-sum optimization in multi-agent systems.
method DESTRESS uses stochastic recursive gradient updates, gradient tracking, and careful hyper-parameter choices to achieve optimal IFO complexity with efficient communication.
result DESTRESS matches the optimal IFO complexity of centralized algorithms while maintaining communication efficiency.
SignSVRG improves SignSGD by reducing variance, achieving similar convergence rates.
problem Minimizing finite sums of convex and Lipschitz functions.
method Incorporates variance reduction techniques into SignSGD.
result Achieves convergence rates of O ( 1 / T ) \mathcal{O}(1 / \sqrt{T}) O ( 1/ T ) for expected norm of the gradient and O ( 1 / T ) \mathcal{O}(1/T) O ( 1/ T ) for smooth convex functions. A new algorithm improves convergence rates for convex optimization problems.
problem Convex optimization problems with finite-sum structure.
method Nesterov Accelerated Shuffling Gradient (NASG) integrating Nesterov's acceleration with different shuffling schemes.
result Improved convergence rate of O(1/T) for unified shuffling schemes.
A new algorithm speeds up machine learning by solving large-scale problems more efficiently.
problem Efficiently solving large-scale machine learning problems with regularization.
method Subsampled proximal Newton-type method that leverages finite sum structure and recent stochastic first-order methods.
result The method achieves faster convergence than state-of-the-art methods for non-smooth regularizers.
This paper establishes lower bounds for smooth nonconvex finite-sum optimization.
problem Understanding the complexity of finding optimal solutions in nonconvex finite-sum optimization.
method Proving tight lower bounds for the complexity of finding ε-suboptimal points and ε-approximate stationary points.
result Existing algorithms achieve optimal IFO complexity up to logarithmic factors.
The paper analyzes the variance of different shuffling methods in stochastic gradient descent.
problem Understanding the variance of different shuffling methods in stochastic gradient descent.
method Power spectral density analysis to study the noise sequences of stochastic gradients.
result The stationary variances of iterates decrease in the order of SGD, SGD-RR, and SGD-SO.
Unified framework for decentralized optimization combining gradient tracking and variance reduction.
problem Solving finite-sum minimization problems in distributed systems with privacy and resource constraints.
method Unified algorithmic framework combining variance-reduction and gradient tracking.
result Unified methods achieve robust performance and fast convergence for smooth and strongly-convex objectives, and are applicable to non-convex problems.
Stochastic heavy ball method achieves linear convergence for general loss minimization.
problem Minimizing generalization error in machine learning models.
method SGD steps with heavy ball momentum, focusing on expected loss, not finite-sum minimization.
result Established the first linear convergence result for the stochastic heavy ball method.
New lower bounds for gradient methods in strongly convex finite-sum optimization.
problem Developing tight lower bounds for randomized gradient methods in finite-sum optimization.
method Deriving tight lower complexity bounds for SAG, SAGA, SVRG, SARAH, and related methods.
result Tight matches between lower bounds and upper bounds for various methods under specific conditions.
Freya PAGE optimizes nonconvex optimization with heterogeneous, asynchronous workers.
problem Optimizing nonconvex finite-sum problems with varying worker processing times.
method Freya PAGE, a parallel method robust to stragglers and adaptive to slow computations.
result Freya PAGE offers improved time complexity guarantees compared to previous methods.
New optimization method speeds up learning from data.
problem Efficiently optimizing large datasets for machine learning.
method Minibatch stochastic variance reduced proximal iterations.
result Improved convergence speed for quadratic objectives.
New algorithms improve ICA performance without manual tuning.
problem Improving Independent Component Analysis (ICA) performance.
method Developed majorization-minimization framework for non-convex loss function.
result Stochastic algorithms guarantee loss function decrease at each iteration.
This paper presents a lower bound for optimizing a finite sum of n n n functions, where each function is L L L -smooth and the sum is μ μ μ -strongly convex. We show that no algorithm can reach an error ε ε ε in minimizing all functions from this class in fewer than Ω ( n + n ( κ − 1 ) log ( 1 / ε ) ) Ω(n + \sqrt{n(κ-1)}\log(1/ε)) Ω ( n + n ( κ − 1 ) log ( 1/ ε )) iterations, where κ = L / μ κ=L/μ κ = L / μ is a …
Second-order methods for finite-sum problems are not more efficient in worst-case scenarios.
problem Optimizing finite-sum problems using second-order methods.
method Analysis of second-order methods for finite-sum problems, focusing on worst-case complexity.
result Second-order methods do not offer significant efficiency gains in worst-case scenarios.
Lower bounds for higher-order methods in non-convex optimization.
problem Proving lower bounds for higher-order methods in smooth non-convex finite-sum optimization.
method Analyzing deterministic and randomized algorithms, proposing a new smoothness assumption.
result Proves optimal lower bounds for simulating pth-order regularized methods on the whole function.
Study on Nesterov's method in stochastic settings, revealing divergence under certain conditions.
problem Understanding Nesterov's method in stochastic settings, especially finite-sum.
method Analysis of Nesterov's accelerated gradient method in stochastic and finite-sum settings.
result Nesterov's method may diverge in finite-sum settings without additional conditions.
Katyusha accelerates stochastic gradient methods for convex finite-sum problems.
problem Counterexamples prevent Nesterov's momentum from accelerating stochastic gradient methods.
method Introduces Katyusha, a direct, primal-only stochastic gradient method.
result Katyusha achieves optimal accelerated convergence rate in convex finite-sum stochastic optimization.
Paper proposes a faster SPIDER-EM variant for large-scale nonconvex optimization.
problem High computational cost of EM algorithm in large-scale learning.
method Extension of SPIDER-EM for nonconvex finite-sum optimization problems.
result Achieves state-of-the-art complexity bounds and linear convergence under certain conditions.
Paper avoids strict saddles in stochastic optimization without unit excitation assumption.
problem Avoiding strict saddles in stochastic optimization without unit excitation assumption.
method Pathwise Lyapunov-Perron framework, local smoothness, finite-moment assumptions.
result Avoidance of strict saddles for stochastic mirror descent and proximal-type methods.
New algorithms find near-stationary points in convex optimization.
problem Finding near-stationary points in convex optimization.
method Memory-saving variant of OGM-G, accelerated SVRG, adaptively regularized accelerated SVRG.
result Schemes achieve fast rates for minimizing gradient norm and function value.
Recent advances in optimization theory have shown that smooth strongly convex finite sums can be minimized faster than by treating them as a black box "batch" problem. In this work we introduce a new method in this class with a theoretical convergence rate four times faster than existing methods, for sums with sufficie…
Paper analyzes SARAH algorithm for smooth finite-sum optimization, achieving optimal complexity.
problem Optimizing smooth finite-sum nonconvex objectives efficiently.
method Modified SARAH algorithm for convergence analysis and practical implementation.
result Achieves optimal complexity matching lower-bound for nonconvex problems.
SAGA algorithm improved for arbitrary sampling and minibatching.
problem Minimizing the average of a large number of smooth functions.
method Proposes a general and flexible variant of SAGA with arbitrary sampling.
result Establishes linear convergence rates under various conditions.
New method solves root-finding problems with faster convergence.
problem Finite-sum co-coercive equations
method Variance-reduced Krasnoselkii--Mann methods
result Achieves both O ( 1 / k 2 ) \mathcal{O}(1/k^2) O ( 1/ k 2 ) and o ( 1 / k 2 ) o(1/k^2) o ( 1/ k 2 ) convergence rates Katyusha X adds momentum to SVRG for faster non-convex optimization.
problem Minimizing sum-of-nonconvex functions in machine learning.
method Adding momentum to SVRG method.
result Provable accelerated stochastic algorithm for sum-of-nonconvex functions.
New algorithm finds local minima in non-convex problems efficiently.
problem Finding local minima in non-convex finite-sum minimization problems.
method Stochastic Trust Region (STR) algorithm combining inexact gradient and Hessian estimation.
result STR finds ( ε , ε ) (ε, \sqrtε) ( ε , ε ) -approximate local minimum with improved efficiency. Two algorithms find local minima faster in finite-sum and general stochastic optimization.
problem Finding local minima in finite-sum and general stochastic nonconvex optimization.
method Stochastic Nested Variance Reduction (SNVRG) + Neon2.
result Achieves better gradient complexity for convergence to ( ε , ε H ) (ε, ε_H) ( ε , ε H ) -second-order stationary points. Optimal SGD rates achieved with shuffling, covering non-convex and convex cases.
problem Optimizing finite-sum optimization problems with shuffling strategies.
method RandomShuffle and SingleShuffle algorithms for SGD, analyzing convergence rates.
result Minimax optimal convergence rates established, generalizing to non-convex costs.
Study co-dimension one area-minimizing currents with tangentially immersed boundaries and co-oriented mean curvature.
problem Understanding area-minimizing currents with specific boundary conditions.
method Introducing and studying co-dimension one area-minimizing currents with tangentially immersed boundaries and co-oriented mean curvature.
result Any such currents are supported in a smooth hypersurface near the boundary, with tangent cones being hyperplanes of constant orientation but non-constant multiplicity.
Stochastic Gradient Descent underperforms on some problems, contrary to expectations.
problem Understanding the generalization performance of SGD on specific problem instances.
method Analysis of stochastic convex optimization framework, proving empirical and generalization gaps for SGD.
result SGD exhibits both empirical risk and generalization gap of Ω ( 1 ) Ω(1) Ω ( 1 ) on some problem instances, contradicting its conventional understanding. New algorithm reduces distributed optimization time with stochastic delays.
problem Optimizing distributed data with stochastic delays.
method Developed ADSAGA, a variant of SAGA for distributed-data settings with stochastic delays.
result ADSAGA converges in $ ilde{O}\left(\left(n + \sqrt{m}κ
ight)\log(1/ε)
ight)$ iterations under mean delay m m m . Accelerates optimization in asynchronous systems with sparse updates.
problem Optimizing finite-sum objectives in asynchronous lock-free environments.
method New accelerated SVRG variant with sparse updates.
result Achieves optimal incremental gradient complexity.