Improved SVRG for quadratic functions achieves better performance and running times.
problem Minimizing quadratic functions with a specific type of Hessian matrix.
method Variant of SVRG algorithm for quadratic functions with improved analysis.
result Improved performance and running times for quadratic functions compared to state-of-the-art methods.
New algorithm for large-scale nonsmooth convex optimization with robust convergence.
problem Minimizing the average of many nonsmooth and convex functions in machine learning.
method Developed a new algorithm called Randomized Smoothing SVRG that achieves robust linear convergence.
result Achieves robust linear convergence rate and superior time and gradient complexity compared to state-of-the-art methods.
In this paper, we propose a simple variant of the original stochastic variance reduction gradient (SVRG), where hereafter we refer to as the variance reduced stochastic gradient descent (VR-SGD). Different from the choices of the snapshot point and starting point in SVRG and its proximal variant, Prox-SVRG, the two vec…
We provide tight upper and lower bounds on the complexity of minimizing the average of m convex functions using gradient and prox oracles of the component functions. We show a significant gap between the complexity of deterministic vs randomized optimization. For smooth functions, we show that accelerated gradient de…
This paper resolves convergence issues for variance-reduced algorithms under random reshuffling.
problem Proving linear convergence of variance-reduced algorithms under random reshuffling.
method Develops a new amortized variance-reduced gradient (AVRG) algorithm and resolves convergence for SAGA under random reshuffling.
result AVRG algorithm converges linearly under random reshuffling with constant storage requirements.
VR-SGD is a simple method for machine learning that uses larger learning rates and averages.
problem Efficiently solving machine learning problems with large datasets.
method A simple variant of SVRG with specific averaging and update rules.
result VR-SGD achieves linear convergence for strongly convex problems and similar performance to momentum methods for non-strongly convex problems.
Improved zeroth-order algorithms for nonconvex optimization with reduced complexity and improved performance.
problem Designing efficient zeroth-order algorithms for nonconvex optimization with reduced function query complexities and improved convergence rates.
method Proposed new algorithms ZO-SVRG-Coord-Rand and ZO-SPIDER-Coord, developed new analyses, and addressed issues of function query complexities and stepsize generation.
result New algorithms outperform existing methods in terms of function query complexities and convergence rates.
AdaSVRG combines adaptive gradient with SVRG for robust optimization.
problem Variance reduction in finite-sum minimization with unknown constants.
method AdaSVRG uses AdaGrad in SVRG's inner loop to make it robust.
result AdaSVRG achieves optimal gradient evaluations with no need for problem-dependent constants.
BP-SVRG improves generalization in deep neural networks.
problem Improving generalization in deep neural networks.
method Investigates effects of mini-batching and learning rate decay on SVRG, derives BP-SVRG.
result BP-SVRG outperforms B-SVRG and SGD in deep neural networks.
Paper presents ZO-SVRG for faster nonconvex optimization.
problem Gradient-free optimization challenges in nonconvex settings.
method Comprehensive theoretical analysis, novel ZO-SVRG algorithm, accelerated versions.
result ZO-SVRG achieves best rate for ZO stochastic optimization.
We propose an optimization method for minimizing the finite sums of smooth convex functions. Our method incorporates an accelerated gradient descent (AGD) and a stochastic variance reduction gradient (SVRG) in a mini-batch setting. Unlike SVRG, our method can be directly applied to non-strongly and strongly convex prob…
Unified view of gradient-based algorithms for stochastic convex composite optimization.
problem Optimization of stochastic convex composite functions.
method Extend the concept of estimate sequence to cover various gradient-based methods.
result Generic convergence proof and new adaptive SVRG variant.
Improved SVRG with a coefficient reduces training loss in deep learning.
problem Demonstrating SVRG's effectiveness in deep learning.
method Introduced a multiplicative coefficient α to control SVRG's variance reduction strength and decay it linearly.
result α-SVRG consistently reduces training loss compared to baseline and standard SVRG across various model architectures and datasets.
SVRG reduces gradient evaluations for policy evaluation in reinforcement learning.
problem Policy evaluation in reinforcement learning with high computational costs.
method Two variants of SVRG for policy evaluation that reduce gradient calculations.
result Significant reduction in the number of gradient evaluations while preserving linear convergence speed.
New lower bounds for gradient methods in strongly convex finite-sum optimization.
problem Developing tight lower bounds for randomized gradient methods in finite-sum optimization.
method Deriving tight lower complexity bounds for SAG, SAGA, SVRG, SARAH, and related methods.
result Tight matches between lower bounds and upper bounds for various methods under specific conditions.
Unified convergence analysis of alpha-SVRG under strong convexity.
problem Analyzing the convergence of alpha-SVRG in strongly convex environments.
method Unified convergence rate expression for alpha-SVRG under fixed learning rate, demonstrating faster convergence than SGD and SVRG.
result alpha-SVRG has a faster convergence rate compared to SGD and SVRG under suitable choice of alpha.
Unified approach combines prediction-powered inference and variance reduction for semi-supervised optimization.
problem Scarcity of labeled data in semi-supervised optimization.
method PPI-SVRG, combining PPI and SVRG methods.
result Unified convergence bound with improved performance under label scarcity.
Stabilized SVRG finds second-order stationary points efficiently in nonconvex optimization.
problem Finding second-order stationary points in nonconvex optimization.
method Stabilized SVRG, a variant of SVRG.
result Stabilized SVRG finds an ε-second-order stationary point using only O(n^2/3/ε^2 + n/ε^1.5) stochastic gradients.
In this paper we study the problem of minimizing the average of a large number (n) of smooth convex loss functions. We propose a new method, S2GD (Semi-Stochastic Gradient Descent), which runs for one or several epochs in each of which a single full gradient and a random number of stochastic gradients is computed, fo…
Unified analysis of SVRG and Katyusha using dissipativity theory.
problem Accelerating variance reduction in stochastic optimization.
method Dissipativity theory applied to SVRG and Katyusha.
result Unified convergence analysis of SVRG and Katyusha.
We study nonconvex finite-sum problems and analyze stochastic variance reduced gradient (SVRG) methods for them. SVRG and related methods have recently surged into prominence for convex optimization given their edge over stochastic gradient descent (SGD); but their theoretical analysis almost exclusively assumes convex…
Improved SVRG method using BB techniques for faster convergence.
problem Improving the convergence speed of stochastic variance reduction methods.
method Incorporates Barzilai-Borwein (BB) techniques as second-order information into SVRG.
result Proves linear convergence of the proposed method and its variants.
SVRG and its variants are among the state of art optimization algorithms for large scale machine learning problems. It is well known that SVRG converges linearly when the objective function is strongly convex. However this setup can be restrictive, and does not include several important formulations such as Lasso, grou…
One of the major issues in stochastic gradient descent (SGD) methods is how to choose an appropriate step size while running the algorithm. Since the traditional line search technique does not apply for stochastic optimization algorithms, the common practice in SGD is either to use a diminishing step size, or to tune a…
A new method, k-SVRG, speeds up large-scale optimization.
problem Efficiently solving large-scale optimization problems with variance reduction.
method k-SVRG, which uses available memory and minimizes stalling phases.
result Proves linear convergence on strongly convex problems and convergence to stationary points on non-convex problems.
We present and analyze several strategies for improving the performance of stochastic variance-reduced gradient (SVRG) methods. We first show that the convergence rate of these methods can be preserved under a decreasing sequence of errors in the control variate, and use this to derive variants of SVRG that use growing…
New method reduces complexity of minimizing convex finite sums without needing individual function indices.
problem Minimizing convex finite sums efficiently without knowing which function is being addressed.
method Exploits finite noise structure to derive upper bounds and proposes a novel SVRG adaptation.
result Achieves optimal complexity bounds of O(n^2) and matches existing lower bounds.
SVRG fails for deep learning due to non-convexity.
problem Applying SVRG to deep learning's non-convex optimization problems.
method Exploring SVRG and related techniques for deep learning.
result SVRG fails for deep learning's non-convex optimization problems.
A new distributed learning method for high-dimensional linear classification.
problem Efficiently performing linear classification on large-scale, high-dimensional data.
method Feature-distributed stochastic variance reduced gradient (FD-SVRG) for high-dimensional linear classification.
result FD-SVRG outperforms other distributed methods in terms of communication cost and wall-clock time.
New SVRG and SARAH schemes reduce tuning effort for variance reduction.
problem Optimal performance of SVRG and SARAH requires tuning of parameters.
method Introduces Barzilai-Borwein step sizes, averaging, and adaptive inner loop length.
result Improves SVRG, SARAH, and BB variants' convergence rates and performance.
Improved sample efficiency in policy gradient methods using SVRG.
problem Excessive variance in policy gradient estimates.
method Applying SVRG to model-free policy gradient methods.
result Significantly better performance on robotic control tasks.
A new method for faster optimization of machine learning problems.
problem Minimization of composition of expected value functions.
method C-SAG, a novel extension of SAG for FS-CEVF problems.
result C-SAG achieves lower oracle query complexity per iteration than C-SVRG and converges faster.
Improved deep Q-learning with SVRG reduces variance and stabilizes training.
problem Excessive variance in gradient estimation hinders deep Q-learning performance.
method Utilized stochastic variance reduced gradient (SVRG) techniques.
result Significantly improved performance on 18 out of 20 Atari games compared to baseline methods.
Here we study non-convex composite optimization: first, a finite-sum of smooth but non-convex functions, and second, a general function that admits a simple proximal mapping. Most research on stochastic methods for composite optimization assumes convexity or strong convexity of each function. In this paper, we extend t…
Unified view of stochastic optimization methods with improved convergence and robustness.
problem Stochastic convex composite optimization with noise.
method Estimate sequence approach, accelerated algorithms, robust strategies.
result Optimal complexity accelerated SVRG algorithm robust to noise.
A new algorithm SRG-DQN reduces variance in deep Q-learning.
problem Inaccurate estimation of anchor points in SVRG for deep Q-learning.
method Introduces recursive gradient variance reduction for stochastic gradient updates.
result Demonstrates improved efficiency and effectiveness of SRG-DQN on reinforcement learning tasks.
Proposes a new sufficient decrease technique for SVRG and SAGA.
problem Improving the convergence of stochastic variance reduction gradient descent methods.
method Introduces a sufficient decrease criterion and scaling coefficient to ensure sufficient decrease in stochastic optimization.
result Demonstrates linear convergence rates for strongly convex problems and better performance for non-strongly convex problems.
TRSVR combines SVRG with trust-region for faster optimization.
problem Unconstrained nonconvex optimization problems.
method Adaptive stochastic trust-region method with variance reduction.
result Converges to first-order stationary points with SVRG.
Paper accelerates SAGA using sampled negative momentum.
problem Directly accelerating SAGA for optimization problems.
method Proposes SSNM for direct acceleration of SAGA.
result Achieves best known oracle complexity for strongly convex problems.
Asynchronous federated learning for vertically partitioned data improves efficiency and privacy.
problem Efficiently train models on vertically partitioned data without a trusted third party.
method Proposed AFSGD-VP and its SVRG and SAGA variants for asynchronous federated learning.
result AFSGD-VP and its variants achieve higher efficiency than synchronous algorithms.
Paper improves SPOS by reducing variance in stochastic particle-optimization sampling.
problem Variance reduction in SPOS.
method Proposes three variants of variance-reduced SPOS: SAGA-POS, SVRG-POS, and SVRG-POS\+.
result Non-asymptotic convergence guarantees and better convergence rates than existing methods.
Katyusha X adds momentum to SVRG for faster non-convex optimization.
problem Minimizing sum-of-nonconvex functions in machine learning.
method Adding momentum to SVRG method.
result Provable accelerated stochastic algorithm for sum-of-nonconvex functions.
Many classical algorithms are found until several years later to outlive the confines in which they were conceived, and continue to be relevant in unforeseen settings. In this paper, we show that SVRG is one such method: being originally designed for strongly convex objectives, it is also very robust in non-strongly co…
Loopless SVRG and Katyusha methods outperform their counterparts in practical applications.
problem Training supervised machine learning models efficiently.
method Designing loopless variants of SVRG and Katyusha methods.
result Loopless variants of SVRG and Katyusha have superior practical behavior.
We study optimization algorithms based on variance reduction for stochastic gradient descent (SGD). Remarkable recent progress has been made in this direction through development of algorithms like SAG, SVRG, SAGA. These algorithms have been shown to outperform SGD, both theoretically and empirically. However, asynchro…
We introduce and analyze stochastic optimization methods where the input to each gradient update is perturbed by bounded noise. We show that this framework forms the basis of a unified approach to analyze asynchronous implementations of stochastic optimization algorithms.In this framework, asynchronous stochastic optim…
Contrary to the situation with stochastic gradient descent, we argue that when using stochastic methods with variance reduction, such as SDCA, SAG or SVRG, as well as their variants, it could be beneficial to reuse previously used samples instead of fresh samples, even when fresh samples are available. We demonstrate t…
New algorithm reduces variance in nonconvex optimization problems.
problem Finite-sum nonconvex optimization problems.
method Stochastic gradient descent with nested variance reduction.
result Converges to an ε-stationary point with improved complexity.