Develops a new SPP algorithm with variance reduction for weakly convex optimization.
problem Weakly convex, composite optimization problems.
method Inexact semismooth Newton framework with variance reduction for stochastic proximal point updates.
result Establishes convergence results for the proposed algorithm.
Neural SDEs reduce variance in stochastic simulations.
problem Efficiency of Monte Carlo simulations in finance.
method Use neural SDEs with control variates parameterized by neural networks.
result Prove optimality conditions for variance reduction in SDEs with infinite activity.
In the paper, we study the stochastic alternating direction method of multipliers (ADMM) for the nonconvex optimizations, and propose three classes of the nonconvex stochastic ADMM with variance reduction, based on different reduced variance stochastic gradients. Specifically, the first class called the nonconvex stoch…
New method reduces variance in complex probabilistic model optimization.
problem High variance in stochastic optimisation of complex models.
method Use recognition network to approximate optimal control variate for each mini-batch.
result Sub-optimal variance reduction is improved with new approach.
Stochastic optimization algorithms with variance reduction have proven successful for minimizing large finite sums of functions. Unfortunately, these techniques are unable to deal with stochastic perturbations of input data, induced for example by data augmentation. In such cases, the objective is no longer a finite su…
New algorithms reduce variance in solving complex mathematical problems.
problem Solving convex-concave saddle point problems, variational inequalities, and inclusions.
method Stochastic variance reduction for extragradient, forward-backward-forward, and forward-reflected-backward methods.
result All proposed methods converge with complexities matching or improving deterministic counterparts.
Optimizes variance reduction in Heston model using large and moderate deviations.
problem Improving variance reduction in stochastic volatility models.
method Large and moderate deviations theory applied to Heston model.
result Derives closed-form solutions for optimal change of measure.
New method reduces variance in stochastic optimization with high confidence.
problem Achieving high-probability guarantees in stochastic optimization with weaker noise assumptions.
method Stochastic proximal point method combining proximal subproblem solver and probability booster.
result Demonstrates convergence with low sample complexity under bounded variance assumptions.
Gradient-based Monte Carlo sampling algorithms, like Langevin dynamics and Hamiltonian Monte Carlo, are important methods for Bayesian inference. In large-scale settings, full-gradients are not affordable and thus stochastic gradients evaluated on mini-batches are used as a replacement. In order to reduce the high vari…
Improved time complexity for parallel stochastic optimization in heterogeneous systems.
problem Time complexity in parallel stochastic optimization for large-scale machine learning models.
method Proposes Rennala MVR, a variance-reduced extension of Rennala SGD based on momentum-based variance reduction.
result Variance reduction improves time complexity in relevant parameter regimes for parallel stochastic optimization in heterogeneous systems.
Paper improves tree probability estimation using stochastic optimization and variance reduction.
problem Improving tree probability estimation in phylogenetic inference.
method Introduces computationally efficient methods for training SBNs and variance reduction for optimization.
result Methods outperform previous baseline methods in tree topology probability estimation and Bayesian phylogenetic inference.
Variance reduction (VR) methods boost the performance of stochastic gradient descent (SGD) by enabling the use of larger, constant stepsizes and preserving linear convergence rates. However, current variance reduced SGD methods require either high memory usage or an exact gradient computation (using the entire dataset)…
SVRN accelerates Newton methods by reducing variance and improving performance.
problem Improving the efficiency of Newton methods for large-scale optimization problems.
method Stochastic Variance-Reduced Newton (SVRN) algorithm that accelerates Subsampled Newton and Iterative Hessian Sketch algorithms.
result SVRN accelerates Newton methods by reducing the number of passes over the data, achieving a significant improvement in performance.
A new framework for bilevel optimization tackles stochastic and global variance reduction.
problem Bilevel optimization challenges in large-scale empirical risk minimization.
method Introducing a novel framework where inner and main variables evolve simultaneously, leading to unbiased estimates and global variance reduction algorithms.
result SABA algorithm achieves $O( rac{1}{T})$ convergence rate and linear convergence under Polyak-Lojasciewicz assumption.
MARS optimizes large model training by reducing variance, outperforming AdamW.
problem Training large models efficiently and scalably.
method Unified optimization framework MARS combining preconditioned gradient updates and variance reduction.
result MARS outperforms AdamW in training GPT-2 models.
Paper improves efficiency in matrix computations for Gaussian processes.
problem Efficiency in matrix computations for Gaussian processes.
method Variance reduction via matrix factorization.
result Factorized estimator can be up to 1,000 times more efficient.
Stochastic variance reduction algorithms have recently become popular for minimizing the average of a large, but finite number of loss functions. The present paper proposes a Riemannian stochastic quasi-Newton algorithm with variance reduction (R-SQN-VR). The key challenges of averaging, adding, and subtracting multipl…
New method improves convergence and reduces variance in noisy optimization problems.
problem Computing exact minimizers with noisy gradient information.
method Stochastic mirror descent with interacting particles.
result Interaction helps improve convergence and reduce variance.
AB-SAGA optimizes distributed optimization over directed graphs using variance reduction and stochastic weights.
problem Optimizing distributed stochastic optimization over directed graphs with stochastic weights.
method AB-SAGA combines variance reduction and network-level gradient tracking, using both row and column stochastic weights.
result AB-SAGA converges linearly to the global optimal with a constant step-size and achieves a linear speed-up over centralized methods.
A new algorithm SRG-DQN reduces variance in deep Q-learning.
problem Inaccurate estimation of anchor points in SVRG for deep Q-learning.
method Introduces recursive gradient variance reduction for stochastic gradient updates.
result Demonstrates improved efficiency and effectiveness of SRG-DQN on reinforcement learning tasks.
Stochastic particle-optimization sampling (SPOS) is a recently-developed scalable Bayesian sampling framework that unifies stochastic gradient MCMC (SG-MCMC) and Stein variational gradient descent (SVGD) algorithms based on Wasserstein gradient flows. With a rigorous non-asymptotic convergence theory developed recently…
Stochastic gradient Markov Chain Monte Carlo (SG-MCMC) has been developed as a flexible family of scalable Bayesian sampling algorithms. However, there has been little theoretical analysis of the impact of minibatch size to the algorithm's convergence rate. In this paper, we prove that under a limited computational bud…
Optimizes MCMC chains with neural control variates.
problem Reducing variance in Markov Chain Monte Carlo (MCMC) simulations.
method Uses neural networks as control variates to minimize asymptotic variance.
result Derives optimal convergence rate under various ergodicity assumptions.
Variance reduction has emerged in recent years as a strong competitor to stochastic gradient descent in non-convex problems, providing the first algorithms to improve upon the converge rate of stochastic gradient descent for finding first-order critical points. However, variance reduction techniques typically require c…
We generalize stochastic smoothing for gradient estimation of non-differentiable functions.
problem Gradient estimation for non-differentiable functions.
method Developed a general framework for relaxation and gradient estimation of non-differentiable black-box functions using stochastic smoothing with reduced assumptions.
result Empirically validated the effectiveness of variance reduction strategies for various non-differentiable tasks.
We study optimization algorithms based on variance reduction for stochastic gradient descent (SGD). Remarkable recent progress has been made in this direction through development of algorithms like SAG, SVRG, SAGA. These algorithms have been shown to outperform SGD, both theoretically and empirically. However, asynchro…
Conjugate gradient (CG) methods are a class of important methods for solving linear equations and nonlinear optimization problems. In this paper, we propose a new stochastic CG algorithm with variance reduction and we prove its linear convergence with the Fletcher and Reeves method for strongly convex and smooth functi…
A new algorithm reduces bias and variance in distributionally robust optimization.
problem Distributionally robust optimization with bias and variance issues.
method Prospect, a stochastic gradient-based algorithm that reduces hyperparameter tuning.
result Prospect achieves linear convergence and 2-3x faster convergence on various benchmarks.
New sparsity operator reduces variance reduction methods' computational cost.
problem Reduce computational cost of variance reduction methods.
method Introduce random-top-k operator to estimate gradient sparsity and reduce operations per update.
result Our algorithm consistently outperforms SpiderBoost in various tasks.
We study finite-sum nonconvex optimization problems, where the objective function is an average of n nonconvex functions. We propose a new stochastic gradient descent algorithm based on nested variance reduction. Compared with conventional stochastic variance reduced gradient (SVRG) algorithm that uses two reference …
New algorithm reduces optimization complexity in adaptive mirror descent.
problem Optimizing complex, non-smooth, non-convex functions efficiently.
method SVRAMD: Variance Reduced Adaptive Mirror Descent.
result Variance reduction accelerates convergence in adaptive mirror descent.
We consider a composite convex minimization problem associated with regularized empirical risk minimization, which often arises in machine learning. We propose two new stochastic gradient methods that are based on stochastic dual averaging method with variance reduction. Our methods generate a sparser solution than the…
Enhances option pricing for American-style options using JDOI method.
problem Pricing American-style options efficiently under stochastic volatility.
method Extends DOI variance reduction technique to Lévy dynamics, combining with LSMC.
result Strong variance reduction in option pricing compared to standard LSMC.
This paper develops scalable control variates for Monte Carlo methods using stochastic optimization.
problem Reducing variance in Monte Carlo estimators for large-scale problems.
method Control variates based on Stein operators, optimized through stochastic optimization.
result Novel theoretical results and empirical validations show effective variance reduction.
Push-SAGA is a decentralized algorithm for directed graphs that converges linearly.
problem Finite-sum minimization over directed graphs with stochastic gradients.
method Combines variance reduction, gradient tracking, and consensus algorithms.
result Achieves linear convergence for smooth and strongly convex problems.
This work extends the variance reduction method for the pricing of possibly path-dependent derivatives, which was developed in (Genin and Tankov, 2016) for exponential Lévy models, to affine stochastic volatility models (Keller-Ressel, 2011). We begin by proving a pathwise large deviations principle for affine stochast…
FGD reduces noisy gradient variance in SGD for neural networks.
problem Noisy and unreliable gradient estimation in SGD for deep learning.
method Solves an adaptive filtering problem to consistently estimate the local gradient.
result Significantly reduces gradient variance and accelerates convergence.
VR methods improve SGD for faster machine learning.
problem Efficiency in stochastic optimization for machine learning.
method Variance reduction techniques for stochastic optimization.
result VR methods achieve faster convergence than SGD.
New technique reduces bias in CSO problems, improving sample complexity.
problem Reducing bias in conditional stochastic optimization problems.
method Introducing a stochastic extrapolation technique combined with variance reduction.
result Achieved significantly better sample complexity for nonconvex smooth objectives.
The paper analyzes variance reduction in stochastic gradient Langevin dynamics.
problem Reducing the variance of stochastic gradient estimators in Langevin dynamics.
method Central limit theorem and Poisson equation analysis for variance characterization.
result Anti-symmetric perturbations can reduce the variance of non-reversible Langevin dynamics.
Paper develops momentum schemes with variance reduction for non-convex composition optimization.
problem Lack of convergence guarantee and efficient momentum design in existing algorithms.
method Develops various momentum schemes with SPIDER-based variance reduction.
result Achieves near-optimal sample complexity and linear convergence rate.
Adaptive importance sampling for stochastic optimization is a promising approach that offers improved convergence through variance reduction. In this work, we propose a new framework for variance reduction that enables the use of mixtures over predefined sampling distributions, which can naturally encode prior knowledg…
New variance-reduction methods solve stochastic composite inclusions.
problem Solving nonmonotone stochastic composite inclusions.
method Developed unbiased and biased variance-reduced estimators for FRBS method.
result Achieved best oracle complexities for finite-sum and expectation settings.
Paper improves convergence rate of Langevin Dynamics algorithms.
problem Sampling problems and non-convex optimization in machine learning.
method Stochastic Variance Reduced Gradient Langevin Dynamics and Stochastic Recursive Gradient Langevin Dynamics with improved convergence rates.
result Proves convergence to objective distribution under weaker conditions.
Improved reSGLD accelerates convergence in non-convex learning problems.
problem Inefficient swaps due to noisy energy estimators in reSGLD.
method Variance reduction for noisy energy estimators, theoretical analysis, and numerical experiments.
result Exponential acceleration in convergence for non-convex learning problems.
To backpropagate the gradients through stochastic binary layers, we propose the augment-REINFORCE-merge (ARM) estimator that is unbiased, exhibits low variance, and has low computational complexity. Exploiting variable augmentation, REINFORCE, and reparameterization, the ARM estimator achieves adaptive variance reducti…
Improved variance reduction for Riemannian non-convex optimization with adaptive batch size.
problem Optimizing non-convex functions on Riemannian manifolds.
method Batch size adaptation in R-SVRG, R-SRG, and R-SPIDER.
result Achieves lower total complexities for various non-convex functions.
TRSVR combines SVRG with trust-region for faster optimization.
problem Unconstrained nonconvex optimization problems.
method Adaptive stochastic trust-region method with variance reduction.
result Converges to first-order stationary points with SVRG.