arXiv research

Data balancing reduces variance in machine learning models.

Neural SDEs reduce variance in stochastic simulations.

MARS optimizes large model training by reducing variance, outperforming AdamW.

Normal distributions ensure asymptotic variance reduction in moment matching Monte Carlo.

In this paper we suggest a modification of the regression-based variance reduction approach recently proposed in Belomestny et al. This modification is based on the stratification technique and allows for a further significant variance reduction. The performance of the proposed approach is illustrated by several numeri…

New algorithm reduces optimization complexity in adaptive mirror descent.

SignSVRG improves SignSGD by reducing variance, achieving similar convergence rates.

Optimizes MCMC chains with neural control variates.

Paper proposes a method to estimate variance reduction in DNN training using importance sampling.

Optimizes variance reduction in Heston model using large and moderate deviations.

In this paper we propose a novel variance reduction approach for additive functionals of Markov chains based on minimization of an estimate for the asymptotic variance of these functionals over suitable classes of control variates. A distinctive feature of the proposed approach is its ability to significantly reduce th…

This work proposes using zero-variance control variates to reduce variance in pathwise gradient estimators for variational inference.

New optimizer MARS-M combines variance reduction with Muon for faster LLM training.

Develops a new SPP algorithm with variance reduction for weakly convex optimization.

Improved time complexity for parallel stochastic optimization in heterogeneous systems.

Paper develops momentum schemes with variance reduction for non-convex composition optimization.

This paper develops a new theory for ensemble learning beyond variance reduction.

Paper improves tree probability estimation using stochastic optimization and variance reduction.

New algorithms reduce variance in solving complex mathematical problems.

Improved variance reduction for Riemannian non-convex optimization with adaptive batch size.

We study the problem of empirical minimization for variance-type functionals over functional classes. Sharp non-asymptotic bounds for the excess variance are derived under mild conditions. In particular, it is shown that under some restrictions imposed on the functional class fast convergence rates can be achieved incl…

RMDA trains structured neural networks with regularization and variance reduction.

Gradient-based Monte Carlo sampling algorithms, like Langevin dynamics and Hamiltonian Monte Carlo, are important methods for Bayesian inference. In large-scale settings, full-gradients are not affordable and thus stochastic gradients evaluated on mini-batches are used as a replacement. In order to reduce the high vari…

Variance reduction has emerged in recent years as a strong competitor to stochastic gradient descent in non-convex problems, providing the first algorithms to improve upon the converge rate of stochastic gradient descent for finding first-order critical points. However, variance reduction techniques typically require c…

Paper improves efficiency in matrix computations for Gaussian processes.

Stochastic optimization algorithms with variance reduction have proven successful for minimizing large finite sums of functions. Unfortunately, these techniques are unable to deal with stochastic perturbations of input data, induced for example by data augmentation. In such cases, the objective is no longer a finite su…

Survey of SDR methods for high-dimensional regression and embedding.

Improves gradient estimation for discrete distributions with variance reduction techniques.

U-statistics improve gradient estimation in importance-weighted variational inference.

We study optimization algorithms based on variance reduction for stochastic gradient descent (SGD). Remarkable recent progress has been made in this direction through development of algorithms like SAG, SVRG, SAGA. These algorithms have been shown to outperform SGD, both theoretically and empirically. However, asynchro…

Ensembles improve classifier performance by reducing bias, not variance.

New method reduces variance in complex probabilistic model optimization.

New algorithm reduces variance in Monte Carlo simulations using deep neural networks and policy gradients.

We show that on-policy policy gradient (PG) and its variance reduction variants can be derived by taking finite difference of function evaluations supplied by estimators from the importance sampling (IS) family for off-policy evaluation (OPE). Starting from the doubly robust (DR) estimator (Jiang & Li, 2016), we provid…

Paper improves Gumbel-Softmax estimator variance reduction.

Evolution Strategies (ES) are a powerful class of blackbox optimization techniques that recently became a competitive alternative to state-of-the-art policy gradient (PG) algorithms for reinforcement learning (RL). We propose a new method for improving accuracy of the ES algorithms, that as opposed to recent approaches…

Ant colonies and boosting algorithms both reduce bias and variance through adaptive mechanisms.

Adaptive importance sampling for stochastic optimization is a promising approach that offers improved convergence through variance reduction. In this work, we propose a new framework for variance reduction that enables the use of mixtures over predefined sampling distributions, which can naturally encode prior knowledg…

In the paper, we study the stochastic alternating direction method of multipliers (ADMM) for the nonconvex optimizations, and propose three classes of the nonconvex stochastic ADMM with variance reduction, based on different reduced variance stochastic gradients. Specifically, the first class called the nonconvex stoch…

Meta-CVs leverage task similarity to reduce variance with limited data.

We reduce variance in monetization metrics for ranking experiments.

Our paper improves uplift model evaluation on randomized controlled trials (RCT) data.

Variance reduction (VR) methods boost the performance of stochastic gradient descent (SGD) by enabling the use of larger, constant stepsizes and preserving linear convergence rates. However, current variance reduced SGD methods require either high memory usage or an exact gradient computation (using the entire dataset)…

VRER selectively reuses past observations to reduce variance in policy optimization.

Machine learning reduces variance in online experiment results.

SVRN accelerates Newton methods by reducing variance and improving performance.

Meta-learning variance reduced via Laplace approximation for regression tasks.

Variance reduction is a simple and effective technique that accelerates convex (or non-convex) stochastic optimization. Among existing variance reduction methods, SVRG and SAGA adopt unbiased gradient estimators and are the most popular variance reduction methods in recent years. Although various accelerated variants o…