arXiv research

A new framework for bilevel optimization tackles stochastic and global variance reduction.

New approach combines PCA and t-sne for better data analysis.

With a new deprivation (or poverty) function, in this paper, we theoretically study the changes in poverty with respect to the `global' mean and variance of the income distribution using Indian survey data. We show that when the income obeys a log-normal distribution, a rising mean income generally indicates a reductio…

AB-SAGA optimizes distributed optimization over directed graphs using variance reduction and stochastic weights.

A new method reduces variance in PG methods for RL, improving efficiency and convergence.

Improved sample complexity for actor-critic algorithms in MDPs.

In this paper, we propose a stochastic optimization method that adaptively controls the sample size used in the computation of gradient approximations. Unlike other variance reduction techniques that either require additional storage or the regular computation of full gradients, the proposed method reduces variance by …

Data balancing reduces variance in machine learning models.

Visualizing high-dimensional data is an essential task in Data Science and Machine Learning. The Centroid-Encoder (CE) method is similar to the autoencoder but incorporates label information to keep objects of a class close together in the reduced visualization space. CE exploits nonlinearity and labels to encode high …

Neural SDEs reduce variance in stochastic simulations.

MARS optimizes large model training by reducing variance, outperforming AdamW.

Paper uses DFL to optimize portfolio risk and outperforms conventional methods.

Normal distributions ensure asymptotic variance reduction in moment matching Monte Carlo.

We study the stochastic Riemannian gradient algorithm for matrix eigen-decomposition. The state-of-the-art stochastic Riemannian algorithm requires the learning rate to decay to zero and thus suffers from slow convergence and sub-optimal solutions. In this paper, we address this issue by deploying the variance reductio…

In this paper we suggest a modification of the regression-based variance reduction approach recently proposed in Belomestny et al. This modification is based on the stratification technique and allows for a further significant variance reduction. The performance of the proposed approach is illustrated by several numeri…

FedGLOMO accelerates FL convergence for non-convex functions.

New algorithm reduces optimization complexity in adaptive mirror descent.

SignSVRG improves SignSGD by reducing variance, achieving similar convergence rates.

Optimizes MCMC chains with neural control variates.

Paper proposes a method to estimate variance reduction in DNN training using importance sampling.

Optimizes variance reduction in Heston model using large and moderate deviations.

In this paper we propose a novel variance reduction approach for additive functionals of Markov chains based on minimization of an estimate for the asymptotic variance of these functionals over suitable classes of control variates. A distinctive feature of the proposed approach is its ability to significantly reduce th…

We study nonconvex finite-sum problems and analyze stochastic variance reduced gradient (SVRG) methods for them. SVRG and related methods have recently surged into prominence for convex optimization given their edge over stochastic gradient descent (SGD); but their theoretical analysis almost exclusively assumes convex…

This work proposes using zero-variance control variates to reduce variance in pathwise gradient estimators for variational inference.

New optimizer MARS-M combines variance reduction with Muon for faster LLM training.

Develops a new SPP algorithm with variance reduction for weakly convex optimization.

Improved time complexity for parallel stochastic optimization in heterogeneous systems.

Paper develops momentum schemes with variance reduction for non-convex composition optimization.

This paper develops a new theory for ensemble learning beyond variance reduction.

Paper improves tree probability estimation using stochastic optimization and variance reduction.

New algorithms reduce variance in solving complex mathematical problems.

Improved variance reduction for Riemannian non-convex optimization with adaptive batch size.

We study the problem of empirical minimization for variance-type functionals over functional classes. Sharp non-asymptotic bounds for the excess variance are derived under mild conditions. In particular, it is shown that under some restrictions imposed on the functional class fast convergence rates can be achieved incl…

RMDA trains structured neural networks with regularization and variance reduction.

Gradient-based Monte Carlo sampling algorithms, like Langevin dynamics and Hamiltonian Monte Carlo, are important methods for Bayesian inference. In large-scale settings, full-gradients are not affordable and thus stochastic gradients evaluated on mini-batches are used as a replacement. In order to reduce the high vari…

While optimizing convex objective (loss) functions has been a powerhouse for machine learning for at least two decades, non-convex loss functions have attracted fast growing interests recently, due to many desirable properties such as superior robustness and classification accuracy, compared with their convex counterpa…

Variance reduction has emerged in recent years as a strong competitor to stochastic gradient descent in non-convex problems, providing the first algorithms to improve upon the converge rate of stochastic gradient descent for finding first-order critical points. However, variance reduction techniques typically require c…

Paper improves efficiency in matrix computations for Gaussian processes.

Stochastic variance reduction algorithms have recently become popular for minimizing the average of a large, but finite, number of loss functions. In this paper, we propose a novel Riemannian extension of the Euclidean stochastic variance reduced gradient algorithm (R-SVRG) to a compact manifold search space. To this e…

Stochastic optimization algorithms with variance reduction have proven successful for minimizing large finite sums of functions. Unfortunately, these techniques are unable to deal with stochastic perturbations of input data, induced for example by data augmentation. In such cases, the objective is no longer a finite su…

In recent years, stochastic variance reduction algorithms have attracted considerable attention for minimizing the average of a large but finite number of loss functions. This paper proposes a novel Riemannian extension of the Euclidean stochastic variance reduced gradient (R-SVRG) algorithm to a manifold search space.…

Survey of SDR methods for high-dimensional regression and embedding.

Improves gradient estimation for discrete distributions with variance reduction techniques.

U-statistics improve gradient estimation in importance-weighted variational inference.

We study optimization algorithms based on variance reduction for stochastic gradient descent (SGD). Remarkable recent progress has been made in this direction through development of algorithms like SAG, SVRG, SAGA. These algorithms have been shown to outperform SGD, both theoretically and empirically. However, asynchro…

Ensembles improve classifier performance by reducing bias, not variance.

New method reduces variance in complex probabilistic model optimization.

New algorithm reduces variance in Monte Carlo simulations using deep neural networks and policy gradients.