MUSE provides unbiased stopping estimates for optimal problems.
problem Estimating the utility of optimal stopping problems.
method Backward recursive construction of the Multilevel Unbiased Stopping Estimator (MUSE).
result MUSE achieves ε-accuracy with O(1/ε^2) computational cost.
Randomized trials, also known as A/B tests, are used to select between two policies: a control and a treatment. Given a corresponding set of features, we can ideally learn an optimized policy P that maps the A/B test data features to action space and optimizes reward. However, although A/B testing provides an unbiased …
New unbiased gradient estimators for complex optimization problems.
problem Unbiased and variance-limited gradient estimation for conditional stochastic optimization.
method Developed multilevel Monte Carlo gradient estimators for conditional stochastic optimization problems.
result Unbiased and finite variance gradient estimators for conditional stochastic optimization problems.
Optimal Gaussian noise mechanisms achieve nearly optimal error in unbiased mean estimation.
problem Efficiently estimating the mean of high-dimensional data while preserving privacy.
method Differential privacy mechanisms with Gaussian noise, focusing on optimal covariance.
result Gaussian noise mechanisms achieve nearly optimal error among all private unbiased mean estimation mechanisms.
A number of optimization approaches have been proposed for optimizing nonconvex objectives (e.g. deep learning models), such as batch gradient descent, stochastic gradient descent and stochastic variance reduced gradient descent. Theory shows these optimization methods can converge by using an unbiased gradient estimat…
New algorithm finds unbiased subnetworks in biased datasets.
problem Finding unbiased subnetworks in biased neural networks.
method Debiased Contrastive Weight Pruning (DCWP) algorithm.
result DCWP significantly outperforms state-of-the-art debiasing methods.
Paper proposes an unbiased optimization method for Bayesian experimental design.
problem Maximizing expected information gain in Bayesian experimental design.
method Randomized multilevel Monte Carlo (MLMC) method combined with stochastic gradient descent.
result An unbiased estimator for the gradient of expected information gain.
The recently proposed Unbiased Online Recurrent Optimization algorithm (UORO, arXiv:1702.05043) uses an unbiased approximation of RTRL to achieve fully online gradient-based learning in RNNs. In this work we analyze the variance of the gradient estimate computed by UORO, and propose several possible changes to the meth…
SUMO provides unbiased log marginal likelihood estimation for latent variable models.
problem Biased estimates of log marginal likelihood in latent variable models.
method Randomized truncation of infinite series for unbiased estimation.
result Models trained with SUMO give better test-set likelihoods than standard methods.
This paper tackles unbiased loss functions for multilabel classification with missing labels.
problem Missing labels in multilabel classification tasks, especially in extreme multi-label classification (XMC).
method Derives unbiased estimators for multilabel reductions, including non-decomposable ones, and addresses increased variance with convex upper-bounds.
result Switching to unbiased estimators can alter the bias-variance trade-off and may require stronger regularization.
Recent neural network and language models rely on softmax distributions with an extremely large number of categories. Since calculating the softmax normalizing constant in this context is prohibitively expensive, there is a growing literature of efficiently computable but biased estimates of the softmax. In this paper …
New method for unbiased regression reduces excess risk.
problem Least squares regression with optimal solution and Hessian matrix.
method Averaged stochastic gradient descent with time-average estimator.
result Unbiased estimator with O(1/k) expected excess risk.
New method reduces deep learning training costs by approximating vector-jacobian products.
problem Efficiently training deep neural networks with reduced computational and memory costs.
method Randomized, unbiased approximations of vector-jacobian products during backpropagation.
result Validated potential for reducing deep learning training costs through unbiased estimates.
We consider distributed optimization problems where forming the Hessian is computationally challenging and communication is a significant bottleneck. We develop unbiased parameter averaging methods for randomized second order optimization that employ sampling and sketching of the Hessian. Existing works do not take the…
In this analytical study we derive the optimal unbiased value estimator (MVU) and compare its statistical risk to three well known value estimators: Temporal Difference learning (TD), Monte Carlo estimation (MC) and Least-Squares Temporal Difference Learning (LSTD). We demonstrate that LSTD is equivalent to the MVU if …
A large class of machine learning techniques requires the solution of optimization problems involving spectral functions of parametric matrices, e.g. log-determinant and nuclear norm. Unfortunately, computing the gradient of a spectral function is generally of cubic complexity, as such gradient descent methods are rath…
Paper introduces a new gradient statistic to improve deep learning convergence.
problem Fluctuation effect of gradient updates between iterations.
method Introduces an unbiased stratified statistic \(\bar{G}_{mst}\) and a new algorithm MSSG.
result MSSG algorithm outperforms other sgd-like algorithms in training deep models.
New theory of sensitivity for unbiased estimators using Wasserstein geometry.
problem Estimating the instability of estimators under small perturbations.
method Developed a new theory based on Wasserstein geometry, analogous to classical Cramér-Rao theory.
result Wasserstein-Cramér-Rao lower bound for sensitivity of unbiased estimators.
We propose and analyze a new stochastic gradient method, which we call Stochastic Unbiased Curvature-aided Gradient (SUCAG), for finite sum optimization problems. SUCAG constitutes an unbiased total gradient tracking technique that uses Hessian information to accelerate con- vergence. We analyze our method under the ge…
Develops new instance-optimality concepts in differential privacy.
problem Improving privacy guarantees in statistical estimation.
method Introduces local minimax risk and unbiased mechanisms, and develops inverse sensitivity mechanisms.
result Inverse sensitivity mechanisms are nearly instance optimal for a wide range of functions.
U-Clip clips gradients to reduce bias without discarding them.
problem Reduces bias in stochastic gradient descent.
method Maintains a buffer of clipped gradients to add to next iteration.
result U-Clip updates are unbiased on average.
A new framework for bilevel optimization tackles stochastic and global variance reduction.
problem Bilevel optimization challenges in large-scale empirical risk minimization.
method Introducing a novel framework where inner and main variables evolve simultaneously, leading to unbiased estimates and global variance reduction algorithms.
result SABA algorithm achieves $O( rac{1}{T})$ convergence rate and linear convergence under Polyak-Lojasciewicz assumption.
Scalable Gaussian process models trained with unbiased stochastic ELBO.
problem Training large capacity Gaussian process models on huge datasets.
method Unbiased stochastic variational inference for scalable GPs.
result Accurate inference on large datasets with up to 10 million basis functions.
A new method speeds up sampling of Boltzmann distribution in high-dimensional systems.
problem High computational cost of obtaining Jacobian of flow-based models in high dimensions.
method Flow perturbation method that incorporates stochastic perturbations and reweighting.
result Achieves unbiased sampling of Boltzmann distribution with orders of magnitude speedup.
Stochastic gradient descent (SGD), which dates back to the 1950s, is one of the most popular and effective approaches for performing stochastic optimization. Research on SGD resurged recently in machine learning for optimizing convex loss functions and training nonconvex deep neural networks. The theory assumes that on…
PES method reduces bias in gradient estimation for unrolled graphs.
problem High variance and bias in gradient estimation for unrolled computation graphs.
method Divide graph into unrolls, apply ES update, accumulate correction terms.
result PES provides unbiased, low-variance gradient estimates.
Clapping reduces memory usage in distributed optimization by reusing data samples.
problem Significant communication overhead and impractical memory overhead in pipeline-parallel distributed optimization.
method Lazy sampling strategy to reuse data samples across steps, supporting convergence without unbiased gradient assumptions.
result Clapping achieves convergence in few-epoch or online training regimes without sample-size memory overhead.
Improved UIVI method shows better performance than state-of-the-art SIVI methods.
problem Estimating the likelihood of samples from complex distributions in high dimensions.
method Replaced the inner MCMC loop of UIVI with importance sampling and learned the optimal proposal distribution.
result The refined UIVI approach demonstrates superior performance or parity with state-of-the-art methods.
New variance-reduction methods solve stochastic composite inclusions.
problem Solving nonmonotone stochastic composite inclusions.
method Developed unbiased and biased variance-reduced estimators for FRBS method.
result Achieved best oracle complexities for finite-sum and expectation settings.
Synthetic construction of 3D complex bases.
problem Creating a complete set of unbiased bases in 3D complex space.
method Synthetic construction using complex projective trigonometry.
result Synthetic construction of mutually unbiased bases in C^3.
Enhanced framework selects features for unbiased causal inference.
problem Unbiased estimation of causal quantities in causal inference.
method Three-stage computational framework balancing treatment and non-treatment variables.
result Significantly reduces bias and variance in estimating causal quantities.
EigenGame improves eigendecomposition by offering unbiased updates for larger datasets.
problem Minibatch bias in EigenGame limits convergence and parallelism.
method Proposed unbiased stochastic update for EigenGame.
result Asymptotic equivalence to EigenGame, greater parallelism, and improved performance.
Unbiased wealth exchanges always lead to inequality.
problem Understanding wealth distribution in unbiased binary exchange systems.
method Analytical demonstration of unbiased binary exchanges leading to perfect inequality.
result Any system driven by unbiased binary exchanges will reach perfect inequality and zero mobility.
We develop unbiased implicit variational inference (UIVI), a method that expands the applicability of variational inference by defining an expressive variational family. UIVI considers an implicit variational distribution obtained in a hierarchical manner using a simple reparameterizable distribution whose variational …
A new method improves communication efficiency in distributed learning.
problem Reducing communication overhead in distributed machine learning.
method Transforming contractive compressors into induced unbiased compressors.
result Significant improvements in memory requirements and communication complexity.
New method improves FO-BLO convergence without increasing memory or time complexity.
problem Lack of theoretical understanding of FO-BLO convergence.
method Unbiased first-order bilevel optimization (UFO-BLO) to improve convergence.
result Theoretical guarantee of convergence for FO-BLO-based stochastic optimization.
Proposes unbiased estimators for training mixture of experts models.
problem Efficiently training large-scale mixture of experts models on modern hardware.
method Two unbiased estimators based on principled stochastic assignment procedures.
result Both estimators are more effective and robust than biased alternatives.
New methods improve distributed optimization on non-iid data.
problem Communication bottleneck in distributed machine learning models.
method Two types of distributed gradient compression methods (D-QSGD and D-EF-SGD) analyzed for non-iid data.
result D-EF-SGD performs better than D-QSGD on non-iid data but can still slow down with high data skewness.
Unbiased methods for alpha-divergence minimization struggle in high dimensions.
problem The difficulty of unbiased alpha-divergence minimization in high dimensions.
method Signal-to-Noise Ratio (SNR) analysis of gradient estimators.
result The SNR of the gradient estimator worsens exponentially with dimensionality.
Stochastic variational inference (SVI) lets us scale up Bayesian computation to massive data. It uses stochastic optimization to fit a variational distribution, following easy-to-compute noisy natural gradients. As with most traditional stochastic optimization methods, SVI takes precautions to use unbiased stochastic g…
Unbiased gradient estimation for Markov chains
problem Estimating gradients of stationary means in Markov chains
method Propose new unbiased estimators
result Improves efficiency for slow mixing Markov chains
In this paper, we introduce a new approach to constructing unbiased estimators when computing expectations of path functionals associated with stochastic differential equations (SDEs). Our randomization idea is closely related to multi-level Monte Carlo and provides a simple mechanism for constructing a finite variance…
Extends unbiased simulation method to Asian options.
problem Simulating path-dependent dynamics for Asian options.
method Extension of unbiased simulation method for SDEs to path-dependent dynamics.
result Extension applies to numerical resolution of path-dependent PDEs.
Developed unbiased estimators for Heston model with stochastic interest rates.
problem Estimating the Heston model with stochastic interest rates.
method Combined unbiased estimators with the Heston model and developed a semi-exact log-Euler scheme.
result Convergence rate of O(h) in the L2 norm for a wide range of models. The estimation of risk measures recently gained a lot of attention, partly because of the backtesting issues of expected shortfall related to elicitability. In this work we shed a new and fundamental light on optimal estimation procedures of risk measures in terms of bias. We show that once the parameters of a model ne…
Paper proposes an unbiased risk estimator for PLLAC, handling unseen classes.
problem Handling unseen classes in PLLAC where some classes are not present in the training set.
method Proposes an unbiased risk estimator that estimates the distribution of augmented classes by differentiating known classes from unlabeled data.
result The estimator provides theoretical guarantees and converges to true risk minimizer as data increases.
We revisit resampling procedures for error estimation in binary classification in terms of U-statistics. In particular, we exploit the fact that the error rate estimator involving all learning-testing splits is a U-statistic. Thus, it has minimal variance among all unbiased estimators and is asymptotically normally dis…
Computing partition functions, the normalizing constants of probability distributions, is often hard. Variants of importance sampling give unbiased estimates of a normalizer Z, however, unbiased estimates of the reciprocal 1/Z are harder to obtain. Unbiased estimates of 1/Z allow Markov chain Monte Carlo sampling of "d…