Proposes unbiased estimators for training mixture of experts models.
problem Efficiently training large-scale mixture of experts models on modern hardware.
method Two unbiased estimators based on principled stochastic assignment procedures.
result Both estimators are more effective and robust than biased alternatives.
Randomized trials, also known as A/B tests, are used to select between two policies: a control and a treatment. Given a corresponding set of features, we can ideally learn an optimized policy P that maps the A/B test data features to action space and optimizes reward. However, although A/B testing provides an unbiased …
Computing partition functions, the normalizing constants of probability distributions, is often hard. Variants of importance sampling give unbiased estimates of a normalizer Z, however, unbiased estimates of the reciprocal 1/Z are harder to obtain. Unbiased estimates of 1/Z allow Markov chain Monte Carlo sampling of "d…
Paper introduces a new sampling method combining Consistency Models with importance sampling.
problem Inherent errors in samples and high NFEs for high-quality samples in Boltzmann distributions.
method Combines Consistency Models with importance sampling to produce unbiased samples with minimal NFEs.
result Produces unbiased samples using only 6-25 NFEs, comparable to 100 NFEs for DDPMs.
A new method speeds up sampling of Boltzmann distribution in high-dimensional systems.
problem High computational cost of obtaining Jacobian of flow-based models in high dimensions.
method Flow perturbation method that incorporates stochastic perturbations and reweighting.
result Achieves unbiased sampling of Boltzmann distribution with orders of magnitude speedup.
New algorithm finds unbiased subnetworks in biased datasets.
problem Finding unbiased subnetworks in biased neural networks.
method Debiased Contrastive Weight Pruning (DCWP) algorithm.
result DCWP significantly outperforms state-of-the-art debiasing methods.
Stochastic bridges are commonly used to impute missing data with a lower sampling rate to generate data with a higher sampling rate, while preserving key properties of the dynamics involved in an unbiased way. While the generation of Brownian bridges and Ornstein-Uhlenbeck bridges is well understood, unbiased generatio…
Improves survey sampling with unbiased machine learning methods.
problem Design-consistent model-assisted estimation lacks a general theory for machine learning.
method Proposes a subsampling Rao-Blackwell method for design-unbiased estimation.
result Yields efficiency gains over standard methods while ensuring valid estimation.
Improved UIVI method shows better performance than state-of-the-art SIVI methods.
problem Estimating the likelihood of samples from complex distributions in high dimensions.
method Replaced the inner MCMC loop of UIVI with importance sampling and learned the optimal proposal distribution.
result The refined UIVI approach demonstrates superior performance or parity with state-of-the-art methods.
New method makes machine learning approximations unbiased and efficient.
problem Efficient sampling of complex probability distributions.
method Uses autoregressive neural networks with cluster updates and physical symmetries.
result Shows unbiased and low-variance approximations for phase transitions.
SUMO provides unbiased log marginal likelihood estimation for latent variable models.
problem Biased estimates of log marginal likelihood in latent variable models.
method Randomized truncation of infinite series for unbiased estimation.
result Models trained with SUMO give better test-set likelihoods than standard methods.
In linear regression we wish to estimate the optimum linear least squares predictor for a distribution over d-dimensional input points and real-valued responses, based on a small sample. Under standard random design analysis, where the sample is drawn i.i.d. from the input distribution, the least squares solution for…
Improved learning theory for kernel distribution regression with two-stage sampling.
problem Distribution regression problem and two-stage sampling setting.
method Kernel methods, near-unbiased condition, new error bounds, convergence rates.
result Strictly improved convergence rates for three important classes of kernels.
Paper proposes an unbiased optimization method for Bayesian experimental design.
problem Maximizing expected information gain in Bayesian experimental design.
method Randomized multilevel Monte Carlo (MLMC) method combined with stochastic gradient descent.
result An unbiased estimator for the gradient of expected information gain.
Temporal networks representing a stream of timestamped edges are seemingly ubiquitous in the real-world. However, the massive size and continuous nature of these networks make them fundamentally challenging to analyze and leverage for descriptive and predictive modeling tasks. In this work, we propose a general framewo…
Clapping reduces memory usage in distributed optimization by reusing data samples.
problem Significant communication overhead and impractical memory overhead in pipeline-parallel distributed optimization.
method Lazy sampling strategy to reuse data samples across steps, supporting convergence without unbiased gradient assumptions.
result Clapping achieves convergence in few-epoch or online training regimes without sample-size memory overhead.
Quantum method generates unbiased samples from discrete graphical models.
problem Sampling from discrete graphical models is challenging and intractable in high dimensions.
method Embedding graphical models into unitary operators and using quantum circuits.
result Provably generates unbiased and independent samples from general discrete factor models.
CARMS improves gradient estimation for categorical variables.
problem Accurately backpropagating gradients through categorical variables.
method CARMS combines REINFORCE with antithetic sampling to create unbiased gradient estimators.
result CARMS outperforms competing methods on various tasks.
A weakly-supervised learning framework named as complementary-label learning has been proposed recently, where each sample is equipped with a single complementary label that denotes one of the classes the sample does not belong to. However, the existing complementary-label learning methods cannot learn from the easily …
ENSURE framework trains deep image recon algorithms without clean data.
problem Lack of clean, fully sampled ground-truth data for deep learning image reconstruction.
method Introduces ENSURE framework, a generalization of SURE and GSURE to random sampling patterns.
result ENSURE loss function is an unbiased estimate for true mean-square error.
Paper proposes unbiased learning for recommendation causal effects.
problem Estimating the causal effect of recommendation when the ground truth is unobservable.
method Inverse propensity scoring technique to construct unbiased estimators, followed by empirical risk minimization with propensity capping.
result The proposed method outperforms other biased learning methods in various settings.
This paper presents a methodology and workflow that overcome the limitations of the conventional Generative Adversarial Networks (GANs) for geological facies modeling. It attempts to improve the training stability and guarantee the diversity of the generated geology through interpretable latent vectors. The resulting s…
SA-GFN corrects biases in GFlowNets due to graph symmetries.
problem Systematic biases in state transition probability computations.
method Incorporates symmetry corrections into the learning process through reward scaling.
result Eliminates need for explicit state transition computations.
EB improves asset pricing by mining large strategies without lookahead bias.
problem Lack of unbiased asset pricing models with out-of-sample performance.
method Empirical Bayes applied to 136,000 long-short strategies.
result EB provides unbiased predictions with transparent intuition.
Improved method for unbiased causal discovery in presence of unobserved confounding.
problem Unbiased data synthesis for causal discovery algorithms in the presence of unobserved confounding.
method Explicit block-hierarchical ancestral sampling to address limitations of implicit parameterization.
result Our approach fully covers the space of causal models, including those generated by implicit parameterization.
New theory of sensitivity for unbiased estimators using Wasserstein geometry.
problem Estimating the instability of estimators under small perturbations.
method Developed a new theory based on Wasserstein geometry, analogous to classical Cramér-Rao theory.
result Wasserstein-Cramér-Rao lower bound for sensitivity of unbiased estimators.
Unbiased gradient estimation improves VAE performance.
problem Training VAEs via maximum likelihood is difficult due to intractable integrals.
method Introduced unbiased estimators of the log-likelihood gradient using coupled Markov chains.
result Unbiased estimators lead to better predictive performance in VAEs.
This paper tackles unbiased loss functions for multilabel classification with missing labels.
problem Missing labels in multilabel classification tasks, especially in extreme multi-label classification (XMC).
method Derives unbiased estimators for multilabel reductions, including non-decomposable ones, and addresses increased variance with convex upper-bounds.
result Switching to unbiased estimators can alter the bias-variance trade-off and may require stronger regularization.
New method uses rank-conditioned Horvitz-Thompson estimation for unbiased sample reuse in Plackett-Luce best-of-K objective.
problem Estimating the expected maximum reward in Plackett-Luce draws without replacement.
method Rank-conditioned Horvitz-Thompson estimation with joint-score REINFORCE for unbiased sample reuse.
result Unbiased estimation of the Plackett-Luce best-of-K objective with finite second moment guarantees.
Unbiased method for Bayesian posterior means using kinetic Langevin dynamics.
problem Estimating Bayesian posterior means efficiently and accurately.
method Combines advanced splitting methods with enhanced gradient approximations in a multilevel Monte Carlo approach.
result The method achieves unbiased estimates with finite variance and central limit theorem properties.
VT-DIS improves sampling from Boltzmann distributions with minimal overhead.
problem Bias in Monte Carlo estimates from score-based diffusion models.
method Variance-Tuned Diffusion Importance Sampling (VT-DIS) adapts noise covariance to correct bias.
result VT-DIS achieves effective sample sizes of 80%, 35%, and 3.5% on benchmarks, using less computational budget.
A promising class of generative models maps points from a simple distribution to a complex distribution through an invertible neural network. Likelihood-based training of these models requires restricting their architectures to allow cheap computation of Jacobian determinants. Alternatively, the Jacobian trace can be u…
Existing deep architectures cannot operate on very large signals such as megapixel images due to computational and memory constraints. To tackle this limitation, we propose a fully differentiable end-to-end trainable model that samples and processes only a fraction of the full resolution input image. The locations to p…
Develops unbiased averaging methods for second order optimization in distributed systems.
problem Computing the Hessian is challenging and communication is a bottleneck in distributed optimization.
method Unbiased parameter averaging methods using sampling and sketching of the Hessian.
result Provably minimizes bias for sketched Newton directions.
We propose a general framework for the estimation of observables with generative neural samplers focusing on modern deep generative neural networks that provide an exact sampling probability. In this framework, we present asymptotically unbiased estimators for generic observables, including those that explicitly depend…
New unbiased variance estimator for random forests using Hoeffding decomposition.
problem Uncertainty quantification in random forests with large kernel sizes and small sample sizes.
method Proposes a new Hoeffding decomposition view for variance estimation, establishing unbiased estimators and ratio consistency.
result Establishes the ratio consistency of the proposed variance estimator, justifying confidence interval coverage rates.
Proposes a sample selection algorithm for fair and robust AI training.
problem Balancing fairness and robustness in AI models, especially with corrupted data.
method Formulates and solves a combinatorial optimization problem for unbiased sample selection, proposing a greedy algorithm.
result Improves fairness and robustness compared to state-of-the-art techniques, both synthetically and on real datasets.
Log-linear models are arguably the most successful class of graphical models for large-scale applications because of their simplicity and tractability. Learning and inference with these models require calculating the partition function, which is a major bottleneck and intractable for large state spaces. Importance Samp…
In applications of Gaussian processes where quantification of uncertainty is of primary interest, it is necessary to accurately characterize the posterior distribution over covariance parameters. This paper proposes an adaptation of the Stochastic Gradient Langevin Dynamics algorithm to draw samples from the posterior …
Lipschitz continuity recently becomes popular in generative adversarial networks (GANs). It was observed that the Lipschitz regularized discriminator leads to improved training stability and sample quality. The mainstream implementations of Lipschitz continuity include gradient penalty and spectral normalization. In th…
Paper provides unbiased spectral moment estimates from finite data.
problem Challenges in estimating spectral moments from limited data.
method Dynamic programming approach to estimate spectral moments of kernel integral operator.
result Demonstrates consistency with theoretical spectra and practical utility in neural networks.
This paper presents a new approach, called perturb-max, for high-dimensional statistical inference that is based on applying random perturbations followed by optimization. This framework injects randomness to maximum a-posteriori (MAP) predictors by randomly perturbing the potential function for the input. A classic re…
Consider a process, stochastic or deterministic, obtained by using a numerical integration scheme, or from Monte-Carlo methods involving an approximation to an integral, or a Newton-Raphson iteration to approximate the root of an equation. We will assume that we can sample from the distribution of the process from time…
Estimates CDF over complex regions using normalizing flows.
problem Challenges in estimating CDF over complex regions using traditional methods.
method Leverages diffeomorphic properties of normalizing flows and divergence theorem.
result Improves sample efficiency in estimating CDF over complex regions.
New characterization limits sampling with inexact scores.
problem Limiting sampling with inexact scores for unbiased results.
method Characterized types of inexact score oracle access.
result Weaker error assumptions rule out tractability of unbiased sampling.
Positive-Unlabeled (PU) learning is an analog to supervised binary classification for the case when only the positive sample is clean, while the negative sample is contaminated with latent instances of positive class and hence can be considered as an unlabeled mixture. The objectives are to classify the unlabeled sampl…
Synthetic construction of 3D complex bases.
problem Creating a complete set of unbiased bases in 3D complex space.
method Synthetic construction using complex projective trigonometry.
result Synthetic construction of mutually unbiased bases in C^3.
A key quantity of interest in Bayesian inference are expectations of functions with respect to a posterior distribution. Markov Chain Monte Carlo is a fundamental tool to consistently compute these expectations via averaging samples drawn from an approximate posterior. However, its feasibility is being challenged in th…