Survey on importance weighting in machine learning applications.
problem Distribution shift in supervised learning.
method Weighting objective function or probability distribution based on instance importance.
result Importance weighting can guarantee desirable statistical properties in distribution shift scenarios.
Adaptive importance sampling is a class of techniques for finding good proposal distributions for importance sampling. Often the proposal distributions are standard probability distributions whose parameters are adapted based on the mismatch between the current proposal and a target distribution. In this work, we prese…
A new variable importance measure for DRFs detects broader impacts on output distributions.
problem Estimating full conditional distributions of multivariate outputs given inputs.
method Based on the drop and relearn principle and MMD distance.
result Consistent and high-performing variable importance measure for DRFs.
Develops method to assess feature importance in black-box models for unconditional distribution.
problem Lack of methods to analyze feature importance in black-box models for unconditional distribution.
method Approximation method to compute feature importance curves for unconditional distribution.
result Produces sparse and faithful results, computationally efficient.
Importance weighting is a general way to adjust Monte Carlo integration to account for draws from the wrong distribution, but the resulting estimate can be highly variable when the importance ratios have a heavy right tail. This routinely occurs when there are aspects of the target distribution that are not well captur…
The paper improves importance sampling and MCMC methods for complex distributions.
problem Improving sampling efficiency for distributions with atoms or heavy tails.
method Develops minimax optimal trial distributions and importance-tempered MCMC.
result Importance-tempered MCMC can be uniformly ergodic for certain distributions.
FIT evaluates time series model feature importance quantifying distributional shift.
problem Lack of explanations for time series models in high-stakes applications.
method FIT framework quantifies feature importance based on distributional shift using KL-divergence.
result FIT identifies important time points and observations superiorly compared to baselines.
Framework improves gradient estimation for faster training convergence.
problem Efficiently estimating noisy gradients in stochastic optimization.
method Dynamic adaptive importance sampling combining multiple distributions.
result Adaptively weighted multiple importance sampling yields superior gradient estimates.
Sharp analysis of out-of-distribution error in overparameterized models with importance weights.
problem Understanding and quantifying the degradation of performance in overparameterized models when faced with underrepresented data.
method Sharp analysis of an overparameterized Gaussian mixture model with spurious features and cost-sensitive interpolating solutions incorporating importance weights.
result Characterization of a novel tradeoff between worst-case robustness and average accuracy as a function of importance weight magnitude.
New framework quantifies variable importance across all good models and is stable across data distribution.
problem Conflicting variable importance conclusions from different models trained on the same data.
method Proposes a new variable importance framework that considers all good models and is stable across data distribution.
result Framework accurately estimates true variable importance and recovers rankings for complex setups.
BIF assesses feature importance using Dirichlet distribution and Bayesian inference.
problem Quantitative feature importance assessment in statistical models.
method Utilizes Dirichlet distribution for probabilistic feature importance assessment via approximate Bayesian inference.
result Learned importance provides relative significance and confidence quantification of features.
Proposes a differentiable hypergeometric distribution for learning group importance.
problem Learning the sizes of subsets in applications like clustering and weakly-supervised learning.
method Introduces a reparameterizable hypergeometric distribution to model group sizes and learn their relative importance.
result Outperforms previous methods in weakly-supervised learning and clustering.
DAIS minimizes symmetrized KL divergence between initial and target distributions.
problem Optimizing over initial distributions in importance sampling.
method Differentiable annealed importance sampling (DAIS) minimizing symmetrized KL divergence.
result DAIS minimizes symmetrized KL divergence between initial and target distributions.
FAB combines flows with AIS to approximate complex distributions.
problem Challenges in flow-based methods, especially on complex targets.
method Combines flows with AIS, using α-divergence for training.
result FAB produces accurate approximations to complex distributions.
AIS uses a suboptimal extended target distribution, which this paper improves using SGM.
problem Improving the efficiency of Annealed Importance Sampling for marginal likelihood estimation.
method Leveraging score-based generative modeling to approximate the optimal extended target distribution.
result Demonstrated novel, differentiable AIS procedures on synthetic and real-world data.
Adaptive sampling method improves efficiency in complex target distributions.
problem Efficiency of importance sampling in complex target distributions, especially multimodal distributions in high-dimensional spaces.
method Proposes an adaptive scheme combining global sampling with delayed weighting to promote efficient exploration of target distributions.
result The proposed algorithm is geometrically convergent under mild assumptions and demonstrates improved efficiency in various numerical experiments.
New loss function restores importance weighting in overparameterized models.
problem Restoring importance weighting in overparameterized neural networks.
method Introduced polynomially-tailed losses to restore effects of importance weighting.
result Polynomially-tailed losses improve performance in correcting distribution shift.
Low-rank MPPCA improves importance sampling in high dimensions.
problem Estimating full-rank GMM covariance matrices in high dimensions is numerically unstable.
method Use MPPCA mixtures as low-rank proposals for importance sampling in high-dimensional spaces.
result Consistent gains in sample efficiency and quality of failure distribution characterization.
Efficiently estimates online variational learning using importance sampling.
problem Online variational estimation in state-space models.
method Variational approach with Monte Carlo importance sampling.
result Proposed efficient algorithm for streaming data.
Importance-weighting is a popular and well-researched technique for dealing with sample selection bias and covariate shift. It has desirable characteristics such as unbiasedness, consistency and low computational complexity. However, weighting can have a detrimental effect on an estimator as well. In this work, we empi…
The standard interpretation of importance-weighted autoencoders is that they maximize a tighter lower bound on the marginal likelihood than the standard evidence lower bound. We give an alternate interpretation of this procedure: that it optimizes the standard variational lower bound, but using a more complex distribut…
Proposes a new method to find features affecting treatment effect distribution.
problem Existing methods fail to detect differences in treatment effect distribution parameters other than the mean.
method Formulates and estimates a feature importance measure that quantifies feature influence on potential outcome distribution discrepancies. Develops a feature selection algorithm to control type I error rate.
result Successfully discovers important features and outperforms existing mean-based methods.
Improves transfer learning by weighting importance based on test-over-training density.
problem Distribution shift in training and test data.
method Joint and dynamic importance-predictor estimation, causal mechanism transfer.
result Enhanced transfer learning performance in complex, high-dimensional tasks.
Importance sampling is often used in machine learning when training and testing data come from different distributions. In this paper we propose a new variant of importance sampling that can reduce the variance of importance sampling-based estimates by orders of magnitude when the supports of the training and testing d…
Unified view of LR and RP gradients with improved importance sampling.
problem Understanding and optimizing gradient estimators in machine learning.
method First principles explanation of LR and RP, divergence theorem, optimal importance sampling schemes.
result Optimal importance sampling schemes derived with analytic probability densities.
BR-SNIS reduces bias in self-normalized IS without increasing variance.
problem Bias in self-normalized IS.
method Iterated sampling-importance resampling (ISIR) to form a bias-reduced estimator.
result Significant reduction in bias without increasing variance.
Improves model calibration and selection in unsupervised domain adaptation.
problem Distribution shifts in unsupervised domain adaptation.
method Developed a novel importance weighted group accuracy estimator.
result Improves state-of-the-art performances by 22% in model calibration and 14% in model selection.
Mitigates anomaly score imbalance in long-tailed distributions.
problem Class imbalance in normal data leads to skewed anomaly detection performance.
method Proposes an importance-weighted loss function to balance anomaly scores.
result Improves anomaly detection performance by 0.043 on real-world datasets.
A new method estimates rare events using tensor trains.
problem Estimating rare event probabilities in high-dimensional problems.
method Approximating optimal importance distribution via tensor-train decompositions and compositions.
result Better variance reduction and efficient computation of rare event probabilities.
LFIS uses a time-dependent velocity field to sample from complex distributions.
problem Sampling from unnormalized density functions.
method LFIS learns a time-dependent velocity field to transport samples from a simple initial distribution to a complex target distribution.
result LFIS achieves state-of-the-art performance on various benchmark problems.
Paper proposes DDA for better transfer learning performance.
problem Domain discrepancy between source and target distributions.
method Dynamic Distribution Adaptation (DDA) to evaluate and adapt distribution importance.
result DDA improves transfer learning performance on various tasks.
We propose a novel adaptive importance sampling algorithm which incorporates Stein variational gradient decent algorithm (SVGD) with importance sampling (IS). Our algorithm leverages the nonparametric transforms in SVGD to iteratively decrease the KL divergence between our importance proposal and the target distributio…
Sampling is an important tool for estimating large, complex sums and integrals over high dimensional spaces. For instance, important sampling has been used as an alternative to exact methods for inference in belief networks. Ideally, we want to have a sampling distribution that provides optimal-variance estimators. In …
Deep generative networks can simulate from a complex target distribution, by minimizing a loss with respect to samples from that distribution. However, often we do not have direct access to our target distribution - our data may be subject to sample selection bias, or may be from a different but related distribution. W…
Improves overparameterized models' robustness to distribution shifts.
problem Accuracy drop on testing distributions different from training.
method Importance tempering to improve decision boundaries.
result State-of-the-art results on worst group classification tasks.
Off-policy learning exhibits greater instability when compared to on-policy learning in reinforcement learning (RL). The difference in probability distribution between the target policy (π) and the behavior policy (b) is a major cause of instability. High variance also originates from distributional mismatch. The var…
In recent years, a large amount of model-agnostic methods to improve the transparency, trustability and interpretability of machine learning models have been developed. We introduce local feature importance as a local version of a recent model-agnostic global feature importance method. Based on local feature importance…
PRoFILE accurately estimates feature importance under distribution shifts.
problem Challenges in estimating feature importance under distribution shifts.
method PRoFILE uses a loss estimator trained with a causal objective to estimate feature importance.
result PRoFILE accurately estimates feature importance under complex distribution shifts.
New algorithm improves accuracy of importance weights for diverse applications.
problem Improving accuracy of importance weights for various applications.
method Formulated multicalibrated partitions and developed an efficient algorithm.
result Algorithm significantly improves accuracy of importance weights.
Corrects distribution shift in target shift scenarios using importance weighting.
problem Analyzes importance weighting for correcting distribution shift under target shift.
method Analyzed importance-weighted kernel ridge regression under target shift.
result Shows that importance weighting corrects the train-test mismatch without altering input-space complexity.
CR-AIS improves AIS efficiency by constant rate annealing.
problem Efficiently sample from intractable distributions.
method Constant rate annealing schedule for AIS.
result CR-AIS outperforms existing Adaptive AIS methods.
AIS algorithm improves heavy-tailed distribution estimation.
problem Inconsistent estimators and slow convergence in AIS for heavy-tailed distributions.
method Adapts Student-t proposal distributions by matching escort moments and minimizing α-divergence.
result Improves estimation accuracy for heavy-tailed distributions.
Algorithm calibrates predictions for covariate shift using domain adaptation.
problem Uncertainty estimates overestimate certainty when real-world data differs from training data.
method Uses importance weighting and learns a feature map to equalize distributions.
result Outperforms existing approaches in calibrated prediction when covariate shift occurs.
As energy markets begin clearing at sub-hourly rates, their interaction with load control systems becomes a potentially important consideration. A simple model for the control of thermal systems using market-based power distribution strategies is proposed, with particular attention to the behavior and dynamics of elect…
A new method for deep learning under distribution shift by iteratively refining importance weighting.
problem Handling distribution shift in deep learning models when training and test data distributions differ.
method Dynamic Importance Weighting (dynamic IW) that iterates between weight estimation and weighted classification, using a pre-trained feature extractor and stochastic optimization.
result Dynamic IW outperforms state-of-the-art methods in experiments with various types of distribution shift on multiple datasets.
DistShap parallelizes GNN explanation for large graphs.
problem Computational expense in attributing GNN predictions to specific edges or features.
method Distributed Shapley values across multiple GPUs for scalable GNN explanations.
result DistShap outperforms existing methods and scales to models with millions of features.
We develop a new method to estimate failure probabilities in complex systems.
problem Estimating failure probabilities in safety-critical autonomous systems is challenging due to the rarity of failures and large state spaces.
method We propose an adaptive importance sampling algorithm that minimizes forward Kullback-Leibler divergence and uses Markov score ascent methods.
result Our method provides more accurate failure probability estimates than existing techniques.
VT-DIS improves sampling from Boltzmann distributions with minimal overhead.
problem Bias in Monte Carlo estimates from score-based diffusion models.
method Variance-Tuned Diffusion Importance Sampling (VT-DIS) adapts noise covariance to correct bias.
result VT-DIS achieves effective sample sizes of 80%, 35%, and 3.5% on benchmarks, using less computational budget.