BIS uses bandits to efficiently sample from expensive-to-evaluate densities.
problem Sampling from computationally expensive target densities.
method Sequential selection through multi-armed bandits, optimizing sample set directly.
result BIS achieves accurate sampling with fewer evaluations than adaptive methods.
New approaches estimate recommendation metrics using sampling.
problem Understanding and resolving the use of sampling for recommendation evaluation.
method MLE and ME principles for empirical rank distribution recovery.
result Advantages of new approaches for top-k metrics estimation.
The study addresses biases in evaluating molecular optimization methods and proposes methods to reduce these biases.
problem Biases in in silico evaluation of molecular optimization methods.
method Discussion and empirical investigation of bias reduction methods for predictor misspecification and sample reuse.
result Empirical investigation of bias reduction methods for predictor misspecification and sample reuse.
Unified evaluation framework for sampling methods.
problem Lack of a standardized evaluation framework for sampling methods.
method Introduces a benchmark suite and performance criteria for sampling methods.
result Insights into strengths and weaknesses of existing sampling methods.
We consider the problem of off-policy evaluation in Markov decision processes. Off-policy evaluation is the task of evaluating the expected return of one policy with data generated by a different, behavior policy. Importance sampling is a technique for off-policy evaluation that re-weights off-policy returns to account…
This paper evaluates targeted data poisoning attacks by focusing on the hardest samples, improving evaluation and defense strategies.
problem The effectiveness of targeted data poisoning attacks is often overestimated due to average evaluation methods.
method The paper introduces metrics to identify the hardest and easiest to poison samples based on clean model information.
result The proposed metrics reliably stratify samples by poisoning vulnerability, enabling rigorous worst-case evaluation and proactive defense.
Study improves off-policy evaluation from non-i.i.d. bandit samples.
problem Improving off-policy evaluation from non-independent bandit samples.
method Constructing an estimator from a standardized martingale difference sequence.
result Proposed estimator performs better than existing methods.
The paper evaluates variational auto-encoders using model criticism methods.
problem Evaluating the quality of variational auto-encoders (VAEs).
method Statistical model criticism, focusing on reproducing statistics of unknown data generating processes.
result The proposed framework offers possibilities for model selection beyond intrinsic metrics.
Cer-Eval saves LLM evaluation costs while maintaining accuracy.
problem Challenges in evaluating large language models due to large dataset requirements.
method Adapts to different evaluation objectives, uses test sample complexity, and develops a partition-based algorithm.
result Cer-Eval can save 20-40% test points with comparable accuracy and 95% confidence guarantee.
Evaluating generative adversarial networks (GANs) is inherently challenging. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the problem of how to evaluate the evaluation metrics. We start with a few necessary conditions for metrics to produce meaningful scores, su…
OSIRIS reduces variance in off-policy evaluation by omitting irrelevant states.
problem High variance in importance sampling-based OPE estimators.
method OSIRIS reduces variance by omitting likelihood ratios associated with states irrelevant to return.
result OSIRIS is unbiased and has lower variance than ordinary importance sampling.
Proposes GM Score to evaluate GANs considering diversity, disentanglement, and discriminability.
problem Evaluation of GANs for sample quality and diversity.
method Integrates various factors including intra-class and inter-class diversity, disentanglement, and discriminability metrics.
result Demonstrates improved evaluation of GANs on MNIST dataset.
EWFM trains continuous flows with only energy evaluations, improving sample quality with fewer computations.
problem Efficiently sampling from complex, high-dimensional Boltzmann distributions using only energy evaluations.
method Energy-Weighted Flow Matching (EWFM) using importance sampling and iterative/annealed training.
result Improved sample quality with up to 3 orders of magnitude fewer energy evaluations compared to existing methods.
Validates policies using past observational data with guarantees about out-of-sample performance.
problem Evaluating decision policies using past data observed under a different policy.
method Sample-splitting method to draw inferences about the entire loss distribution with finite-sample coverage guarantees.
result Valid inferences about out-of-sample loss with finite-sample coverage guarantees, accounting for model misspecifications.
Paper addresses OPE for dependent bandit samples using MDS and batch updates.
problem Evaluating policies from non-i.i.d. historical data in contextual bandits.
method Constructs an MDS-based estimator for dependent samples, solves batch update and deficient support issues.
result Derives an asymptotically normal estimator for evaluation policy value.
The paper tackles sampling bias in credit scoring models and proposes methods to improve their training and evaluation.
problem Sampling bias in credit scoring models leads to an incomplete representation of the borrower population.
method Bias-aware self-learning framework and Bayesian evaluation method to correct for bias.
result Bayesian evaluation outperforms standard accuracy measures in predicting future performance.
A new framework evaluates LLM calibration in open-ended QA.
problem Evaluating LLM calibration in open-ended QA settings.
method Sem-ECE framework: sampling answers, grouping by semantic classes, and using frequencies as confidence.
result Sem-ECE estimators are unbiased and Sem2 achieves smaller calibration error. Evolutionary algorithms (EAs) are a sort of nature-inspired metaheuristics, which have wide applications in various practical optimization problems. In these problems, objective evaluations are usually inaccurate, because noise is almost inevitable in real world, and it is a crucial issue to weaken the negative effect …
Study evaluates synthetic data augmentation for small datasets, highlighting inconsistencies in traditional metrics.
problem Inconsistent validation of synthetic data generated for small sample sizes.
method Proposes a normalized Bottleneck distance metric to evaluate synthetic tabular data.
result Common metrics like propensity scoring and MMD fail for small datasets, showing instability and high variability.
This guide explains statistical distances for evaluating generative models.
problem Evaluating the quality of samples from generative models.
method Four statistical distances: SW, C2ST, MMD, FID.
result Different distances can yield varying results on similar data.
A new first-order sampler improves diffusion probabilistic model sampling quality.
problem The belief that first-order methods are inherently slower for diffusion probabilistic model sampling.
method A novel training-free, first-order sampler that approximates the forward-value evaluation via a one-step lookahead predictor.
result The proposed sampler provably approximates the ideal forward-value trajectory while retaining first-order convergence and can improve sample quality under the same NFE budget.
mAIS improves free energy evaluation efficiency.
problem Computational infeasibility of exact free energy evaluation.
method mAIS, a marginalized version of AIS.
result mAIS is more efficient under certain conditions.
The paper analyzes the sample complexities for policy evaluation with linear function approximation.
problem Policy evaluation with linear function approximation in discounted infinite horizon Markov decision processes.
method Investigates sample complexities for two policy evaluation algorithms: TD and TDC.
result Establishes high-probability sample complexity bounds for policy evaluation algorithms.
Evaluation and validation of complicated control systems are crucial to guarantee usability and safety. Usually, failure happens in some very rarely encountered situations, but once triggered, the consequence is disastrous. Accelerated Evaluation is a methodology that efficiently tests those rarely-occurring yet critic…
A new metric evaluates generative models by comparing real and generated samples.
problem Evaluating the quality of generative models.
method Relative Density Ratio (RDR) function, optimization on variational form of φ-divergence.
result The RDR function provides a clear, interpretable, and numerically stable evaluation metric.
This paper improves sample efficiency for off-policy evaluation with preference data.
problem Improving sample efficiency for off-policy evaluation with preference data.
method Using a deep neural network to learn the value function and leveraging manifold structure.
result Established a provably efficient guarantee for off-policy evaluation with RLHF.
Zigzag sampling algorithm efficiently samples from strongly log-concave distributions with low computational cost.
problem Sampling from strongly log-concave distributions efficiently and with low computational complexity.
method Zigzag sampling algorithm with warm start assumption, focusing on gradient evaluations.
result Achieves ε error in chi-square divergence with computational cost of O(κ²d^(1/2)(log(1/ε))^(3/2)) gradient evaluations.
Active testing reduces label costs for efficient model evaluation.
problem Real-world applications require expensive test labels, disconnecting from existing model evaluation methods.
method Derives acquisition strategies to select test points efficiently, addressing label bias and variance.
result Active testing improves model evaluation efficiency without sacrificing accuracy.
PS-BAX uses posterior sampling to select evaluation points for efficient Bayesian algorithm execution.
problem Efficiently selecting evaluation points for expensive functions with limited evaluations.
method Posterior sampling to guide sequential selection of evaluation points.
result PS-BAX is faster, simpler, and more scalable than existing methods.
DPM-Solver speeds up DPM sampling to 10-20 function evaluations.
problem Slow sampling from Diffusion Probabilistic Models (DPMs).
method Exact formulation of diffusion ODE solutions, using change-of-variable and exponentially weighted integral.
result Generates high-quality samples in 10-20 function evaluations.
New metric evaluates generative models across domains, diagnosing fidelity, diversity, and generalization.
problem Evaluating generative models in diverse domains with limited metrics.
method Introduces a 3D evaluation metric (α-Precision, β-Recall, Authenticity) for domain-agnostic diagnostics. result Unified metric characterizes fidelity, diversity, and generalization, diagnosing model performance.
New method reduces diffusion model function evaluations for discrete data.
problem High computational burden in generating samples from masked diffusion models.
method Modified causal attention mask and speculative sampling mechanism for non-factorized predictions.
result Achieved ~2x reduction in required network forward passes.
The paper evaluates samplers on multi-modal targets, focusing on mode separation and recovery.
problem Handling multi-modality in sampling.
method Synthetic experimental setting focusing on mode relative importance recovery.
result Illustrates the challenges and potential of samplers in multi-modality.
Enhanced metrics improve generative model evaluation reliability.
problem Lack of reliable quality metrics for generative models.
method Introduce Clipped Density and Clipped Coverage metrics.
result Metrics prevent out-of-distribution samples from biasing quality scores.
Paper proposes a diagnostic tool for evaluating model performance out-of-sample.
problem Evaluating model performance on unseen data.
method Uses a finite calibration dataset to assess future losses.
result Provides guarantees under weak assumptions and quantifies distribution shifts.
This paper tackles label-efficient evaluation in extreme class imbalance.
problem Challenges in obtaining a sufficient sample for accurate evaluation in tasks with extreme class imbalance.
method Develops a framework for online evaluation based on adaptive importance sampling.
result Establishes strong consistency and a central limit theorem for performance estimates.
The performance of a machine learning system is usually evaluated by using i.i.d.\ observations with true labels. However, acquiring ground truth labels is expensive, while obtaining unlabeled samples may be cheaper. Stratified sampling can be beneficial in such settings and can reduce the number of true labels require…
ACE improves counterfactual explanations with fewer model queries.
problem Inefficient sampling for counterfactual explanations in machine learning models.
method Adaptive sampling combining Bayesian estimation and stochastic optimization.
result ACE achieves superior evaluation efficiency compared to state-of-the-art methods.
Study evaluates ML methods for two-sample testing with right-censored data.
problem Evaluating ML methods for two-sample testing with right-censored data.
method Developed and compared several ML-based methods with classical tests.
result Proposed methods outperform classical tests in terms of statistical power.
The paper develops methods to estimate treatment effects in sample selection models.
problem Evaluation of treatments when outcomes are only observed for a subpopulation due to sample selection or attrition.
method Combines selection-on-observables and instrumental variable assumptions with double machine learning for treatment evaluation.
result Proposed estimators are asymptotically normal and root-n consistent.
New method reduces sample complexity for robust reinforcement learning.
problem Finite sample analysis in robust reinforcement learning.
method Stochastic approximation framework with controlled bias, using MLMC techniques and geometric truncation.
result Order-optimal sample complexity of ildeO(ε−2) for robust policy evaluation. Adaptive exploration scheme for evaluating multiple policies with different rewards.
problem Online multi-reward multi-policy evaluation.
method Adapted (ε,δ)-PAC perspective and MR-NaS exploration scheme to minimize sample complexity. result Demonstrated effectiveness of adaptive exploration in tabular domains.
Paper finds efficient OPE estimator for multiple logging policies with minimum variance.
problem Finding optimal importance sampling weights for multiple logging policies with varying variances.
method Established efficiency bound under stratified sampling and proposed an estimator achieving this bound.
result Proposed estimator achieves minimum variance for any instance.
Generates multimodal safety-critical scenarios for robustness evaluation of decision-making algorithms.
problem Lack of comprehensive evaluation of neural network robustness under real-world scenarios.
method Proposes a flow-based multimodal scenario generator using weighted likelihood maximization and gradient-based sampling.
result Demonstrates improved testing efficiency and multimodal modeling capability compared to traditional methods.
Paper introduces a new sampling method combining Consistency Models with importance sampling.
problem Inherent errors in samples and high NFEs for high-quality samples in Boltzmann distributions.
method Combines Consistency Models with importance sampling to produce unbiased samples with minimal NFEs.
result Produces unbiased samples using only 6-25 NFEs, comparable to 100 NFEs for DDPMs.
R-AutoEval+ improves model evaluation efficiency and reliability using adaptive synthetic data.
problem Accurate model selection from AI candidates using real-world data is costly and impractical at scale.
method R-AutoEval+ uses adaptive prediction-powered inference to correct bias in autoevaluators while maintaining or improving sample efficiency.
result R-AutoEval+ provides finite-sample reliability guarantees and enhanced sample efficiency compared to conventional methods.
Proposes a method to refine PDE-driven high-dimensional rare-event simulation.
problem Challenges in constructing accurate surrogates for rare-event simulation.
method Adaptive importance sampling framework that refines a locally constructed surrogate.
result Achieves accuracy comparable to true-model adaptive importance sampling with fewer high-fidelity evaluations.
A new method combines AIS and SMCI for efficient evaluation of Ising models.
problem Efficiently evaluating expectations on Ising models under various temperatures.
method Combining Annealed Importance Sampling (AIS) and Spatial Monte Carlo Integration (SMCI).
result The proposed method performs efficiently in both high- and low-temperature regions.