Paper develops robust policy evaluation for reinforcement learning with outlier and heavy-tailed rewards.
problem Outlier contamination and heavy-tailed rewards in reinforcement learning.
method Develops a fully online robust policy evaluation procedure and efficient statistical inference.
result Establishes the Bahadur-type representation of the estimator and develops an online inference procedure.
The paper evaluates variational auto-encoders using model criticism methods.
problem Evaluating the quality of variational auto-encoders (VAEs).
method Statistical model criticism, focusing on reproducing statistics of unknown data generating processes.
result The proposed framework offers possibilities for model selection beyond intrinsic metrics.
This guide explains statistical distances for evaluating generative models.
problem Evaluating the quality of samples from generative models.
method Four statistical distances: SW, C2ST, MMD, FID.
result Different distances can yield varying results on similar data.
Cramming method evaluates learned policies from contextual bandits efficiently.
problem Evaluating final learned policies from contextual bandit algorithms.
method On-policy evaluation using a single pass of data, ensuring consistency and asymptotic normality.
result Cramming method reduces evaluation standard error by approximately 40% compared to off-policy methods.
Proposes a method to evaluate generalizability in causal inference models.
problem Lack of formal procedures to statistically evaluate generalizability in causal inference.
method Frugal parameterization to simulate from causal benchmarks, using mean and distributional regression methods.
result Ensures more realistic evaluations of causal inference models, avoiding over-reliance on conventional metrics.
New method efficiently evaluates policies using trajectory data.
problem Statistically efficient policy evaluation with limited data.
method Trajectory-based approach for policy evaluation.
result Improved sample complexity for policy evaluation.
The paper shows how shared random seeds can reduce variance in machine learning evaluations.
problem The statistical structure of comparative evaluation under shared random seeds is not well understood.
method An extended learning-based multi-agent economic simulator was used to demonstrate the effects of shared random seeds on variance reduction.
result Pairing seeds can reduce variance in machine learning evaluations, especially when outcomes are positively correlated at the seed level.
How can we measure whether a natural language generation system produces both high quality and diverse outputs? Human evaluation captures quality but not diversity, as it does not catch models that simply plagiarize from the training set. On the other hand, statistical evaluation (i.e., perplexity) captures diversity b…
Deep RL evaluation underestimates uncertainty, leading to misleading conclusions.
problem Statistical uncertainty in deep RL performance evaluations is underestimated, leading to misleading conclusions.
method Advocates for reporting interval estimates of aggregate performance and proposes performance profiles to account for variability.
result Substantial discrepancies in prior performance comparisons are revealed, highlighting the need for more rigorous evaluation methods.
This paper reviews off-policy evaluation methods in reinforcement learning.
problem Efficiency and accuracy of off-policy evaluation methods in reinforcement learning.
method Discussion of existing OPE methods, their statistical properties, and related research directions.
result Efficiency bounds and state-of-the-art OPE methods in reinforcement learning.
Paper proposes a sequential statistical test for comparing imitation learning policies with near-optimal stopping.
problem Challenges in rigorously comparing imitation learning policies due to small sample sizes and potential p-hacking.
method Sequential statistical test that adapts the number of trials based on intermediate results, achieving near-optimal stopping.
result Reduces the number of evaluation trials by up to 32% compared to state-of-the-art baselines, saving significant time and effort.
The article reviews scoring rules for estimating and evaluating forecasts.
problem Evaluating probabilistic forecasts and estimating probability distributions.
method Mathematical foundations and characterization of scoring rules.
result Important families of scoring rules and their applications in statistics and machine learning.
Novel Orlicz regrets consistently bound environmental variable statistics.
problem Consistent evaluation of stochastic environmental variables like water quality indices.
method Proposed novel Orlicz regrets for upper and lower bounds.
result Explicit linkage between Orlicz regrets and divergence risk measures.
Bayesian approach quantifies uncertainty in LLM evaluations.
problem Statistical uncertainty in evaluating LLM behavior.
method Bayesian evaluation of LLM behavior using probabilistic text generation strategies.
result Bayesian approach provides useful uncertainty quantification about LLM behavior.
Unified approach for quantum and classical learning from evaluation oracles.
problem Learning from evaluation oracles in quantum and classical settings.
method Inspired by Kearns' SQ and Valiant's weak evaluation oracle, a unified framework is established.
result Characterizes query complexity for learning linear function classes and extends learnability results for quantum circuits.
The paper proposes a new evaluation framework for causal inference models.
problem Challenges in estimating causal effects from observational data.
method Complements evaluation of causal inference models with statistical evidence and non-parametric tests.
result Eliminates the influence of a few instances or simulations on benchmarking results.
Proposes a novel path generation and evaluation method for video games.
problem Generating and evaluating realistic navigation paths for video games.
method Combines nonparametric model-free transformations and copula models.
result Demonstrates precise and interpretable generation of diverse navigation paths.
Researchers use DL and XAI to evaluate climate downscaling models.
problem Evaluating complex DL models for climate downscaling.
method Intercompare DL models, expand standard evaluation methods with XAI.
result XAI techniques provide new evaluation dimensions and model insights.
New method uses model comparison signals to improve LLM evaluation accuracy.
problem Limited benchmark sizes and model stochasticity in evaluating LLMs' mathematical reasoning.
method Combines standard labeled outcomes with model comparison signals to design a statistically efficient evaluation framework.
result Semiparametric estimator achieves the semiparametric efficiency bound and substantially improves ranking accuracy.
This paper tackles label-efficient evaluation in extreme class imbalance.
problem Challenges in obtaining a sufficient sample for accurate evaluation in tasks with extreme class imbalance.
method Develops a framework for online evaluation based on adaptive importance sampling.
result Establishes strong consistency and a central limit theorem for performance estimates.
The paper evaluates and improves uncertainty estimates in neural networks for safety-critical applications.
problem Quantifying uncertainty in neural networks for safety-critical systems.
method Proposes a statistical test for evaluating uncertainty realism in neural networks and transfers a classification architecture to image-to-image tasks.
result The variational U-Net architecture significantly improves uncertainty realism in image-to-image tasks compared to a plain model.
Paper improves bootstrapping for off-policy reinforcement learning inference.
problem Improving bootstrapping for off-policy reinforcement learning inference.
method Proposes a bootstrapping FQE method for off-policy statistical inference and a subsampling procedure to improve runtime.
result Asymptotically efficient and distributionally consistent bootstrapping FQE method for off-policy inference.
LLMs fail to match statistical ground truth despite stable run-to-run performance.
problem LLMs lack validation against statistical ground truth in automated scientific workflows.
method Introduced a behavioral evaluation framework for LLMs, separating four decision-making dimensions.
result LLMs can exhibit near-perfect stability but diverge from statistical ground truth.
The paper evaluates index-based allocation policies using data from randomized control trials.
problem Evaluating index-based allocation policies in resource-scarce scenarios.
method Using data from randomized control trials, the paper introduces an efficient estimator and methods for computing asymptotically correct confidence intervals.
result Valid statistical conclusions can be drawn for index-based allocation policies.
Paper uses K-NN resampling to simulate and evaluate LOB markets.
problem Simulating and evaluating limit order book (LOB) markets.
method Applies K-nearest neighbor (K-NN) resampling to LOB simulation and evaluation. result Demonstrates the effectiveness and efficiency of K-NN resampling in LOB simulation and evaluation. Proposes a modified Morgan-Pitman test for evaluating variances in machine learning models.
problem Limited ability to account for sampling variability in model selection.
method Enhances the classic Morgan-Pitman test for robustness in non-linear models with heavy-tailed distributions or outliers.
result Demonstrates the test's effectiveness and practical utility in model evaluation and selection.
Study uses online bootstrap for RL inference, showing effectiveness.
problem Statistical inference for RL parameters in online settings.
method Online bootstrap method applied to TD and GTD algorithms in RL.
result Method is distributionally consistent for policy evaluation inference.
This work uses statistical bootstrapping to provide accurate confidence intervals for policy value in reinforcement learning.
problem Bias in estimating policy value using empirical transitions and rewards.
method Statistical bootstrapping to produce calibrated confidence intervals for the true policy value.
result Statistical bootstrapping can yield correct confidence intervals under certain conditions, and mechanisms are proposed to mitigate these conditions.
Suitability filter detects model performance degradation in real-world deployment.
problem Ensuring model reliability in safety-critical domains without access to ground truth labels.
method Uses suitability signals to evaluate classifier performance on unlabeled user data.
result The suitability filter reliably detects performance deviations due to covariate shift.
FAQ efficiently evaluates LLMs with statistical guarantees using adaptive query selection.
problem Efficiently evaluating many LLMs on a large suite of benchmarks is expensive.
method FAQ uses Bayesian factor models, adaptive sampling, and proactive active inference to select queries.
result FAQ delivers up to 5x effective sample size gains over baselines, matching CI width with fewer queries.
This paper compares and evaluates methods for evaluating statistical models using benchmarking data and simulations.
problem Choosing between benchmarking data sets and simulation studies for method comparison studies.
method Borrowing ideas from mixed methods research and Clinical Scenario Evaluation, the paper investigates and develops new approaches to evaluate methods.
result Develops new approaches to evaluate methods by combining the strengths of benchmarking data sets and simulation studies.
The CLT fails for LLM evaluations with small data, leading to underestimation of uncertainty.
problem Inaccurate uncertainty estimates in LLM evaluations with small datasets.
method Alternative frequentist and Bayesian methods for uncertainty quantification.
result CLT-based methods underestimate uncertainty in small data settings.
The study evaluates financial risk using copulas and statistical tests.
problem Validating bivariate forecasts in risk evaluation.
method Using copulas to characterize dependencies, applying statistical tests to validate forecasts, removing heteroskedasticity.
result A Student copula accurately describes financial time series dependencies.
This work characterizes conditions for offline policy evaluation in reinforcement learning.
problem Understanding when classical methods succeed in offline policy evaluation for linear function approximation.
method Control-theoretic and linear-algebraic conditions for classical methods (FQI and LSTD).
result A precise hierarchy of regimes under which these estimators succeed, and a complete picture of their behavior.
New method for distributional off-policy evaluation using Bellman residual minimization.
problem Learning return distribution from offline data generated by a different policy.
method Energy Bellman Residual Minimizer (EBRM) method.
result Established finite-sample error bound for EBRM estimator.
mAIS improves free energy evaluation efficiency.
problem Computational infeasibility of exact free energy evaluation.
method mAIS, a marginalized version of AIS.
result mAIS is more efficient under certain conditions.
Proposes a new measure to evaluate stability of statistical parameters under distributional shifts.
problem Difficulty in transferring knowledge across data sets due to distributional changes.
method Introduces a measure of instability quantifying sensitivity of statistical parameters to Kullback-Leibler divergence and directional shifts.
result The proposed measure can elucidate the type of shifts a parameter is sensitive to and improve estimation accuracy under shifted distributions.
This research improves calorimeter simulations by creating a faster model.
problem Efficiently simulating detailed calorimeter data for high-energy physics.
method Developed a conditional normalizing flow model for superresolution.
result The model successfully reproduces reference distributions.
Our research extends the Bilingual Evaluation Understudy (BLEU) evaluation technique for statistical machine translation to make it more adjustable and robust. We intend to adapt it to resemble human evaluation more. We perform experiments to evaluate the performance of our technique against the primary existing evalua…
Study improves distributional regression evaluation with CRPS, finding optimal rates of convergence.
problem Improving probabilistic forecasts in meteorology using distributional regression.
method Extends theoretical properties of CRPS evaluation to include covariates and finite sample sizes, analyzing convergence rates for different methods.
result Optimal minimax rate of convergence for distributional regression methods is achieved by k-nearest neighbor and kernel methods.
Statistical test evaluates if personalizing interventions is cost-effective.
problem Balancing the benefits of personalizing interventions with their potential costs.
method Developed a statistical hypothesis test to assess the performance of personalized interventions.
result The test shows that personalized interventions can outperform standard approaches under certain conditions.
Proposes incorporating noise sources in machine learning evaluation for more reliable conclusions.
problem Inadequate handling of nondeterminism in machine learning research leads to unreliable results.
method Uses linear mixed effects models (LMEMs) and generalized likelihood ratio tests (GLRT) to analyze performance evaluation scores and assess performance differences.
result Demonstrates how to incorporate various sources of noise and data properties into statistical significance testing and reliability analysis.
Statistical framework improves LLM chatbot ranking.
problem Improving evaluation of LLM-based chatbots through pairwise comparisons.
method Factored tie model, covariance modeling, and parameter constraints.
result Substantial improvements in modeling pairwise comparison data.
Developing state-of-the-art approaches for specific tasks is a major driving force in our research community. Depending on the prestige of the task, publishing it can come along with a lot of visibility. The question arises how reliable are our evaluation methodologies to compare approaches? One common methodology to i…
This paper tackles sandbagging in AI safety evaluations.
problem AI agents may hide dangerous capabilities to avoid being deactivated.
method Developed a simple model of strategic deception in sequential decision-making tasks.
result Demonstrated that optimal rational agents exhibit sandbagging behavior.
Generative model improves intraday electricity price forecasting.
problem Intraday electricity price forecasting for improved trading strategies.
method Generative neural network model for probabilistic path forecasts.
result Generative model leads to higher profit gains than benchmark methods.
Framework assesses autograders' reliability and biases.
problem Mixed reliability and biases in autograders for LLM evaluation.
method Bayesian GLMs to model evaluation outcomes.
result Explicit quantification of scoring differences and biases.
Our paper improves uplift model evaluation on randomized controlled trials (RCT) data.
problem Variance in uplift evaluation metrics makes their signals arbitrary and unreliable.
method Theoretical analysis and statistical adjustment of the outcome to reduce variance.
result Variance reduction methods improve uplift evaluation metrics on RCT data.