A computational theory reduces agent evaluation errors and speeds up processes.
problem Efficient evaluation of mini agents at reduced cost.
method Developed a computational theory and a meta-learner to handle heterogeneous agents.
result Reduced evaluation errors by 24.1% to 99.0% across various scenarios.
A new framework evaluates HTE estimators using relative error.
problem Lack of robust evaluation methods for HTE estimators.
method Proposes a relative error-based evaluation framework and neural network architecture to estimate nuisance parameters and robustly compare HTE estimators.
result Demonstrates reliable comparisons and improved HTE estimation through the proposed framework and learning algorithm.
Study off-policy evaluation in partially observable environments, reducing bias and errors.
problem Bias and large errors in off-policy evaluation for partially observable environments.
method Defined and solved off-policy evaluation for POMDPs, introduced Decoupled POMDP model.
result Demonstrated and compared off-policy evaluation methods, showing benefits of new approach.
AHS framework selects hyperparameters for FQE with error guarantees.
problem Hyperparameter selection for FQE is challenging and affects its utility.
method AHS framework defines optimality criteria without hyperparameters.
result Error bounds match empirical observations.
Paper improves off-policy evaluation by estimating behavior policy.
problem Evaluating policies with data from a different behavior policy.
method Importance sampling with an estimated behavior policy.
result Estimating behavior policy reduces mean squared error.
Researchers compute Bayes error for classification models using normalizing flows.
problem Evaluating the inherent difficulty of classification problems.
method Invertible transformations and Gaussian base distributions to compute Bayes error.
result State-of-the-art models can achieve near-optimal accuracy but not always.
This paper compares two clustering evaluation metrics, revealing their differences and properties.
problem Understanding the differences between misclassification error distance and adjusted Rand index.
method Population origins, data analysis examples, detailed case studies, and simulation study.
result Reveals previous misconceptions about the two metrics and their distributions.
Large language models correlate in errors, even with different architectures and providers.
problem Lack of empirical evidence on whether different large language models differ meaningfully.
method Empirical evaluation of over 350 large language models using two leaderboards and a resume-screening task.
result Large language models have substantial correlation in errors, even with distinct architectures and providers.
Study error bounds in evaluating distributional computational graphs.
problem Error analysis in evaluating graphs with inputs as probability distributions.
method Establish non-asymptotic error bounds using Wasserstein-1 distance.
result Non-asymptotic error bounds for discretization errors in distributional computational graphs.
Neural networks for stock price prediction often misrepresent model performance due to flawed error metrics.
problem Flawed prediction error metrics lead to unreliable model evaluations in the securities market.
method Used data from 20 stock datasets across multiple markets and evaluated with four prediction error measures.
result Prediction error value only partially reflects model accuracy and fails to represent stock price direction.
Develops a method to find costly high-confidence errors in black box models.
problem Finding rare high-confidence errors missed by random sampling.
method Adversarial perturbation-guided search technique to find errors at rates greater than expected given model confidence.
result Our Adversarial Distance search discovers high-confidence errors at a rate greater than expected given model confidence.
New method stabilizes FQE by reweighting Bellman targets.
problem Stability guarantees for FQE often rely on Bellman completeness, which can fail with function approximation.
method Proposes stationary-weighted FQE, reweighting Bellman targets by stationary target-to-behavior density ratio.
result Proves finite-sample linear convergence to stationary projected Bellman fixed point without Bellman completeness.
Unified NICEk metrics improve solar forecasting accuracy.
problem Lack of suitable error metrics for multidimensional solar irradiance forecasting.
method Introducing NICEk framework with Lk norms for evaluating forecasting models.
result NICESigma consistently outperforms traditional metrics in discriminative power and statistical significance.
This research evaluates generalization measures in deep learning.
problem Understanding why deep learning models generalize well despite small training error.
method Empirical evaluation of generalization bounds and measures.
result Generalization measures should be evaluated using distributional robustness.
Recently, \citet{SuttonMW15} introduced the emphatic temporal differences (ETD) algorithm for off-policy evaluation in Markov decision processes. In this short note, we show that the projected fixed-point equation that underlies ETD involves a contraction operator, with a γ \sqrtγ γ -contraction modulus (where γ γ γ is the …
The paper analyzes cross-validation for correlated data and introduces a bias-corrected estimator.
problem Cross-validation with squared error loss assumes independent and identically distributed (i.i.d.) data, which is often violated in correlated data.
method The paper presents a criterion for standard CV suitability and introduces a bias-corrected estimator ( C V c CV_c C V c ) for correlated data. result The bias-corrected estimator ( C V c CV_c C V c ) yields an unbiased estimate of prediction error in settings where standard CV is invalid. Paper introduces a new method for error estimation in classification tasks with limited data.
problem Challenges in designing accurate classifiers and evaluating their performance with limited training data.
method Introduces a novel Bayesian MMSE estimator for optimal Bayesian transfer learning (OBTL) using Monte Carlo importance sampling.
result Proposed OBTL error estimation scheme outperforms standard methods, especially in small-sample settings.
FEAST improves FNN's range accuracy in ocean waveguides.
problem Evaluating FNN's range accuracy on unlabeled test data in ocean waveguides.
method Introducing FEAST to evaluate and improve FNN's range accuracy.
result FEAST helps improve FNN's ranging accuracy on test data.
New approach to model evaluation and selection using generalization error minimization.
problem Addressing concerns about the external validity of models through generalization ability.
method Generalization error minimization (GEM) framework for model selection, unifying penalized regression estimators.
result Established finite-sample and asymptotic properties of the GEM estimator, including consistency.
Unified framework for error quantification in off-policy evaluation with distributional shift.
problem Establishing high-confidence CI for target policy value from offline data.
method Unified error analysis quantifying misspecification and sampling errors.
result Achieves tightest possible CI and robustness against distributional shifts.
Unified evaluation for both quality and diversity in NLP.
problem Measuring both quality and diversity in NLP models.
method Proposes HUSE, a metric combining human and statistical evaluation.
result HUSE detects both quality and diversity defects in NLP models.
Combines experimental and historical data for robust policy evaluation.
problem Policy evaluation with mixed data sources, especially experimental vs historical.
method Linear integration of estimators from experimental and historical data, optimized for MSE minimization.
result Proposed estimators outperform traditional methods in ridesharing company data.
Researchers propose a new SSL risk decomposition method to evaluate and improve self-supervised learning models.
problem Self-supervised learning evaluation is limited to a single metric, providing little insight into model performance and improvement.
method Proposes an SSL risk decomposition that considers four error components: approximation, representation usability, probe generalization, and encoder generalization.
result Analysis of 169 SSL vision models reveals the main sources of error and provides insights for improving SSL models in specific settings.
This paper optimizes off-policy evaluation in reinforcement learning with function approximation.
problem Estimating cumulative value of a new policy from logged data generated by an unknown policy.
method Regression-based fitted Q iteration method, equivalent to estimating conditional mean embedding of transition operator.
result The method is minimax-optimal, with nearly minimal estimation error.
The paper evaluates forecast accuracy of realized volatility measures in large cross-sections.
problem Forecast evaluation of realized volatility measures in large cross-sections of financial data.
method Equal predictive accuracy testing procedures, LASSO shrinkage, measurement error correction, cross-sectional jump component measures.
result The augmented HAR model outperforms the standard HAR model in forecasting realized volatility.
RBI improves RL policies by reducing evaluation errors and avoiding performance degradation.
problem Evaluation errors in Q-function learning cause policy improvement penalties.
method RBI attenuates low-probability actions to minimize improvement penalties.
result RBI reduces regret and improves data efficiency in RL tasks.
A new framework for semi-supervised ordinal regression.
problem Lack of evaluation metrics and theoretical guarantees in existing semi-supervised ordinal regression.
method Empirical risk minimization principle, flexible model choices, and estimation error bound.
result Consistent risk estimator and improved performance across various metrics.
New research shows calibration error is flawed when dealing with model uncertainty.
problem Current model evaluation techniques conflate model uncertainty with aleatoric uncertainty.
method Posterior predictive checks to evaluate deep learning models.
result Calibration error and variants are incorrect when model uncertainty is present.
Generative AI reduces IR evaluation costs but introduces errors; this work provides reliable CIs.
problem Generating relevance annotations using AI introduces errors that affect IR evaluation metrics.
method Proposes two methods: prediction-powered inference and conformal risk control to place reliable CIs around IR metrics.
result Proposed methods accurately capture both variance and bias in evaluation based on AI-generated annotations.
Second-order estimator improves continuous-time policy evaluation.
problem Estimating value surfaces from discrete data with time-inhomogeneous dynamics.
method Moment-matching coefficients for high-order generator regression.
result Second-order estimator consistently outperforms Bellman baseline.
PredictaBoard benchmarks LLM score predictors to assess their ability to anticipate errors.
problem Inconsistent performance of LLMs in common sense reasoning tasks.
method Collaborative benchmarking framework evaluating pairs of LLMs and assessors using rejection rate at different tolerance errors.
result Highlights the need to evaluate predictability alongside performance for safer AI systems.
New algorithm optimally evaluates policies with linear approximations.
problem Policy evaluation with linear function approximation.
method Accelerated, variance-reduced fast temporal difference algorithm (VRFTD).
result VRFTD matches both deterministic and stochastic lower bounds.
Paper evaluates squared-exponential covariance function for Gaussian processes with integral observations.
problem Evaluating double line integrals of the squared exponential covariance function in Gaussian processes.
method Proposes a new approach to reduce double integrals to a single integral using the error function and efficiently computed with numerical techniques.
result Shows superior numerical robustness and accuracy compared to existing methods.
The study sets lower bounds on MMSE for inferring sensitive features from noisy data.
problem Estimating sensitive features from noisy observations of correlated features.
method Adversarial evaluation framework based on MMSE estimation with theoretical lower bounds.
result Derives closed-form bounds for linear models, showing optimality in noise variance.
Sparse feature selection improves batch RL efficiency.
problem High-dimensional batch RL with many features.
method Sparse linear function approximation, Lasso, group Lasso, fitted Q-evaluation, fitted Q-iteration.
result Sparse feature selection makes batch RL more sample efficient.
Proposes an alternative method for quantifying uncertainty in complex models.
problem Quantifying uncertainty in complex models and evaluations.
method Infinitesimally regularizes the training loss to assess downstream uncertainty.
result Provides reliable quantification of uncertainty and calibrated confidence intervals.
This paper bounds errors in data-driven power grid models using Rademacher complexity.
problem Ensuring accuracy of data-driven power grid models under incomplete physical information.
method Rademacher complexity theory for error bounds and evaluation implementation.
result Generalization error bounds for branch flow linearization and external network equivalent models.
The study corrects measurement error in evaluating health effects of multiple pollutants.
problem Bias in estimating health effects of air pollution constituents due to mismeasurement.
method Used a linear regression calibration model and extended DML approach to correct for measurement error.
result Identified two PM2.5 constituents (Br and Mn) that show a negative causal effect on cognitive function after correction.
Combines parametric and nonparametric models for better off-policy evaluation.
problem Improving off-policy evaluation in reinforcement learning.
method Mixture-of-experts approach combining parametric and nonparametric models.
result Mixture-based approach outperforms individual models and state-of-the-art estimators.
We use learning curves to analyze deep networks and evaluate model design.
problem Evaluate design choices in deep networks.
method Propose a method to robustly estimate learning curves, abstract their parameters, and evaluate different parameterizations.
result Interesting observations on the effectiveness of different parameterizations.
Paper develops streaming algorithms to estimate classifier accuracy on unlabeled data.
problem Estimating classifier accuracy on unlabeled data with noisy decisions.
method Two algebraic evaluators: majority voting and a novel method to handle correlated classifiers.
result The novel method can be as accurate as 1% when handling small amounts of correlation.
New method for distributional off-policy evaluation using Bellman residual minimization.
problem Learning return distribution from offline data generated by a different policy.
method Energy Bellman Residual Minimizer (EBRM) method.
result Established finite-sample error bound for EBRM estimator.
Study evaluates 13 CI methods for generalization error across 19 problems.
problem Evaluating prediction model quality using confidence intervals for generalization error.
method Compared 13 CI methods on 19 tabular regression and classification problems using cross-validation and bootstrapping.
result Identified a subset of methods with good coverage, width, and runtime.
A new method estimates Bayes error for deep networks, suggesting they may have reached the limit.
problem Evaluating the performance of deep learning models and detecting overfitting.
method A simple and direct Bayes error estimator based on uncertainty of class assignments.
result Deep networks may have reached the Bayes error limit for benchmark datasets.
New metric reduces estimation error in survival model evaluation.
problem Dependent censoring complicates survival model evaluation.
method Dependent Brier score based on Archimedean copula and Copula-Graphic estimator.
result Reduces estimation error by 12-16% on average.
Study examines error correction in ECG segmentation neural networks.
problem Improving accuracy of ECG segmentation using neural networks.
method Training deep convolutional neural networks for ECG segmentation and analyzing ensemble errors.
result Outliers in ensemble can be used to evaluate data representation quality.
This post introduces model calibration and evaluation measures, highlighting issues with a common measure.
problem Ensuring model confidence accurately reflects true outcomes.
method Explains common calibration definition, ECE, and its drawbacks.
result New evaluation measures needed for comprehensive model calibration.
Proposes a method to evaluate generalizability in causal inference models.
problem Lack of formal procedures to statistically evaluate generalizability in causal inference.
method Frugal parameterization to simulate from causal benchmarks, using mean and distributional regression methods.
result Ensures more realistic evaluations of causal inference models, avoiding over-reliance on conventional metrics.