Paper addresses off-policy evaluation and learning with covariate shift.
problem Evaluating and training a new policy using historical data with a covariate shift.
method Derives efficiency bounds and proposes doubly robust estimators for OPE and OPL under covariate shift.
result Proposes estimators for off-policy evaluation and learning under covariate shift.
This paper evaluates and validates cluster results using external and internal evaluation methods.
problem Evaluating and validating the quality of clustering results.
method External evaluation using Homogeneity, Correctness, and V-measure scores; internal evaluation using Silhouette Index and Sum of Square Errors.
result Validation of the number of clusters using dendrogram and statistical frequency distribution.
Study evaluates saliency maps on artificial data with different backgrounds.
problem Objective evaluation of saliency methods on artificial data with varying backgrounds.
method Developed a framework to generate artificial data with synthetic lesions and a known ground truth map, evaluated two data sets with different backgrounds (Perlin noise and 2D brain MRI slices).
result Heatmaps vary strongly between saliency methods and backgrounds.
CONE evaluates treatment assignment functions using networked observational data to mitigate hidden confounding bias.
problem Evaluate treatment assignment functions using networked observational data with hidden confounders.
method CONE framework that learns partial representations of latent confounders and combines them for counterfactual evaluation.
result Network information mitigates hidden confounding bias in counterfactual evaluation.
Proposes clustering as a new evaluation method for clinical knowledge embedding.
problem Traditional Link Prediction evaluation protocol loses information and harms model accuracy.
method Proposes Clustering Evaluation Protocol as an alternative.
result Experimental results show the proposed protocol can potentially replace Link Prediction.
The performance of deep neural networks crucially depends on good hyperparameter configurations. Bayesian optimization is a powerful framework for optimizing the hyperparameters of DNNs. These methods need sufficient evaluation data to approximate and minimize the validation error function of hyperparameters. However, …
New evaluation methods for causal models improve algorithm performance assessment.
problem Primitive evaluation techniques limit understanding of causal modeling algorithms.
method Use interventional measures and empirical data for evaluation.
result Evaluation techniques produce significantly different results than structural measures.
Toolkit evaluates causal inference models for real-world data.
problem Validating causal inference models for observational data.
method Developed a comprehensive evaluation toolkit for machine learning models in causal inference.
result Improves reproducibility and ensures data extraction criteria drive model selection.
Paper evaluates synthetic retail data for fidelity, utility, and privacy.
problem Ensuring accurate synthetic data in retail.
method Differentiates between continuous and discrete data, measures fidelity and utility, and uses Differential Privacy for privacy.
result Validated framework for reliable and scalable synthetic data evaluation.
Paper applies NEAT for dynamic credit evaluation using streaming data.
problem Dynamic credit evaluation using streaming data.
method Neuroevolution of Augmenting Topologies (NEAT) with enhancements.
result NEAT effectively handles dynamic credit evaluation with streaming data.
Combines trial and observational data to improve policy evaluation.
problem External validity of randomized trial results in target populations.
method Uses covariate data to model trial sampling and certifies policy evaluations.
result Valid trial-based policy evaluations under model miscalibration.
New protocol evaluates synthetic data for temporal consistency.
problem Synthetic data generators can produce invalid timestamps and trajectories.
method Characterize datasets by four properties, then measure timestamp validity and dynamics.
result Temporal fidelity must be measured, not inferred from static data.
A framework evaluates synthetic tabular data quality objectively.
problem Lack of an objective interpretation of tabular data metrics.
method Proposes a single mathematical objective for synthetic tabular data distribution, structurally decomposes it, and unifies existing metrics.
result Synthesizers that represent tabular structure outperform other methods, especially on smaller datasets.
Evaluates change point detection algorithms on real-world data.
problem Insufficient evaluation of change point detection algorithms on real-world time series.
method Developed a data set of 37 time series from various domains, annotated by human experts, and evaluated 14 algorithms using consistency metrics.
result Demonstrates the need for better evaluation methods in change point detection.
This paper evaluates targeted data poisoning attacks by focusing on the hardest samples, improving evaluation and defense strategies.
problem The effectiveness of targeted data poisoning attacks is often overestimated due to average evaluation methods.
method The paper introduces metrics to identify the hardest and easiest to poison samples based on clean model information.
result The proposed metrics reliably stratify samples by poisoning vulnerability, enabling rigorous worst-case evaluation and proactive defense.
Our paper improves uplift model evaluation on randomized controlled trials (RCT) data.
problem Variance in uplift evaluation metrics makes their signals arbitrary and unreliable.
method Theoretical analysis and statistical adjustment of the outcome to reduce variance.
result Variance reduction methods improve uplift evaluation metrics on RCT data.
The paper evaluates variational auto-encoders using model criticism methods.
problem Evaluating the quality of variational auto-encoders (VAEs).
method Statistical model criticism, focusing on reproducing statistics of unknown data generating processes.
result The proposed framework offers possibilities for model selection beyond intrinsic metrics.
Study finds AUC is most consistent across different prevalence in binary classification.
problem Consistency of model evaluation metrics across varying prevalence in binary classification.
method Analysis of 156 data scenarios with 18 metrics, 5 models, and a random guess model.
result AUC has the smallest variance in evaluating individual models and ranking of models.
Combines curvature descriptors with TDA for graph model evaluation.
problem Evaluating graph generative models efficiently and accurately.
method Combines graph curvature descriptors with topological data analysis.
result Robust, expressive descriptors for graph generative models.
SureMap estimates model performance across subpopulations efficiently.
problem Estimating model performance across subpopulations with scarce data.
method Simultaneous Gaussian mean estimation with external data.
result High accuracy in both multi-task and single-task disaggregated evaluations.
New framework for interpreting disaggregated fairness evaluations using causal models.
problem Misinterpretation of disaggregated fairness evaluations due to data representativeness and selection bias.
method Causal graphical models to characterize fairness properties and metric stability under different data generating processes.
result Disaggregated evaluations are unreliable without explicit assumptions regarding bias mechanisms.
Paper assesses uncertainty in safety evaluation of self-driving tech.
problem Estimating uncertainty in safety-critical event probabilities.
method Combines bootstrap method and likelihood ratio scheme for input uncertainty assessment.
result Demonstrates the impact of data variability on safety evaluation estimates.
New framework evaluates MIA without retraining, addressing biases.
problem Evaluate MIA without retraining and under distribution shift.
method Causal inference approach to MIA evaluation.
result Practical estimators for MIA metrics without retraining.
DUET optimizes LLM training data mixtures using feedback from unseen tasks.
problem Optimizing LLM performance on unseen tasks without prior data knowledge.
method Interleaves influence function with Bayesian optimization for data selection.
result DUET converges to optimal training data mixture for unseen tasks.
Recommendation systems have been integrated into the majority of large online systems. They tailor those systems to individual users by filtering and ranking information according to user profiles. This adaptation process influences the way users interact with the system and, as a consequence, increases the difficulty …
Proposes a new framework to evaluate causal discovery methods for time series data.
problem Lack of ground truth for causal discovery in time series data.
method Flexible framework for generating synthetic time series data.
result Demonstrates degradation in performance when assumptions are violated.
Paper develops a method to highlight influential transitions for off-policy evaluation.
problem Validating off-policy evaluation in high-stakes settings.
method Highlight influential transitions, develop influence functions for Q-evaluation.
result Method can identify limitations and make evaluation more robust.
Paper tackles policy selection with logged data and limited online interactions.
problem Safe evaluation and deployment of offline reinforcement learning policies.
method Active offline policy selection combining logged data with online interaction.
result Improves upon state-of-the-art OPE estimates and pure online policy evaluation.
Paper develops streaming algorithms to estimate classifier accuracy on unlabeled data.
problem Estimating classifier accuracy on unlabeled data with noisy decisions.
method Two algebraic evaluators: majority voting and a novel method to handle correlated classifiers.
result The novel method can be as accurate as 1% when handling small amounts of correlation.
Develops a method to evaluate OPE robustness to hyperparameters and policies.
problem Difficulty in selecting and tuning OPE estimators due to limited experimental evaluations.
method Introduces IEOE (Interpretable Evaluation for Offline Evaluation) to assess robustness.
result Demonstrates improved evaluation of OPE estimators' reliability.
The paper improves DR estimators for off-policy evaluation in contextual bandits.
problem Challenges in evaluating policies with adaptively collected data.
method Adaptive weighting to control variance in DR estimators.
result Improved t-statistic based on the estimator is asymptotically normal.
CrossLID evaluates GAN quality by comparing intrinsic dimensions.
problem Measuring GAN quality and detecting mode collapse.
method CrossLID measures local intrinsic dimensionality of real-world data vs. GAN-generated samples.
result CrossLID correlates with GAN training progress and is robust to noise.
New methods evaluate data representations by complexity of low-loss predictor learning.
problem Evaluating quality of data representations for downstream tasks.
method Surplus Description Length (SDL) and ε Sample Complexity (εSC) methods.
result Methods measure the information needed to approximate optimal predictor up to specified tolerance.
The paper tackles sampling bias in credit scoring models and proposes methods to improve their training and evaluation.
problem Sampling bias in credit scoring models leads to an incomplete representation of the borrower population.
method Bias-aware self-learning framework and Bayesian evaluation method to correct for bias.
result Bayesian evaluation outperforms standard accuracy measures in predicting future performance.
New method efficiently evaluates policies using trajectory data.
problem Statistically efficient policy evaluation with limited data.
method Trajectory-based approach for policy evaluation.
result Improved sample complexity for policy evaluation.
This paper improves sample efficiency for off-policy evaluation with preference data.
problem Improving sample efficiency for off-policy evaluation with preference data.
method Using a deep neural network to learn the value function and leveraging manifold structure.
result Established a provably efficient guarantee for off-policy evaluation with RLHF.
R-AutoEval+ improves model evaluation efficiency and reliability using adaptive synthetic data.
problem Accurate model selection from AI candidates using real-world data is costly and impractical at scale.
method R-AutoEval+ uses adaptive prediction-powered inference to correct bias in autoevaluators while maintaining or improving sample efficiency.
result R-AutoEval+ provides finite-sample reliability guarantees and enhanced sample efficiency compared to conventional methods.
Combines experimental and historical data for robust policy evaluation.
problem Policy evaluation with mixed data sources, especially experimental vs historical.
method Linear integration of estimators from experimental and historical data, optimized for MSE minimization.
result Proposed estimators outperform traditional methods in ridesharing company data.
Machine learning suffers from poor design, data, and evaluation practices.
problem Ad hoc design, poor data hygiene, and lack of statistical rigor in model evaluation.
method Examines the entire machine learning process from design to evaluation, highlighting common pitfalls and providing recommendations.
result Common pitfalls in machine learning research and development are identified and actionable recommendations are provided.
Presents SPEED, an algorithm for optimal policy evaluation in linear bandits with heteroscedastic noise.
problem Optimal data collection for policy evaluation in linear bandits with heteroscedastic reward noise.
method Formulated an optimal design for weighted least squares estimates, derived the optimal sample allocation, introduced SPEED algorithm, and derived regret bounds.
result SPEED leads to policy evaluation with MSE comparable to oracle strategy and significantly lower than random policy execution.
Paper introduces metrics to evaluate missing data imputation without ground truth.
problem Handling missing data in time series without ground truth.
method Introduces Wasserstein distance (WD) and Jensen-Shannon divergence (JSD) as metrics to evaluate imputation quality.
result WD and JSD are effective metrics for assessing missing data imputation quality.
Proposes standards for evaluating online machine learning methods in evolving data streams.
problem Difficulty in evaluating online machine learning methods under realistic conditions.
method Proposes comprehensive evaluation standards, performance measures, and evaluation strategies.
result Provides a new Python framework (float) for modular integration of libraries and custom code.
This paper compares and evaluates methods for evaluating statistical models using benchmarking data and simulations.
problem Choosing between benchmarking data sets and simulation studies for method comparison studies.
method Borrowing ideas from mixed methods research and Clinical Scenario Evaluation, the paper investigates and develops new approaches to evaluate methods.
result Develops new approaches to evaluate methods by combining the strengths of benchmarking data sets and simulation studies.
Paper uses LightGBM for mobile user credit assessment.
problem Improving credit evaluation methods for communication operators.
method Data preprocessing, feature engineering, multiple machine learning models integration.
result Established a suitable fusion model for operator user credit evaluation.
This study evaluates clustering algorithms on high-dimensional data.
problem Comparing clustering algorithms on high-dimensional datasets.
method Evaluation of K-means, DBSCAN, and Spectral Clustering using PCA, t-SNE, UMAP, and multiple metrics.
result UMAP preprocessing improves clustering quality across all algorithms, with Spectral Clustering excelling.
This research evaluates measures of dependence for financial time-series data.
problem Accurately preparing time series data and selecting an appropriate measure of dependence is challenging.
method Review and establishment of a comprehensive analysis framework for shaping time-series data and evaluating measures of dependence.
result A method, framework, and example for selecting and evaluating a suitable measure of dependence are presented.
The paper introduces negative controls to evaluate causal discovery algorithms, improving their reliability.
problem Lack of a general guideline for evaluating causal discovery algorithms.
method Derive exact distributional results under random guessing for evaluation metrics and propose a pipeline for using negative controls.
result Evaluation metrics can achieve very favorable values under random guessing, highlighting the need for negative control results.
Proposes DCV-ROOD framework for robust OOD detection evaluation.
problem Ensuring reliable OOD detection methods under diverse conditions.
method Dual Cross-Validation (DCV) adapted for OOD detection evaluation.
result Demonstrates fast convergence to true performance of OOD detection methods.