This work shows how evaluation metrics can be seen as fair gambles.
problem The relationship and evaluation of machine learning forecasts.
method Using game-theoretic probability, the authors show evaluation metrics as fair gambles.
result Standard evaluation metrics are fair gambler outcomes, with calibration and regret metrics on two dimensions.
Decision-alignment evaluates uncertainty quantification for decision-relevant UQ
problem Evaluation of uncertainty quantification metrics
method Introduce decision-alignment
result Proper scoring rules align with decision utility
CCE improves anomaly detection metrics by measuring both confidence and consistency.
problem Existing anomaly detection metrics lack discriminative power, hyperparameter dependency, and robustness to perturbations.
method CCE uses Bayesian estimation to quantify uncertainty and constructs global and event-level confidence and consistency scores.
result CCE demonstrates strict boundedness, robustness, and linear time complexity.
Study categorizes time series anomaly detection metrics based on evaluation challenges.
problem Challenges in evaluating time series anomaly detection due to diverse application objectives and metric assumptions.
method Problem-oriented framework categorizing metrics into six dimensions based on evaluation challenges.
result Quantifies each metric's discriminative ability and reveals limitations of widely used metrics.
The paper introduces new metrics for evaluating generative models of behavior.
problem Lack of quantitative evaluation criteria for unsupervised behavior discovery.
method Proposed and investigated several metrics for generative models of behavior.
result The proposed metrics correspond with biologists' intuitions and allow for model evaluation and bias understanding.
EAST aligns neural network classifiers with user-defined evaluation metrics.
problem Mismatch between neural network training and evaluation metrics leads to suboptimal performance.
method EAST uses dynamic thresholding, soft-set confusion matrix, and annealing to align neural network predictions with target evaluation metrics.
result EAST improves alignment between training objectives and evaluation metrics, outperforming existing methods.
Topic models are typically evaluated with respect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic assignments. Token-level assignments are important for downstream tasks such as classification. Even recent models, which aim to improv…
Interactive tool helps choose and understand classification metrics.
problem Common metrics for binary classification have limitations.
method Graphical application to visualize and explore evaluation metrics.
result Promotes careful attention to interpretation of metrics.
New approaches estimate recommendation metrics using sampling.
problem Understanding and resolving the use of sampling for recommendation evaluation.
method MLE and ME principles for empirical rank distribution recovery.
result Advantages of new approaches for top-k metrics estimation.
Study finds AUC is most consistent across different prevalence in binary classification.
problem Consistency of model evaluation metrics across varying prevalence in binary classification.
method Analysis of 156 data scenarios with 18 metrics, 5 models, and a random guess model.
result AUC has the smallest variance in evaluating individual models and ranking of models.
This paper evaluates and improves metrics for identifying important features in machine learning models.
problem Evaluation metrics for explainable AI are limited by multicollinearity and model accuracy.
method Proposes Expected Accuracy Interval (EAI) to predict model accuracy with multicollinearity.
result EAI is a useful metric for identifying important features in models with multicollinearity.
Study flaws in generative model evaluation metrics, especially for diffusion models.
problem Flaws in existing metrics for evaluating generative models, particularly for diffusion models.
method Systematic study of generative models, human perception experiments, and analysis of feature extractors.
result State-of-the-art perceptual realism of diffusion models is not reflected in commonly reported metrics.
With success on controlled tasks, generative models are being increasingly applied to humanitarian applications [1,2]. In this paper, we focus on the evaluation of a conditional generative model that illustrates the consequences of climate change-induced flooding to encourage public interest and awareness on the issue.…
Develops RES metrics for stable rare-event forecasting evaluation.
problem Challenges in evaluating forecasts of rare events.
method Rare-event-stable (RES) metrics designed to maintain stable thresholds under extreme rarity.
result RES metrics maintain stable thresholds, consistent model rankings, and near-complete prevalence invariance.
Study reveals gaps between simulated and real-world treatment effect evaluation metrics.
problem Evaluation of treatment effect estimation models differs between academic and practical settings.
method Comprehensive empirical study comparing semi-simulated benchmarks and real-world datasets.
result Counterfactual metrics do not reliably predict observable metrics, and rankings from simulated benchmarks do not generalize to real-world data.
Study improves machine learning models for GI tract disease detection using comprehensive evaluations and cross-dataset testing.
problem Incomplete or incorrect evaluation of machine learning models for GI tract diseases.
method Comprehensive evaluations of five machine learning models using Global Features and Deep Neural Networks, introducing performance hexagons and cross-dataset testing.
result Demonstrates the need for more sophisticated performance metrics and evaluation methods to build generalizable models.
Proposes a method to choose thresholds for LLM evaluation metrics.
problem Ensuring reliable large language models (LLMs) with correct threshold selection.
method Identify risks, stakeholders' risk tolerance, and use ground-truth data to determine thresholds.
result Demonstrates a concrete example with the Faithfulness metric and HaluBench dataset.
In most machine learning training paradigms a fixed, often handcrafted, loss function is assumed to be a good proxy for an underlying evaluation metric. In this work we assess this assumption by meta-learning an adaptive loss function to directly optimize the evaluation metric. We propose a sample efficient reinforceme…
CAT framework improves AI medical screening fairness and reliability.
problem Imbalanced data, varying performance across cohorts, and patient-level inconsistencies in traditional metrics.
method CAT framework introduces patient-level assessment, entropy-based distribution weighting, and cohort-weighted sensitivity and specificity.
result Enhanced predictive reliability, fairness, and interpretability of AI-driven medical screening models.
Evaluating the style of handwriting generation is a challenging problem, since it is not well defined. It is a key component in order to develop in developing systems with more personalized experiences with humans. In this paper, we propose baseline benchmarks, in order to set anchors to estimate the relative quality o…
In machine learning, the choice of a learning algorithm that is suitable for the application domain is critical. The performance metric used to compare different algorithms must also reflect the concerns of users in the application domain under consideration. In this work, we propose a novel probability-based performan…
Paper proposes a new metric to evaluate survival models, especially for censored data.
problem Challenges in evaluating survival prediction models due to censored data.
method Developed a novel approach to estimate Mean Absolute Error (MAE) for survival datasets with censored data.
result The proposed MAE metric using pseudo-observations accurately ranks model performance and closely matches true MAE.
New metric reduces estimation error in survival model evaluation.
problem Dependent censoring complicates survival model evaluation.
method Dependent Brier score based on Archimedean copula and Copula-Graphic estimator.
result Reduces estimation error by 12-16% on average.
Evaluation metrics for prediction models don't fully reflect intervention impact.
problem Standard metrics don't accurately reflect reduction in patient outcomes from model use.
method Synthesized and discussed various evaluation methods, analyzed with simulated and real data.
result Evaluations without interventional data are limited or require strong assumptions.
Evaluating generative adversarial networks (GANs) is inherently challenging. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the problem of how to evaluate the evaluation metrics. We start with a few necessary conditions for metrics to produce meaningful scores, su…
A framework evaluates synthetic tabular data quality objectively.
problem Lack of an objective interpretation of tabular data metrics.
method Proposes a single mathematical objective for synthetic tabular data distribution, structurally decomposes it, and unifies existing metrics.
result Synthesizers that represent tabular structure outperform other methods, especially on smaller datasets.
PolyGraph Discrepancy improves graph generative model evaluation.
problem Inability of existing metrics to provide an absolute performance measure and comparability across different graph descriptors.
method Approximates Jensen-Shannon distance using binary classifiers trained to distinguish between real and generated graphs.
result PGD provides a more robust and insightful evaluation compared to MMD metrics.
New metrics improve regression evaluation across different data distributions.
problem Difficulty in comparing regression evaluations across datasets with varying distributions.
method Modification of regression metrics by weighting with the inverse distribution of function values or samples using a Gaussian kernel density estimator.
result New metrics are less sensitive to changing distributions, especially when correcting by the marginal distribution in X. New metric evaluates generative models across domains, diagnosing fidelity, diversity, and generalization.
problem Evaluating generative models in diverse domains with limited metrics.
method Introduces a 3D evaluation metric (α-Precision, β-Recall, Authenticity) for domain-agnostic diagnostics. result Unified metric characterizes fidelity, diversity, and generalization, diagnosing model performance.
Unified approach optimizes neural network training for various metrics.
problem Training and evaluation of neural network binary classifiers often use different metrics.
method Combines differentiable approximation and probabilistic soft sets.
result Effective in optimizing for metrics like F1-Score across various domains.
Paper explores how text generation quality and diversity metrics relate to distribution fitting.
problem Unclear relation between text generation quality and diversity metrics and distribution fitting.
method Theoretical approach to prove a linear combination of quality and diversity metrics can be a divergence metric.
result CR/NRR proposed as a better substitute for BLEU/Self-BLEU metrics.
Paper introduces metrics to evaluate missing data imputation without ground truth.
problem Handling missing data in time series without ground truth.
method Introduces Wasserstein distance (WD) and Jensen-Shannon divergence (JSD) as metrics to evaluate imputation quality.
result WD and JSD are effective metrics for assessing missing data imputation quality.
IRT metrics improve model evaluation by assessing latent characteristics.
problem Limitations of classic metrics like precision and F1.
method Introducing psychometric metrics like Item Response Theory (IRT).
result IRT complements classical metrics, offering new insights.
This work evaluates and benchmarks calibration metrics for data-driven regression models.
problem Conflicting results from different calibration metrics make it hard to compare and interpret model performance.
method Systematically extracted and benchmarked 14 regression calibration metrics across various data types and recalibration methods.
result Many metrics disagree on the same recalibration result, highlighting the need for careful metric selection.
Spatially-aware metrics improve uncertainty evaluation in segmentation.
problem Uncertainty evaluation metrics treat voxels independently, ignoring spatial context.
method Proposed three spatially aware metrics incorporating structural and boundary information.
result Improved alignment with clinically important factors and better discrimination between uncertainty patterns.
New metrics needed for streaming ML due to delayed labels.
problem Streaming ML evaluation fails to identify unexpected performance.
method Recommend additional metrics for streaming ML performance.
result New metrics are needed for streaming ML due to delayed labels.
State representation learning aims at learning compact representations from raw observations in robotics and control applications. Approaches used for this objective are auto-encoders, learning forward models, inverse dynamics or learning using generic priors on the state characteristics. However, the diversity in appl…
This paper reviews metrics for evaluating multi-class classification models.
problem Evaluating and comparing multi-class classification models.
method Review and analysis of metrics.
result Promising multi-class metrics are highlighted and their usages are demonstrated.
Devising indicative evaluation metrics for the image generation task remains an open problem. The most widely used metric for measuring the similarity between real and generated images has been the Fréchet Inception Distance (FID) score. Because it does not differentiate the fidelity and diversity aspects of the genera…
This paper evaluates metrics for graph generative models, addressing common pitfalls.
problem Evaluating and comparing graph generative models effectively.
method Systematic evaluation of MMD, analysis of synthetic and real graphs, practical recommendations.
result MMD can be problematic; practical solutions are provided.
New method reduces variance in subpopulation model performance estimates.
problem High variance in subpopulation performance metrics for small groups.
method Using an evaluation model to form model-based metric (MBM) estimates.
result MBMs produce more accurate and lower variance estimates for small subpopulations.
Proposes RDASS for better Korean text summarization evaluation.
problem ROUGE scores fail to capture semantic meaning in Korean text summarization.
method Introduces RDASS metrics and a method to improve their correlation with human judgment.
result RDASS metrics correlate better with human judgment than ROUGE scores.
Study evaluates relevance metrics for similarity-based model explanations.
problem Providing understandable explanations for complex model predictions.
method Evaluated three relevance metrics using three tests.
result Cosine similarity of gradients performs best for explanations.
New method uses contrastively trained GNNs for more reliable graph model evaluation.
problem Need effective methods to evaluate Graph Generative Models.
method Use representations from contrastively trained Graph Neural Networks (GNNs) for evaluation.
result Contrastively trained GNNs provide more reliable evaluation metrics than traditional or GNN-based approaches.
Enhanced metrics improve generative model evaluation reliability.
problem Lack of reliable quality metrics for generative models.
method Introduce Clipped Density and Clipped Coverage metrics.
result Metrics prevent out-of-distribution samples from biasing quality scores.
New method improves consistency of reinforcement learning performance evaluations.
problem Inconsistent performance results in reinforcement learning due to flawed evaluation metrics.
method Proposes a new comprehensive evaluation methodology for reinforcement learning algorithms.
result Demonstrates improved reliability of performance measurements for reinforcement learning algorithms.
New metrics CWSA and CWSA+ improve model evaluation under confidence thresholds.
problem Lack of metrics capturing model reliability under confidence thresholds.
method Introducing CWSA and CWSA+ metrics that reward confident accuracy and penalize overconfident mistakes.
result CWSA and CWSA+ outperform classical metrics in trust-sensitive tests.
Building an open-domain conversational agent is a challenging problem. Current evaluation methods, mostly post-hoc judgments of static conversation, do not capture conversation quality in a realistic interactive context. In this paper, we investigate interactive human evaluation and provide evidence for its necessity; …