This work shows how evaluation metrics can be seen as fair gambles.
problem The relationship and evaluation of machine learning forecasts.
method Using game-theoretic probability, the authors show evaluation metrics as fair gambles.
result Standard evaluation metrics are fair gambler outcomes, with calibration and regret metrics on two dimensions.
Decision-alignment evaluates uncertainty quantification for decision-relevant UQ
problem Evaluation of uncertainty quantification metrics
method Introduce decision-alignment
result Proper scoring rules align with decision utility
CCE improves anomaly detection metrics by measuring both confidence and consistency.
problem Existing anomaly detection metrics lack discriminative power, hyperparameter dependency, and robustness to perturbations.
method CCE uses Bayesian estimation to quantify uncertainty and constructs global and event-level confidence and consistency scores.
result CCE demonstrates strict boundedness, robustness, and linear time complexity.
Study categorizes time series anomaly detection metrics based on evaluation challenges.
problem Challenges in evaluating time series anomaly detection due to diverse application objectives and metric assumptions.
method Problem-oriented framework categorizing metrics into six dimensions based on evaluation challenges.
result Quantifies each metric's discriminative ability and reveals limitations of widely used metrics.
New metric correlates local topic quality with human judgments.
problem Evaluation of topic models focuses on global metrics, ignoring token-level assignments.
method Proposed a human evaluation task and automated metrics to assess local topic quality.
result Consistency metric correlates best with human judgments of local topic quality.
The paper introduces new metrics for evaluating generative models of behavior.
problem Lack of quantitative evaluation criteria for unsupervised behavior discovery.
method Proposed and investigated several metrics for generative models of behavior.
result The proposed metrics correspond with biologists' intuitions and allow for model evaluation and bias understanding.
Adaptive loss function improves performance by aligning training and evaluation metrics.
problem Loss-metric mismatch in machine learning training.
method Adaptive loss alignment through meta-learning of a dynamic loss function.
result Significant performance improvements across various tasks and data.
Paper proposes benchmarks and metrics for evaluating handwriting generation.
problem Evaluating the style of handwriting generation is challenging.
method Deep learning techniques for generating temporal sequences.
result Proposes evaluation metrics for handwriting generation.
EAST aligns neural network classifiers with user-defined evaluation metrics.
problem Mismatch between neural network training and evaluation metrics leads to suboptimal performance.
method EAST uses dynamic thresholding, soft-set confusion matrix, and annealing to align neural network predictions with target evaluation metrics.
result EAST improves alignment between training objectives and evaluation metrics, outperforming existing methods.
Interactive tool helps choose and understand classification metrics.
problem Common metrics for binary classification have limitations.
method Graphical application to visualize and explore evaluation metrics.
result Promotes careful attention to interpretation of metrics.
This study evaluates GAN metrics and identifies strengths and weaknesses.
problem Evaluating GANs is challenging due to the complexity of their evaluation metrics.
method Identify necessary conditions for meaningful GAN metrics, design experiments, and analyze popular metrics.
result Kernel Maximum Mean Discrepancy (MMD) and 1-Nearest-Neighbor (1-NN) seem most suitable for GAN evaluation.
New metrics improve reliability of image generation evaluation.
problem Lack of reliable metrics for evaluating fidelity and diversity in generative models.
method Proposed density and coverage metrics to diagnose fidelity and diversity separately.
result Density and coverage metrics provide more interpretable and reliable signals.
New approaches estimate recommendation metrics using sampling.
problem Understanding and resolving the use of sampling for recommendation evaluation.
method MLE and ME principles for empirical rank distribution recovery.
result Advantages of new approaches for top-k metrics estimation.
Study finds AUC is most consistent across different prevalence in binary classification.
problem Consistency of model evaluation metrics across varying prevalence in binary classification.
method Analysis of 156 data scenarios with 18 metrics, 5 models, and a random guess model.
result AUC has the smallest variance in evaluating individual models and ranking of models.
Paper proposes evaluation methods for climate change illustrations.
problem Lack of metrics to compare realism in conditional generative models.
method Adapted and assessed several existing metrics, including FID.
result FID with Inception-V3 embeddings correlates best with human realism.
This paper evaluates and improves metrics for identifying important features in machine learning models.
problem Evaluation metrics for explainable AI are limited by multicollinearity and model accuracy.
method Proposes Expected Accuracy Interval (EAI) to predict model accuracy with multicollinearity.
result EAI is a useful metric for identifying important features in models with multicollinearity.
The Inception Score fails to guide model comparisons.
problem The Inception Score is insufficient for model evaluation.
method Analysis of Inception Score's limitations and application issues.
result Inception Score does not provide useful guidance for model comparison.
Study flaws in generative model evaluation metrics, especially for diffusion models.
problem Flaws in existing metrics for evaluating generative models, particularly for diffusion models.
method Systematic study of generative models, human perception experiments, and analysis of feature extractors.
result State-of-the-art perceptual realism of diffusion models is not reflected in commonly reported metrics.
Toolbox provides datasets and metrics for state representation learning.
problem Lack of standard evaluation datasets, metrics and tasks for state representation learning.
method Provides a set of environments, data generators, robotic control tasks, metrics and tools.
result Facilitates iterative state representation learning and evaluation in reinforcement learning settings.
Develops RES metrics for stable rare-event forecasting evaluation.
problem Challenges in evaluating forecasts of rare events.
method Rare-event-stable (RES) metrics designed to maintain stable thresholds under extreme rarity.
result RES metrics maintain stable thresholds, consistent model rankings, and near-complete prevalence invariance.
New metric R2 improves topic model evaluation.
problem Lack of standard cross-contextual evaluation metrics for topic modeling.
method Introduces R2 as a coefficient of determination for topic models. result Improves topic model evaluation by providing a standard metric.
Study reveals gaps between simulated and real-world treatment effect evaluation metrics.
problem Evaluation of treatment effect estimation models differs between academic and practical settings.
method Comprehensive empirical study comparing semi-simulated benchmarks and real-world datasets.
result Counterfactual metrics do not reliably predict observable metrics, and rankings from simulated benchmarks do not generalize to real-world data.
The paper introduces metrics to evaluate NILM algorithms' performance on unseen buildings.
problem Assessing NILM algorithms' performance on new, unseen buildings.
method Developed several metrics to evaluate NILM algorithms' generalization ability.
result Demonstrated the utility of the proposed metrics through two case studies.
The paper evaluates and compares dimensionality reduction quality metrics without tuning.
problem Evaluating the quality of nonlinear dimensionality reduction visualizations is challenging.
method Comparison of dimensionality reduction quality metrics on datasets with known ground truth manifolds.
result A few methods consistently perform well, with one proposed as a benchmark.
Study improves machine learning models for GI tract disease detection using comprehensive evaluations and cross-dataset testing.
problem Incomplete or incorrect evaluation of machine learning models for GI tract diseases.
method Comprehensive evaluations of five machine learning models using Global Features and Deep Neural Networks, introducing performance hexagons and cross-dataset testing.
result Demonstrates the need for more sophisticated performance metrics and evaluation methods to build generalizable models.
Proposes a method to choose thresholds for LLM evaluation metrics.
problem Ensuring reliable large language models (LLMs) with correct threshold selection.
method Identify risks, stakeholders' risk tolerance, and use ground-truth data to determine thresholds.
result Demonstrates a concrete example with the Faithfulness metric and HaluBench dataset.
This research examines anomaly detection metrics under class imbalance.
problem Challenges in interpreting evaluation metrics under class imbalance.
method Analysis of four common anomaly detection metrics (AUROC, AUPR, F1-score, MCC) under varying imbalance ratios.
result Visualisations of metric landscapes provide an intuitive view of metric preferences and stability.
CAT framework improves AI medical screening fairness and reliability.
problem Imbalanced data, varying performance across cohorts, and patient-level inconsistencies in traditional metrics.
method CAT framework introduces patient-level assessment, entropy-based distribution weighting, and cohort-weighted sensitivity and specificity.
result Enhanced predictive reliability, fairness, and interpretability of AI-driven medical screening models.
Proposes a new metric to evaluate uncertainty in deep learning predictions.
problem Evaluating the reliability of deep learning predictions, especially uncertainty.
method Develops a novel metric for evaluating relative uncertainty in regression tasks with deep neural networks.
result Validates the new metric on a toy dataset and applies it to monocular depth estimation.
New metrics improve evaluation of EEG event detection algorithms.
problem Lack of standard evaluation metrics for EEG event detection.
method Proposed and demonstrated new metrics: ATWV and TAES.
result Deep learning algorithms need improvement for strict user acceptance.
Paper proposes a new metric to evaluate survival models, especially for censored data.
problem Challenges in evaluating survival prediction models due to censored data.
method Developed a novel approach to estimate Mean Absolute Error (MAE) for survival datasets with censored data.
result The proposed MAE metric using pseudo-observations accurately ranks model performance and closely matches true MAE.
In machine learning, the choice of a learning algorithm that is suitable for the application domain is critical. The performance metric used to compare different algorithms must also reflect the concerns of users in the application domain under consideration. In this work, we propose a novel probability-based performan…
Paper proposes metrics to evaluate AI explanations without ground truth.
problem Challenges in evaluating neural network explanations without ground truth.
method Designs four metrics to evaluate explanation results.
result New insights into neural network interpretation methods.
New metric reduces estimation error in survival model evaluation.
problem Dependent censoring complicates survival model evaluation.
method Dependent Brier score based on Archimedean copula and Copula-Graphic estimator.
result Reduces estimation error by 12-16% on average.
This research evaluates neural network robustness through loss visualization and a new metric.
problem Neural networks' robustness property is insufficiently investigated compared to adversarial attacks and defenses.
method Loss visualization and a new robustness metric to evaluate model stability.
result The proposed robustness metric provides a more reliable evaluation of model stability, uniformed across different models and settings.
Evaluation metrics for prediction models don't fully reflect intervention impact.
problem Standard metrics don't accurately reflect reduction in patient outcomes from model use.
method Synthesized and discussed various evaluation methods, analyzed with simulated and real data.
result Evaluations without interventional data are limited or require strong assumptions.
A framework evaluates synthetic tabular data quality objectively.
problem Lack of an objective interpretation of tabular data metrics.
method Proposes a single mathematical objective for synthetic tabular data distribution, structurally decomposes it, and unifies existing metrics.
result Synthesizers that represent tabular structure outperform other methods, especially on smaller datasets.
PolyGraph Discrepancy improves graph generative model evaluation.
problem Inability of existing metrics to provide an absolute performance measure and comparability across different graph descriptors.
method Approximates Jensen-Shannon distance using binary classifiers trained to distinguish between real and generated graphs.
result PGD provides a more robust and insightful evaluation compared to MMD metrics.
New metrics improve regression evaluation across different data distributions.
problem Difficulty in comparing regression evaluations across datasets with varying distributions.
method Modification of regression metrics by weighting with the inverse distribution of function values or samples using a Gaussian kernel density estimator.
result New metrics are less sensitive to changing distributions, especially when correcting by the marginal distribution in X. New metric evaluates generative models across domains, diagnosing fidelity, diversity, and generalization.
problem Evaluating generative models in diverse domains with limited metrics.
method Introduces a 3D evaluation metric (α-Precision, β-Recall, Authenticity) for domain-agnostic diagnostics. result Unified metric characterizes fidelity, diversity, and generalization, diagnosing model performance.
The paper defines metrics for evaluating disentangled representations in learning models.
problem Evaluating disentangled representations in learning models.
method Defining semantics and metrics for disentanglement learning.
result Proposed metrics correctly characterize representations learned by different methods.
Unified approach optimizes neural network training for various metrics.
problem Training and evaluation of neural network binary classifiers often use different metrics.
method Combines differentiable approximation and probabilistic soft sets.
result Effective in optimizing for metrics like F1-Score across various domains.
Paper explores how text generation quality and diversity metrics relate to distribution fitting.
problem Unclear relation between text generation quality and diversity metrics and distribution fitting.
method Theoretical approach to prove a linear combination of quality and diversity metrics can be a divergence metric.
result CR/NRR proposed as a better substitute for BLEU/Self-BLEU metrics.
Paper introduces metrics to evaluate missing data imputation without ground truth.
problem Handling missing data in time series without ground truth.
method Introduces Wasserstein distance (WD) and Jensen-Shannon divergence (JSD) as metrics to evaluate imputation quality.
result WD and JSD are effective metrics for assessing missing data imputation quality.
IRT metrics improve model evaluation by assessing latent characteristics.
problem Limitations of classic metrics like precision and F1.
method Introducing psychometric metrics like Item Response Theory (IRT).
result IRT complements classical metrics, offering new insights.
New methods improve evaluation of models under varying class imbalance.
problem Optimistic evaluation metrics lead to incorrect conclusions.
method Methods focusing on evaluation under non-constant class imbalance.
result Order of classifiers can change with class imbalance rate.
This work evaluates and benchmarks calibration metrics for data-driven regression models.
problem Conflicting results from different calibration metrics make it hard to compare and interpret model performance.
method Systematically extracted and benchmarked 14 regression calibration metrics across various data types and recalibration methods.
result Many metrics disagree on the same recalibration result, highlighting the need for careful metric selection.
Spatially-aware metrics improve uncertainty evaluation in segmentation.
problem Uncertainty evaluation metrics treat voxels independently, ignoring spatial context.
method Proposed three spatially aware metrics incorporating structural and boundary information.
result Improved alignment with clinically important factors and better discrimination between uncertainty patterns.