Proposes a method to select fair performance metrics through metric elicitation.
problem Choosing fair performance metrics in multiclass classification with multiple sensitive groups.
method Metric elicitation strategy that requires only relative preference feedback and is robust to noise.
result Elicits group-fair performance metrics for multiclass classification problems.
Given a binary prediction problem, which performance metric should the classifier optimize? We address this question by formalizing the problem of Metric Elicitation. The goal of metric elicitation is to discover the performance metric of a practitioner, which reflects her innate rewards (costs) for correct (incorrect)…
Unified framework for comparing classification metrics across different imbalance rates.
problem Differences in scale and sensitivity to class imbalance rates in classification metrics.
method Introduces outperformance standardization (OPS) function to map metrics to a common scale.
result Unified o-value metric provides clear comparison across different imbalance rates.
Performance metrics (error measures) are vital components of the evaluation frameworks in various fields. The intention of this study was to overview of a variety of performance metrics and approaches to their classification. The main goal of the study was to develop a typology that will help to improve our knowledge a…
New metrics improve performance in imbalanced classification problems.
problem Established metrics favor classifiers ignoring minority classes.
method Introduce robust modifications of F-score and MCC.
result TPR is bounded away from 0 in imbalanced settings.
This thesis formalizes metric selection for machine learning applications.
problem Selecting the right performance metric for machine learning models.
method Metric Elicitation framework using pairwise preference feedback.
result Novel strategies for eliciting various classification metrics robust to noise.
New method quantifies classifier uncertainty, revealing large variability in performance metrics.
problem Uncertainty in classifier performance metrics due to small data sets.
method Probability model of the confusion matrix to quantify uncertainty.
result Large uncertainties in classification performance metrics can lead to misleading conclusions.
This paper visualizes uncertainty in classifier performance metrics.
problem Overemphasis on model performance metrics risks overlooking uncertainty.
method Developed visualizations of confusion matrix metric distributions.
result Uncertainty in performance metrics can overshadow model differences.
We study consistency of learning algorithms for a multi-class performance metric that is a non-decomposable function of the confusion matrix of a classifier and cannot be expressed as a sum of losses on individual data points; examples of such performance metrics include the macro F-measure popular in information retri…
AlphaSharpe uses LLMs to improve financial metrics robustness and predictive power.
problem Traditional financial metrics struggle with robustness and generalization in volatile markets.
method Iterative optimization of financial metrics using LLMs, including crossover, mutation, and evaluation.
result AlphaSharpe discovers enhanced risk-return metrics with 3x predictive power and 2x portfolio performance.
Study analyzes neural network models to understand generalization performance.
problem Understanding good generalization in neural networks.
method Analyzed a corpus of models from a public contest, breaking ALPHAHAT into scale and shape metrics.
result Identified a Simpson's paradox in metric performance across different model depths and regularization hyperparameters.
Examines performance metrics for ML models in engineering.
problem Identifying proper assessment metrics for ML models.
method Analyzes commonly-used metrics for regression and classification.
result Identifies key performance metrics for ML models.
New method reduces variance in subpopulation model performance estimates.
problem High variance in subpopulation performance metrics for small groups.
method Using an evaluation model to form model-based metric (MBM) estimates.
result MBMs produce more accurate and lower variance estimates for small subpopulations.
New algorithm optimizes complex metrics in online learning.
problem Optimizing non-decomposable metrics in sequential learning.
method General online algorithm for various metrics.
result Achieves O(nlnn) regret for concave and smooth metrics. The paper quantifies uncertainty in aggregated machine learning metrics.
problem Uncertainty in summarizing model performance across multiple tasks.
method Statistical methodologies including bootstrapping and Bayesian modeling.
result Insights into model performance dominance for specific tasks.
This paper introduces a new metric for deep learning networks based on their classification performance.
problem The mystery and black-box nature of deep learning networks.
method Proposes a new distance measure based on the probabilistic performance of deep learning networks.
result The proposed metric space is compact and coincides with the quotient topological space.
We study the model selection problem in conditional average treatment effect (CATE) prediction. Unlike previous works on this topic, we focus on preserving the rank order of the performance of candidate CATE predictors to enable accurate and stable model selection. To this end, we analyze the model performance ranking …
Complex classification performance metrics such as the Fβ-measure and Jaccard index are often used, in order to handle class-imbalanced cases such as information retrieval and image segmentation. These performance metrics are not decomposable, that is, they cannot be expressed in a per-example manner, which hinder…
New metrics improve scRNA-seq perturbation modeling by reducing mode collapse.
problem Outperformed by simple mean prediction in scRNA-seq perturbation modeling.
method Introduce DEG-aware metrics (WMSE, Rw2(Δ)) and negative/positive baselines. result WMSE loss function reduces mode collapse and improves model performance.
Paper extends ranking metrics theory for financial positions.
problem Developing a new class of functionals for evaluating financial positions.
method Axiomatic framework based on monotonicity and cash-quasiconcavity.
result Linking ranking metrics to families of acceptance sets and risk measures.
Paper extends ranking metrics theory for financial positions.
problem Developing a new class of performance evaluation methods.
method Axiomatic framework based on monotonicity and cash-quasiconcavity.
result Linking ranking metrics to families of acceptance sets and risk measures.
This paper develops a new method for eliciting more flexible metrics, improving fairness and applicability.
problem Limited flexibility in existing metric elicitation strategies for reflecting user preferences.
method Develops a strategy for eliciting quadratic metrics based on predictive rates, requiring only relative preference feedback.
result Achieves near-optimal query complexity and broadens the use cases for metric elicitation.
This paper reformulates Fβ for better model performance and interpretation.
problem Optimizing model performance and interpretation using Fβ metric. method Reformulate Fβ metric to facilitate statistical distributions and dynamic penalty weights. result Better and interpretable results with a 14% boost in F1 score for IMDB data. New metrics reveal oversmoothing in GNNs more accurately than traditional methods.
problem Oversmoothing in graph neural networks reduces model performance.
method Rank-based metrics to measure oversmoothing in GNNs.
result Rank-based metrics consistently capture oversmoothing, while energy-based metrics often fail.
The distance metric plays an important role in nearest neighbor (NN) classification. Usually the Euclidean distance metric is assumed or a Mahalanobis distance metric is optimized to improve the NN performance. In this paper, we study the problem of embedding arbitrary metric spaces into a Euclidean space with the goal…
To assess the performance of load disaggregation algorithms it is common practise to train a candidate algorithm on data from one or multiple households and subsequently apply cross-validation by evaluating the classification and energy estimation performance on unseen portions of the dataset derived from the same hous…
In machine learning, the choice of a learning algorithm that is suitable for the application domain is critical. The performance metric used to compare different algorithms must also reflect the concerns of users in the application domain under consideration. In this work, we propose a novel probability-based performan…
Rank-based metrics are some of the most widely used criteria for performance evaluation of computer vision models. Despite years of effort, direct optimization for these metrics remains a challenge due to their non-differentiable and non-decomposable nature. We present an efficient, theoretically sound, and general met…
This work improves metric learning models by incorporating class hierarchies.
problem Class hierarchies are often ignored in classification-based metric learning models.
method Trained softmax classifier and metric learning models with predefined class hierarchies.
result ProxyDR model performs better in hierarchical inference and hierarchy-informed performance.
Enhances trading metrics with financially grounded loss functions.
problem Challenges in financial deep learning, especially interpretability.
method Introduces loss functions derived from finance metrics and turnover regularization.
result Proposed loss functions outperform traditional methods in trading metrics.
We provide a general theoretical analysis of expected out-of-sample utility, also referred to as decision-theoretic classification, for non-decomposable binary classification metrics such as F-measure and Jaccard coefficient. Our key result is that the expected out-of-sample utility for many performance metrics is prov…
New algorithms improve robustness in metric learning.
problem Improving robustness in metric learning for better performance.
method Robust Geometric Metric Learning (RGML) approach, shrinking covariance matrices towards their barycenter.
result Strong performance and robustness to mislabeled data demonstrated.
Locally adaptive nearest neighbors improve automated systems' performance and are easier to interpret.
problem Improving automated systems' performance and interpretability.
method Developed a method for k nearest neighbors algorithms to learn locally adaptive metrics.
result Locally adaptive metrics improve performance and are interpretable.
Consistent algorithms for multiclass learning with complex metrics and constraints.
problem Learning with complex performance metrics and constraints.
method General framework for designing consistent algorithms by viewing the problem as an optimization over feasible confusion matrices.
result Rates of convergence to the optimal (feasible) classifier, showing asymptotic consistency.
The study compares M6 competitors' performance to industry benchmarks and discusses incentives for investment managers.
problem Investors seek to understand the performance and skill of M6 competitors beyond the competition's metrics.
method Comparative analysis using financial metrics, factor models, and new strategies.
result Most competitors do not generate significant out-performance compared to industry benchmarks, but some show skill in recent performance.
metric-learn is an open source Python package implementing supervised and weakly-supervised distance metric learning algorithms. As part of scikit-learn-contrib, it provides a unified interface compatible with scikit-learn which allows to easily perform cross-validation, model selection, and pipelining with other machi…
New metric evaluates generative models across domains, diagnosing fidelity, diversity, and generalization.
problem Evaluating generative models in diverse domains with limited metrics.
method Introduces a 3D evaluation metric (α-Precision, β-Recall, Authenticity) for domain-agnostic diagnostics. result Unified metric characterizes fidelity, diversity, and generalization, diagnosing model performance.
We address the problem of training models with black-box and hard-to-optimize metrics by expressing the metric as a monotonic function of a small number of easy-to-optimize surrogates. We pose the training problem as an optimization over a relaxed surrogate space, which we solve by estimating local gradients for the me…
Proposes a Siamese NN for algorithm selection focusing on alike performing instances.
problem Lack of effective meta-features for algorithm selection via meta-learning.
method Siamese Neural Network architecture with 'Algorithm-Performance Personas' concept.
result Proposed metric outperforms standard performance metrics in training sample selection.
Metric learning enhances combinatorial coverage metrics' ability to predict classification errors.
problem Dataset dependence of combinatorial coverage metrics in anticipating classification errors.
method Metric learning to improve latent space separation of data classes.
result Metric learning increases SDCCMs' ability to distinguish between correctly and incorrectly classified data.
Study evaluates relevance metrics for similarity-based model explanations.
problem Providing understandable explanations for complex model predictions.
method Evaluated three relevance metrics using three tests.
result Cosine similarity of gradients performs best for explanations.
Study improves machine learning models for GI tract disease detection using comprehensive evaluations and cross-dataset testing.
problem Incomplete or incorrect evaluation of machine learning models for GI tract diseases.
method Comprehensive evaluations of five machine learning models using Global Features and Deep Neural Networks, introducing performance hexagons and cross-dataset testing.
result Demonstrates the need for more sophisticated performance metrics and evaluation methods to build generalizable models.
Operations is a key challenge in the domain of machine learning pipeline deployments involving monitoring and management of real-time prediction quality. Typically, metrics like accuracy, RMSE etc., are used to track the performance of models in deployment. However, these metrics cannot be calculated in production due …
This paper reviews metrics for evaluating multi-class classification models.
problem Evaluating and comparing multi-class classification models.
method Review and analysis of metrics.
result Promising multi-class metrics are highlighted and their usages are demonstrated.
New metric MADD assesses fairness of predictive student models.
problem Predictive student models can be biased and unfair, leading to discrimination.
method Proposes MADD metric to analyze model's discriminatory behaviors.
result Fair predictive performance does not guarantee fair behaviors or outcomes.
Proposes a new metric learning method for image recognition.
problem Improving image recognition performance using learned distance representations.
method Introduces a Generalized Hybrid Metric Loss (GHM-Loss) to learn hybrid proximity features combining geometric and probabilistic spaces.
result Demonstrates superior performance compared to existing methods on public datasets.
Deep Neural Networks(DNN) have excessively advanced the field of computer vision by achieving state of the art performance in various vision tasks. These results are not limited to the field of vision but can also be seen in speech recognition and machine translation tasks. Recently, DNNs are found to poorly fail when …
New metrics using Laplace approximation improve Gaussian process model selection.
problem Finding a balance between model accuracy, interpretability, and simplicity.
method Introducing multiple metrics based on the Laplace approximation to evaluate Gaussian process models.
result Our metrics provide comparable performance to dynamic nested sampling but are significantly faster.