Paper introduces new metrics for evaluating model accuracy.
problem Improving model accuracy and calibration.
method Developed two second-order accuracy metrics with integral and numerical representations.
result Validates model calibration settings and reveals distortions.
This paper reviews metrics to assess AI model calibration accuracy.
problem AI model probabilities do not always match their true accuracy.
method Comprehensive review of 82 probability calibration metrics.
result Identified 4 classifier families and 1 object detection family of metrics.
This paper evaluates and improves metrics for identifying important features in machine learning models.
problem Evaluation metrics for explainable AI are limited by multicollinearity and model accuracy.
method Proposes Expected Accuracy Interval (EAI) to predict model accuracy with multicollinearity.
result EAI is a useful metric for identifying important features in models with multicollinearity.
We propose a novel classifier accuracy metric: the Bayesian Area Under the Receiver Operating Characteristic Curve (CBAUC). The method estimates the area under the ROC curve and is related to the recently proposed Bayesian Error Estimator. The metric can assess the quality of a classifier using only the training datase…
LxCIM metric improves binary classification performance evaluation.
problem Evaluation metrics for binary classification are often not invariant to local class exchange.
method Proposes LxCIM, a rank-based metric invariant to local class exchange.
result LxCIM addresses limitations of existing metrics like AUROC.
We identify and optimize the fairness-accuracy tradeoff through TAF Curves and FAUC metrics.
problem Balancing fairness and accuracy in machine learning models for high-stakes decisions.
method Developed TAF Curves and FAUC metric to quantify the tradeoff, and introduced FairStacks framework to expand the Pareto frontier.
result FairStacks framework expands the empirical Pareto frontier and improves the FAUC for model ensembles.
ID-ExpO fine-tunes neural networks for more faithful explanations.
problem Improving the faithfulness of explanations for complex machine learning models.
method Differentiable insertion/deletion metric-aware regularizers for optimization.
result Fine-tuned predictors produce more faithful explanations.
Much of the focus in the design of deep neural networks has been on improving accuracy, leading to more powerful yet highly complex network architectures that are difficult to deploy in practical scenarios, particularly on edge devices such as mobile and other consumer devices given their high computational and memory …
New metrics needed for streaming ML due to delayed labels.
problem Streaming ML evaluation fails to identify unexpected performance.
method Recommend additional metrics for streaming ML performance.
result New metrics are needed for streaming ML due to delayed labels.
We introduce a new metric to evaluate corruption robustness of ML classifiers.
problem Evaluating corruption robustness of machine learning classifiers.
method We propose a test data augmentation method using minimal class separation distance to derive a robustness distance ε and a metric MSCR.
result The MSCR metric allows interpretable comparison of classifier robustness on different datasets.
Deconfounds neural network representation similarity metrics to improve consistency and accuracy.
problem Confounding by population structure in similarity metrics like RSA and CKA.
method Covariate adjustment regression to adjust for confounders.
result Improves detection of semantically similar neural networks and consistency in transfer learning.
Metric learning makes it plausible to learn distances for complex distributions of data from labeled data. However, to date, most metric learning methods are based on a single Mahalanobis metric, which cannot handle heterogeneous data well. Those that learn multiple metrics throughout the space have demonstrated superi…
New metric space for ReLU codes connects to network safety and robustness.
problem Lack of metrics capturing network safety and robustness beyond accuracy.
method Introduces a metric space of ReLU activation codes with a truncated Hamming distance.
result Establishes an isometry between ReLU codes and polyhedral bodies related to safety and robustness.
New metrics CWSA and CWSA+ improve model evaluation under confidence thresholds.
problem Lack of metrics capturing model reliability under confidence thresholds.
method Introducing CWSA and CWSA+ metrics that reward confident accuracy and penalize overconfident mistakes.
result CWSA and CWSA+ outperform classical metrics in trust-sensitive tests.
Synthesizes sensor likelihoods to enforce accuracy constraints in uncertain systems.
problem Designing sensing architectures for systems with uncertain or unavailable sensor models and accuracy requirements.
method Inverts the design flow, synthesizing measurement likelihoods that minimize Kullback-Leibler divergence from the prior while enforcing an accuracy bound.
result The method synthesizes a maximum-entropy posterior and induced likelihood, accommodating various discrepancy metrics.
Predicting discomfort glare in open-plan offices is a challenging problem. Although glare research has existed for more than 50 years, all current glare metrics have accuracy limitations, especially in open-plan offices with low lighting levels. Thus, it is crucial to develop a new method to predict glare more accurate…
New metric measures dynamical richness without relying on accuracy.
problem Lack of a reliable metric for measuring dynamical richness.
method Developed a computationally efficient, performance-independent metric based on low-rank bias.
result Metric recovers neural collapse as a special case and captures known transitions without accuracy.
A number of machine learning algorithms are using a metric, or a distance, in order to compare individuals. The Euclidean distance is usually employed, but it may be more efficient to learn a parametric distance such as Mahalanobis metric. Learning such a metric is a hot topic since more than ten years now, and a numbe…
Score based learning (SBL) is a promising approach for learning Bayesian networks in the discrete domain. However, when employing SBL in the continuous domain, one is either forced to move the problem to the discrete domain or use metrics such as BIC/AIC, and these approaches are often lacking. Discretization can have …
Paper introduces negative margin loss for better few-shot classification accuracy.
problem Improving few-shot classification accuracy with metric learning.
method Introduces negative margin loss and analyzes its impact on feature discriminability.
result Negative margin loss outperforms regular softmax loss on few-shot classification benchmarks.
In critical decision-making scenarios, optimizing accuracy can lead to a biased classifier, hence past work recommends enforcing group-based fairness metrics in addition to maximizing accuracy. However, doing so exposes the classifier to another kind of bias called infra-marginality. This refers to individual-level bia…
With the increasing availability of AI-based decision support, there is an increasing need for their certification by both AI manufacturers and notified bodies, as well as the pragmatic (real-world) validation of these systems. Therefore, there is the need for meaningful and informative ways to assess the performance o…
New method makes quality metrics scale-invariant for high-dimensional data.
problem Scale sensitivity in quality metrics affects the accuracy of data projections.
method Analytical and empirical investigation of stress and KL divergence; introduction of a scale-invariant technique.
result The proposed technique accurately captures expected behavior and makes metrics scale-invariant.
In this paper, we propose a new measure to gauge the complexity of image classification problems. Given an annotated image dataset, our method computes a complexity measure called the cumulative spectral gradient (CSG) which strongly correlates with the test accuracy of convolutional neural networks (CNN). The CSG meas…
This paper investigates the theoretical foundations of metric learning, focused on three key questions that are not fully addressed in prior work: 1) we consider learning general low-dimensional (low-rank) metrics as well as sparse metrics; 2) we develop upper and lower (minimax)bounds on the generalization error; 3) w…
New metric detects adversarial samples with high accuracy.
problem Vulnerability of deep neural networks to adversarial samples.
method Analyzed adversarial samples through their contributions to principal components of images.
result Proposed new metric (k,p) point for measuring robustness to adversarial samples.
COMPASS improves uncertainty quantification for medical segmentation metrics.
problem Uncertainty quantification for medical segmentation metrics is crucial for clinical decision-making.
method COMPASS leverages deep neural network inductive biases to generate efficient, metric-based conformal prediction intervals.
result COMPASS produces significantly tighter intervals than traditional conformal prediction methods on medical image segmentation tasks.
The study improves volatility model pricing accuracy with new statistical expansions.
problem Improving option pricing accuracy in volatility models.
method Developed Edgeworth expansions for various volatility models.
result Enhanced statistical expansions for volatility models.
A commonly used evaluation metric for text-to-image synthesis is the Inception score (IS) \cite{inceptionscore}, which has been shown to be a quality metric that correlates well with human judgment. However, IS does not reveal properties of the generated images indicating the ability of a text-to-image synthesis method…
The paper discusses thresholds and bounds for accuracy in binary classification systems.
problem The accuracy of binary classification systems and its dependence on prevalence.
method Analyzing the precision-prevalence curve and negative predictive value-prevalence curve to find thresholds and bounds.
result Thresholds (φe and φn) bound various accuracy metrics (Fβ, F1, FM, MCC) and the ratio of maximum accuracy to prevalence. SAT improves adversarial training by smoothing the loss landscape through curriculum learning.
problem Adversarial training sacrifices clean accuracy for robustness and suffers from large generalization error.
method SAT uses curriculum learning to smooth the adversarial loss landscape, improving both clean and robust accuracy.
result SAT models improve clean and robust accuracy significantly compared to adversarial training and other baselines.
Unified NICEk metrics improve solar forecasting accuracy.
problem Lack of suitable error metrics for multidimensional solar irradiance forecasting.
method Introducing NICEk framework with Lk norms for evaluating forecasting models.
result NICESigma consistently outperforms traditional metrics in discriminative power and statistical significance.
In machine learning, the choice of a learning algorithm that is suitable for the application domain is critical. The performance metric used to compare different algorithms must also reflect the concerns of users in the application domain under consideration. In this work, we propose a novel probability-based performan…
IVON improves LoRA finetuning with minimal overhead and significant accuracy gains.
problem Improving LoRA finetuning with Bayesian methods.
method IVON, a variational algorithm, with posterior pruning.
result Significant accuracy improvements over AdamW and other Bayesian methods.
CPML efficiently learns new metrics for categorical data.
problem Metric learning for categorical data.
method CPML (categorical projected metric learning) using Schatten p-norms.
result CPML provides efficient metric learning with improved accuracy.
Improves material discovery through better model evaluation metrics.
problem Standard error metrics mislead in material discovery.
method Introduces Pareto shell-scope error for model evaluation.
result Novel diagnostic tools and insights for acquisition function design.
We generalise a theorem of Engman and Abreu--Freitas on the first invariant eigenvalue of non-negatively curved S1-invariant metrics on CP1 to general toric Kähler metrics with non-negative scalar curvature. In particular, a simple upper bound of the first non-zero invariant eigenvalue for such metri…
Most of the work on interpretable machine learning has focused on designing either inherently interpretable models, which typically trade-off accuracy for interpretability, or post-hoc explanation systems, which lack guarantees about their explanation quality. We propose an alternative to these approaches by directly r…
Deep metric learning has been demonstrated to be highly effective in learning semantic representation and encoding information that can be used to measure data similarity, by relying on the embedding learned from metric learning. At the same time, variational autoencoder (VAE) has widely been used to approximate infere…
New scoring rules compare probabilistic top lists in classification.
problem Evaluation of probabilistic top lists in classification.
method Elicitability through symmetric proper scoring rules.
result Brier score provides a well-suited metric for comparison.
This paper evaluates forecast quality in electricity markets beyond traditional accuracy measures.
problem Traditional accuracy measures fail to reflect the economic value of electricity price forecasts.
method Investigates four quality dimensions: accuracy, dispersion, association, and extremum identification.
result Dispersion- and association-based measures better capture forecast economic value.
EAST aligns neural network classifiers with user-defined evaluation metrics.
problem Mismatch between neural network training and evaluation metrics leads to suboptimal performance.
method EAST uses dynamic thresholding, soft-set confusion matrix, and annealing to align neural network predictions with target evaluation metrics.
result EAST improves alignment between training objectives and evaluation metrics, outperforming existing methods.
GNMC reduces XCSF population size while preserving function approximation and policy accuracy.
problem Population bloat in XCSF compaction.
method Introduced GNMC, a novel compaction algorithm.
result GNMC reduces population size significantly without compromising function approximation or policy accuracy.
This paper visualizes uncertainty in classifier performance metrics.
problem Overemphasis on model performance metrics risks overlooking uncertainty.
method Developed visualizations of confusion matrix metric distributions.
result Uncertainty in performance metrics can overshadow model differences.
New insights on Bayesian models show stochasticity doesn't always improve accuracy.
problem The impact of stochasticity in Bayesian models and the role of temperature parameters.
method Detailed investigation, empirical testing, and PAC-Bayesian analysis.
result Stochasticity does not generally improve test accuracy and the coldest temperature is often optimal.
Recently, adversarial deception becomes one of the most considerable threats to deep neural networks. However, compared to extensive research in new designs of various adversarial attacks and defenses, the neural networks' intrinsic robustness property is still lack of thorough investigation. This work aims to qualitat…
Road Network Metric Learning improves ETA prediction accuracy by addressing data sparsity.
problem Data sparsity in road network embedding vectors affects ETA prediction accuracy.
method Proposes Road Network Metric Learning (RNML-ETA) framework with an auxiliary metric learning task and triangle loss.
result RNML-ETA outperforms state-of-the-art models and improves prediction accuracy for cold links.
Photo-identification (photo-id) of dolphin individuals is a commonly used technique in ecological sciences to monitor state and health of individuals, as well as to study the social structure and distribution of a population. Traditional photo-id involves a laborious manual process of matching each dolphin fin photogra…