New method calibrates multi-class predictions efficiently without sacrificing accuracy.
problem Efficiently calibrating multi-class predictions without sacrificing accuracy.
method Formulated robust projected smooth calibration and new recalibration algorithms.
result Achieves strong guarantees for binary classification tasks with polynomial complexity.
In safety-critical applications a probabilistic model is usually required to be calibrated, i.e., to capture the uncertainty of its predictions accurately. In multi-class classification, calibration of the most confident predictions only is often not sufficient. We propose and study calibration measures for multi-class…
This paper improves multi-class calibration methods using mutual information maximization-based binning.
problem Calibration of deep neural network predictions, especially for small prior classes.
method I-Max concept for binning, shared class-wise calibration strategy.
result Improves multi-class ranking and calibration performance using a small calibration set.
New approach calibrates predictions for better decision-making.
problem Achieving reliable predictions for multi-class problems is hard.
method Introduces decision calibration, a new approach to calibrate predictions.
result Designs a recalibration algorithm that makes predictions reliable for decision-making.
Proposes a new method for multi-class classification with well-calibrated predictions.
problem Improving the accuracy and reliability of multi-class classification models.
method Trains data in a latent space induced by an (n−1)-dimensional simplex, then extends and fits a regression model. result Demonstrates a well-calibrated classifier with improved prediction and calibration properties.
We propose a new scalable multi-class Gaussian process classification approach building on a novel modified softmax likelihood function. The new likelihood has two benefits: it leads to well-calibrated uncertainty estimates and allows for an efficient latent variable augmentation. The augmented model has the advantage …
Develops geometric framework for uncertainty-aware multi-class classification.
problem Silent failure of AI models when uncertain, especially in multi-class settings.
method Geometric framework treating probability vectors as points on the (c−1)-dimensional probability simplex, using Fisher--Rao metric for calibration and uncertainty quantification. result Empirical validation shows 72.5% of errors captured while deferring 34.5% of ambiguous predictions, reducing automated decision error rates from 16.8% to 6.9%.
Many classification applications require accurate probability estimates in addition to good class separation but often classifiers are designed focusing only on the latter. Calibration is the process of improving probability estimates by post-processing but commonly used calibration algorithms work poorly on small data…
New method calibrates deep networks by preserving top-k predictions.
problem Calibrated confidence scores for multi-class deep networks to avoid rare mistakes.
method Intra order-preserving functions combined with neural network architecture.
result Outperforms state-of-the-art methods in evaluation metrics.
Two methods improve calibration of probabilistic classifiers, especially for multi-class problems.
problem Improving calibration of probabilistic classifiers, especially for multi-class problems.
method Two techniques: reduced calibration and class-wise calibration.
result Class-wise reduced calibration algorithms reduce prediction and per-class calibration errors.
In many classification problems it is desirable to output well-calibrated probabilities on the different classes. We propose a robust, non-parametric method of calibrating probabilities called SplineCalib that utilizes smoothing splines to determine a calibration function. We demonstrate how applying certain transforma…
The task of calibration is to retrospectively adjust the outputs from a machine learning model to provide better probability estimates on the target variable. While calibration has been investigated thoroughly in classification, it has not yet been well-established for regression tasks. This paper considers the problem…
Meta-Cal improves post-hoc calibration of neural networks.
problem Improving the accuracy of uncalibrated neural network predictions.
method Meta-Cal uses a base calibrator and a ranking model with constraints to provide high-probability bounds.
result Meta-Cal significantly outperforms existing methods in post-hoc multi-class classification calibration.
The paper extends calibration to sets of probabilistic classifiers, finding many ensembles are poorly calibrated.
problem Evaluating the validity of epistemic uncertainty in sets of probabilistic classifiers.
method Proposed a novel nonparametric calibration test for sets of probabilistic classifiers.
result Ensembles of deep neural networks are often not well calibrated.
KCal calibrates deep networks by embedding logits in a metric space.
problem Overconfident predictions from DNNs, especially in high-risk applications.
method KCal learns a metric space on the penultimate-layer latent embedding and generates predictions using kernel density estimates.
result KCal provides a provable full calibration guarantee and consistently outperforms baselines.
New truthful calibration errors improve model ranking in multiclass prediction.
problem Non-truthful calibration errors can mislead model comparisons.
method Introduced perfectly truthful calibration errors for multiclass predictions.
result Truthful calibration errors preserve decision-theoretic dominance and stabilize model rankings.
Paper proposes a unified framework for evaluating calibration of probabilistic models.
problem Evaluation of calibration for general probabilistic predictive models.
method Unified framework for calibration evaluation and tests for any probabilistic model.
result Generalization and reformulation of existing measures and tests.
Many applications of classification methods not only require high accuracy but also reliable estimation of predictive uncertainty. However, while many current classification frameworks, in particular deep neural networks, achieve high accuracy, they tend to incorrectly estimate uncertainty. In this paper, we propose a …
This paper improves neural network predictions with early stopping using conformal calibration.
problem Lack of precise statistical guarantees for neural networks trained with early stopping.
method Conformalized early stopping that combines early stopping with conformal calibration.
result Models provide both accuracy and precise inferences without additional data splits.
This research sets limits on how complex multi-class learning problems can be.
problem Understanding the complexity of multi-class classification problems.
method Established upper bounds on Natarajan dimensions for specific function classes.
result Upper bounds on Natarajan dimensions for multi-class decision trees, random forests, and neural networks.
Study on H-consistency bounds for machine learning surrogates.
problem Estimating target loss error relative to surrogate loss error in machine learning.
method Developed H-consistency bounds for various surrogates and loss functions. result Stronger guarantees than existing methods, offering distribution-dependent and -independent bounds.
Proposes squentropy loss for improved classification accuracy and model calibration.
problem Theoretical and empirical evidence for cross-entropy loss is lacking.
method Introduces squentropy loss as the sum of cross-entropy and average square loss over incorrect classes.
result Squentropy loss outperforms cross-entropy and rescaled square losses in classification accuracy and model calibration.
Better uncertainty estimates for neural networks using Gaussian process priors.
problem Poor uncertainty estimates in neural networks, especially on out-of-distribution data.
method Characterize the function-space prior of an ensemble of infinitely-wide neural networks as a Gaussian process and use it to build a probabilistic model.
result The approach improves calibration of neural networks, especially under distributional shift.
This paper reviews metrics for evaluating multi-class classification models.
problem Evaluating and comparing multi-class classification models.
method Review and analysis of metrics.
result Promising multi-class metrics are highlighted and their usages are demonstrated.
Develops algorithms for multi-class Neyman-Pearson classification with cost sensitivity.
problem Asymmetric misclassification costs in multi-class classification problems.
method Establishes connection with cost-sensitive learning, proposes two algorithms, extends NP oracle properties.
result Proposes algorithms with theoretical guarantees for multi-class Neyman-Pearson classification.
This work presents a new strategy for multi-class classification that requires no class-specific labels, but instead leverages pairwise similarity between examples, which is a weaker form of annotation. The proposed method, meta classification learning, optimizes a binary classifier for pairwise similarity prediction a…
In this paper, we study the formalism of unsupervised multi-class domain adaptation (multi-class UDA), which underlies a few recent algorithms whose learning objectives are only motivated empirically. Multi-Class Scoring Disagreement (MCSD) divergence is presented by aggregating the absolute margin violations in multi-…
We consider a problem of risk estimation for large-margin multi-class classifiers. We propose a novel risk bound for the multi-class classification problem. The bound involves the marginal distribution of the classifier and the Rademacher complexity of the hypothesis class. We prove that our bound is tight in the numbe…
Paper proposes GEG to enhance fairness in binary and multi-class classification.
problem Fairness in multi-class classification tasks is under-explored.
method Formulates multi-objective problem between effectiveness and fairness constraints, proposes GEG algorithm.
result GEG improves fairness up to 92% and decreases accuracy up to 14%.
Combines neural networks and STL for multi-class time-series classification.
problem Lack of interpretability in neural networks for time-series data.
method Proposes a method that uses neural networks to classify time-series data using STL specifications, introducing margin for multi-class classification and STL-based attributes for interpretability.
result Evaluations show improved interpretability and performance compared to state-of-the-art baselines.
A new method identifies class-specific covariates in multi-class prediction tasks.
problem Identifying covariates specifically associated with one or more outcome classes in multi-class prediction tasks.
method Introducing multi forests (MuFs) with multi-way and binary splits to measure class-associated discriminatory ability.
result The multi-class VIM specifically ranks class-associated covariates highly, unlike conventional VIMs.
This paper introduces new loss functions for balanced multi-class classification.
problem Balancing class imbalance in multi-class classification.
method Introduces two new surrogate loss families: GLA and GCA.
result GCA losses offer stronger theoretical guarantees in imbalanced settings.
A new algorithm reduces imbalanced data classification errors in multi-class settings.
problem Imbalanced data classification, especially with noise and overlapping classes.
method MC-CCR algorithm combining cleaning and resampling.
result High robustness to noise and superior performance compared to state-of-the-art methods.
Symmetrizes loss functions to improve neural network robustness against noisy labels.
problem Designing robust loss functions for noisy labels in neural networks.
method Symmetrization of multi-class loss functions, focusing on cross-entropy and unhinged loss.
result The multi-class unhinged loss is the unique convex symmetric loss under suitable assumptions.
The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels. Smoothing the labels in this way prevents the network from becoming over-confident and label smoothing…
Due to myriads of classes, designing accurate and efficient classifiers becomes very challenging for multi-class classification. Recent research has shown that class structure learning can greatly facilitate multi-class learning. In this paper, we propose a novel method to learn the class structure for multi-class clas…
We study consistency properties of machine learning methods based on minimizing convex surrogates. We extend the recent framework of Osokin et al. (2017) for the quantitative analysis of consistency properties to the case of inconsistent surrogates. Our key technical contribution consists in a new lower bound on the ca…
Conditional generators learn the data distribution for each class in a multi-class scenario and generate samples for a specific class given the right input from the latent space. In this work, a method known as "Versatile Auxiliary Classifier with Generative Adversarial Network" for multi-class scenarios is presented. …
We address the problem of multi-class classification in the case where the number of classes is very large. We propose a double sampling strategy on top of a multi-class to binary reduction strategy, which transforms the original multi-class problem into a binary classification problem over pairs of examples. The aim o…
We study consistency of learning algorithms for a multi-class performance metric that is a non-decomposable function of the confusion matrix of a classifier and cannot be expressed as a sum of losses on individual data points; examples of such performance metrics include the macro F-measure popular in information retri…
This paper improves risk bounds and calibration for smart predict-then-optimize method.
problem Improving risk bounds and calibration for smart predict-then-optimize method.
method Develops risk bounds and uniform calibration results for the SPO+ loss relative to the SPO loss.
result Empirical minimizer of the SPO+ loss achieves low excess true risk with high probability.
Improved risk assessment for UBI using telematics data and AdaBoost.
problem Class imbalance in predicting claims frequency for UBI.
method Cost-sensitive multi-class AdaBoost (SAMME.C2) algorithm.
result SAMME.C2 outperforms other models in handling class imbalances.
Paper tackles cybersecurity attack detection with an ensemble approach.
problem Challenges in multi-class classification for cyber security breaches.
method Designing a multi-node multi-class classification ensemble approach.
result Proposed approach outperforms full-data approach in multi-node data-censoring cases.
New method learns multi-class from single-class data with confidences.
problem Learning multi-class from single-class data without additional data.
method Empirical risk minimization framework for multi-class classification.
result Method achieves Bayes-consistency with noisy confidences.
New method improves deep learning models' uncertainty estimates.
problem Overconfidence in deep learning predictions.
method Develops a novel training algorithm using conformal inference.
result Produces more reliable uncertainty estimates without sacrificing accuracy.
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
problem Analyzing multi-class classification with the teacher-student perceptron.
method Deriving asymptotic expressions for Bayes-optimal and empirical risk minimization (ERM) generalization errors.
result Regularised cross-entropy minimization yields close-to-optimal accuracy for multi-class classification.
We consider the problem of multi-class classification and a stochastic opti- mization approach to it. We derive risk bounds for stochastic mirror descent algorithm and provide examples of set geometries that make the use of the algorithm efficient in terms of error in k.
Develops risk-averse fair multi-class classification methods.
problem Noisy, scarce, unreliable data in multi-class classification problems.
method Systemic risk models and risk-averse regularized decomposition method.
result Enforces fairness and improves performance with unreliable data.