Improves binary classification from positive data with skewed confidence.
problem Skewed confidence in positive data affects the performance of Pconf classifiers.
method Parameterized model of skewed confidence and hyperparameter selection.
result Proposed method effectively cancels out the negative impact of skewed confidence.
The paper proposes using entropy to assess model confidence.
problem Discarded information in probability distributions hinders model confidence assessment.
method Entropy methods applied to assess model confidence.
result Theoretical explanation of confidence degradation in Naive Bayes model.
The paper shows over-confidence in models isn't just due to over-parametrization.
problem Over-confidence in machine learning models, especially in binary classification.
method Theoretical analysis of logistic regression and other binary classification problems.
result Logistic regression is inherently over-confident in certain settings, but over-confidence is not always the case.
Can we learn a binary classifier from only positive data, without any negative data or unlabeled data? We show that if one can equip positive data with confidence (positive-confidence), one can successfully learn a binary classifier, which we name positive-confidence (Pconf) classification. Our work is related to one-c…
Study three types of uncertainty quantification for binary classification without distributional assumptions.
problem Uncertainty quantification for binary classification in a distribution-free setting.
method Established theorems connecting calibration, confidence intervals, and prediction sets for score-based classifiers.
result Distribution-free calibration is only possible using scoring functions that partition feature space into countably many sets.
DBLE improves confidence calibration of DNNs by learning distances in representation space.
problem Poor confidence calibration of deep neural networks (DNNs).
method DBLE trains a confidence model jointly with the classification model, using distances in the representation space.
result DBLE outperforms alternative single-model confidence calibration approaches and ensemble methods.
New method corrects skewed confidence for PbN classification.
problem Weakly supervised binary classification with biased negative data.
method Corrects skewed confidence in negative data to improve classifier.
result Reduces distortion in posterior probability for PbN classification.
Paper develops a BERT-based classifier to reduce pathology report annotation workload.
problem Manual annotation of pathology reports is labor-intensive and time-consuming.
method Developed an automatic text classifier using BERT and introduced a human-centric metric to identify low-confidence cases.
result The model reduces manual annotation workload by 80% to 98%.
Develops a method to find costly high-confidence errors in black box models.
problem Finding rare high-confidence errors missed by random sampling.
method Adversarial perturbation-guided search technique to find errors at rates greater than expected given model confidence.
result Our Adversarial Distance search discovers high-confidence errors at a rate greater than expected given model confidence.
Mahalanobis distance detects anomalies well, but not for classification.
problem Detecting anomalies in neural classifier outputs.
method Analyzes Mahalanobis distance-based anomaly detection method, revealing its reliance on information not useful for classification.
result Combining Mahalanobis and ODIN methods improves anomaly detection performance and robustness.
A method to approximate instance-dependent label noise using instance-confidence embedding.
problem Real-world label noise that depends on individual instances.
method Variational approximation with instance embedding to capture instance-specific label corruption.
result ICE method effectively approximates instance-dependent noise and detects ambiguous instances.
A new method uses SVM classification to efficiently compute confidence sets.
problem Computing confidence sets for moment inequalities is computationally intensive.
method Converts confidence set construction into a classification problem using SVM.
result Asymptotically reproduces the test in the confidence set using SVM classification.
The paper develops methods for constructing confidence regions for regression functions in binary classification.
problem Building distribution-free confidence regions for regression functions in binary classification.
method Resampling test and empirical risk minimization approach for model classes with finite pseudo-dimensions and inverse Lipschitz parameterizations.
result Strong uniform consistency and exponential probably approximately correct bounds on the L2 sizes of the regions. Paper proposes sparse classification method for high-dimensional data.
problem Sparse classification in high-dimensional data with positive-confidence samples.
method Developed a novel sparse-penalization framework using L1, SCAD, and MCP penalties for convex and non-convex shrinkage.
result Proved near minimax-optimal sparse recovery rates under Restricted Strong Convexity condition.
E-Stitchup augments pre-trained embeddings for better model performance and confidence calibration.
problem Improving classification accuracy and confidence calibration of models using pre-trained embeddings.
method Data augmentation methods inspired by Mixup combined with label softening.
result Significantly increased classification accuracy and reduced training time.
New method learns multi-class from single-class data with confidences.
problem Learning multi-class from single-class data without additional data.
method Empirical risk minimization framework for multi-class classification.
result Method achieves Bayes-consistency with noisy confidences.
Modified AUC improves CNN training by considering model confidence.
problem Improving binary classifier performance metrics.
method Proposes a modified AUC metric that incorporates model confidence into BCE loss for CNN training.
result Demonstrates improved performance on three datasets: MNIST, prostate MRI, and brain MRI.
We study the tradeoff between computational effort and classification accuracy in a cascade of deep neural networks. During inference, the user sets the acceptable accuracy degradation which then automatically determines confidence thresholds for the intermediate classifiers. As soon as the confidence threshold is met,…
This paper improves confidence measurement in deep metric learning models.
problem Measuring confidence in deep metric learning models is challenging.
method Approximates class distributions using Gaussian kernel smoothing and calibrates the confidence metric.
result Improves generalization and robustness of deep metric learning models.
Discriminatively trained neural classifiers can be trusted, only when the input data comes from the training distribution (in-distribution). Therefore, detecting out-of-distribution (OOD) samples is very important to avoid classification errors. In the context of OOD detection for image classification, one of the recen…
PaRCE estimates model confidence for CNNs across various uncertainties.
problem Limited holistic approach to estimating perception model confidence in CNNs.
method Probabilistic and reconstruction-based competency estimation.
result PaRCE best distinguishes between various types of samples and regions.
Confidence intervals improve evaluation of binary prediction rules in data mining.
problem Uncertainty in performance measures estimation from finite datasets.
method Asymptotic normal approximations for confidence intervals, with a blurring correction.
result Improved finite sample coverage probabilities and general performance measures inference.
The paper tackles confidence calibration for exploratory machine learning problems.
problem Difficulty in curating datasets and confusion about category validity.
method Introduces four new algorithms for category-specific confidence estimation, including kernel density ratios.
result Kernel density ratios provide a novel approach to confidence calibration, especially for exploratory problems.
Paper proposes Pcomp classification for binary classification with pairwise confidence comparisons.
problem Lack of pointwise labels due to privacy, confidentiality, or security reasons.
method Developed Pcomp classification, derived an unbiased risk estimator (URE), and improved it using correction functions and consistency regularization.
result Demonstrated the effectiveness of Pcomp classification methods.
Focal loss improves classification but not class-posterior probability estimation.
problem Improving class-posterior probability estimation from focal loss.
method Proved classification-calibration and derived a transformation to recover true class-posterior probabilities.
result A transformation of the confidence score from focal loss minimization allows recovery of true class-posterior probabilities.
A new method for semi-supervised learning with missing data using GMM and margin confidence.
problem Handling missing data in semi-supervised learning with classification uncertainty.
method Explicitly models missingness mechanism, uses margin confidence and Aranda Ordaz function, develops ECM algorithm.
result Effective reduction of bias and robustness in semi-supervised learning with substantial missing labels.
Study on learning to defer to multiple experts with consistent surrogates and confidence calibration.
problem Addressing the open problems of consistent surrogates, confidence calibration, and ensembling of experts.
method Derive two consistent surrogates (softmax and OvA) and propose a conformal inference technique for choosing experts.
result The OvA-based loss does not cause mis-calibration propagation, while the softmax-based loss does.
Novel framework for unbiased confidence estimates in object detection.
problem Unbiased confidence estimates for safety-critical object detection.
method Combines regression output with additional information for calibration.
result Calibrated confidence estimates for image location and scale.
MARGINATTACK improves zero-confidence adversarial attacks' accuracy and efficiency.
problem Improving zero-confidence adversarial attacks' accuracy and efficiency.
method Proposes MARGINATTACK, a zero-confidence attack framework that computes margin with improved accuracy and efficiency.
result MARGINATTACK computes a smaller margin than state-of-the-art zero-confidence attacks and matches state-of-the-art fix-perturbation attacks.
FROB model improves robustness and reliable confidence for few-shot OoD detection.
problem Challenges in few-shot classification and OoD detection due to limited samples and adversarial attacks.
method FROB model combines support boundary generation and few-shot Outlier Exposure (OE) for improved robustness and reliable confidence.
result FROB achieves generalization to unseen OoD and maintains robustness independent of few-shot number.
A modification of the confidence screening mechanism based on adaptive weighing of every training instance at each cascade level of the Deep Forest is proposed. The idea underlying the modification is very simple and stems from the confidence screening mechanism idea proposed by Pang et al. to simplify the Deep Forest …
Classifiers used in the wild, in particular for safety-critical systems, should not only have good generalization properties but also should know when they don't know, in particular make low confidence predictions far away from the training data. We show that ReLU type neural networks which yield a piecewise linear cla…
Paper proposes a method to predict MOBA game winners with calibrated confidence.
problem Predicting MOBA game winners with noisy data and uncertain noise.
method A novel confidence-calibration method considering data uncertainty.
result Achieves outstanding expected calibration error (ECE) of 0.57%.
We describe Information Forests, an approach to classification that generalizes Random Forests by replacing the splitting criterion of non-leaf nodes from a discriminative one -- based on the entropy of the label distribution -- to a generative one -- based on maximizing the information divergence between the class-con…
Paper improves competence estimation of machine learning models.
problem Estimating machine learning performance in real-world scenarios.
method ALICE Score: a pointwise competence estimator considering distributional, data, and model uncertainty.
result Significant improvements in competence prediction over state-of-the-art methods.
Bayes-TrEx finds in-distribution examples for model inspection.
problem Challenges in interpreting neural networks, especially high-confidence failures and ambiguous classifications.
method Bayesian sampling approach to find in-distribution examples with specified prediction confidence.
result Bayes-TrEx enables more flexible holistic model analysis than just inspecting the test set.
In classification applications, we often want probabilistic predictions to reflect confidence or uncertainty. Dropout, a commonly used training technique, has recently been linked to Bayesian inference, yielding an efficient way to quantify uncertainty in neural network models. However, as previously demonstrated, conf…
RADE improves anomaly detection efficiency on small devices.
problem Anomaly detection in large datasets consumes excessive resources.
method RADE uses coarse-grained and expert fine-grained DTEM models.
result RADE reduces memory footprint, training time, and latency.
This paper rethinks confidence calibration under covariate shifts.
problem Calibration methods struggle with covariate shifts and unstable importance weighting.
method Derives Expectation consistency condition and proposes Expectation consistency loss (ECL).
result ECL loss is compatible with various types of calibration and has the same sample complexity as ECE.
Paper introduces a novel method for estimating model confidence in deep neural classifiers.
problem Reliable confidence estimation for deep neural classifiers in safety-critical applications.
method Proposes a novel target criterion (true class probability) and learns it from data with an auxiliary model.
result The proposed method outperforms strong baselines in various tasks and network architectures.
Cubic predicts stock market indices by fusing stock latent embeddings and converting to binary classification.
problem Challenges in predicting stock market indices due to isolated time series treatment and simple regression.
method Fusion of stock latent embeddings, binary encoding classification, and confidence-guided prediction.
result Cubic outperforms state-of-the-art baselines in stock index prediction tasks.
Paper proposes a method to improve deep neural networks' confidence estimates.
problem Overconfident predictions limit practical use of deep neural networks in safety-critical applications.
method Proposes a novel loss function, Correctness Ranking Loss, to regularize class probabilities.
result The method produces well-ranked confidence estimates and is effective for out-of-distribution detection and active learning.
The reliable measurement of confidence in classifiers' predictions is very important for many applications and is, therefore, an important part of classifier design. Yet, although deep learning has received tremendous attention in recent years, not much progress has been made in quantifying the prediction confidence of…
New method calibrates confidence for object detection and segmentation models.
problem Intrinsically miscalibrated confidence estimates in object detection and segmentation models.
method Introduces multivariate confidence calibration for object detection and segmentation, extending ECE.
result Improves calibration, positively impacts segmentation quality.
Paper tackles weakly supervised learning from similarity-confidence data.
problem Learning binary classifier from unlabeled data pairs with confidence of similarity.
method Proposes an unbiased estimator of classification risk from Sconf data and risk correction scheme.
result Demonstrates effectiveness of proposed methods through experiments.
New priors improve Bayesian neural networks without cooling.
problem Bayesian neural networks underfit on clean datasets.
method Introduce DirClip and confidence priors to replace cooling.
result DirClip and confidence priors outperform cold posterior.
Develops methods for valid and validated confidence sets in multiclass and multilabel prediction.
problem Challenges of typical conformal prediction methods in multiclass and multilabel problems, especially uneven coverage.
method Leverages quantile regression to build methods that always guarantee correct coverage and asymptotically optimal conditional coverage, addressing label interactions with tree-structured classifiers.
result Empirical evaluation suggests more robust coverage of confidence sets.
Paper proposes a method to predict deep neural network confidences with guarantees.
problem Quantifying uncertainty in deep neural networks for safety-critical applications.
method Uses Clopper-Pearson confidence intervals and histogram binning for calibrated prediction.
result Demonstrates the effectiveness of predicted confidences in improving DNN performance and safety.