The paper extends calibration to sets of probabilistic classifiers, finding many ensembles are poorly calibrated.
problem Evaluating the validity of epistemic uncertainty in sets of probabilistic classifiers.
method Proposed a novel nonparametric calibration test for sets of probabilistic classifiers.
result Ensembles of deep neural networks are often not well calibrated.
Develops probabilistic safety regions for scalable classifiers.
problem Minimizing misclassification errors in supervised classification.
method Introduces probabilistic safety regions and scalable classifiers.
result Probabilistic certifications for classifier performance.
A new attack for probabilistic classifiers adapts to noise levels.
problem Adversarial examples can mislead probabilistic classifiers.
method Adapts HopSkipJump attack for probabilistic classifiers, adjusting queries based on noise levels.
result Decision-based attacks are effective against probabilistic classifiers, even with noise.
A framework assesses the trustworthiness of probabilistic classifiers using local calibration error.
problem Assessing the trustworthiness of probabilistic classifiers beyond traditional metrics.
method I-trustworthy framework linking local calibration to trustworthiness; Kernel Local Calibration Error (KLCE) method for hypothesis testing.
result The effectiveness of the proposed test statistic demonstrated through simulated and real-world datasets.
Calibrating classifiers reduces grouping loss using sufficiency criteria.
problem Grouping loss in probabilistic classifier calibration is often overlooked.
method Revisited Langford & Zadrozny's probing reduction approach and introduced Brier curves.
result The probing reduction approach reduces grouping loss and supports sufficient calibration.
A method for consensus prediction from probabilistic classifier outputs.
problem Combining probabilistic classifier outputs without prior information.
method Iteratively updating predictions based on classifier diversity and convergence.
result Consensus predictions converge to a decision based on classifier outputs' correspondence.
Probabilistic classifiers output a probability distribution on target classes rather than just a class prediction. Besides providing a clear separation of prediction and decision making, the main advantage of probabilistic models is their ability to represent uncertainty about predictions. In safety-critical applicatio…
ProBoost boosts probabilistic classifiers by focusing on uncertain samples.
problem Improving probabilistic classifiers through targeted learning.
method ProBoost uses epistemic uncertainty to select challenging samples, increasing their weight for subsequent learners.
result ProBoost significantly improves classifier performance, especially with few weak learners.
Transfer learning assumes classifiers of similar tasks share certain parameter structures. Unfortunately, modern classifiers uses sophisticated feature representations with huge parameter spaces which lead to costly transfer. Under the impression that changes from one classifier to another should be ``simple'', an effi…
The paper analyzes recalibration methods for binary classifiers under distribution shift.
problem Recalibrating binary classifiers to match a target prior probability.
method Analysis of distribution shift assumptions and proposal of new recalibration methods.
result QMM methods provide conservative results for risk weights functions.
Proposes and evaluates three diagnostic graphics for probabilistic classifiers.
problem Evaluating and comparing probabilistic classifiers.
method Triptych of diagnostic graphics: reliability diagram, ROC curve, Murphy diagram.
result Visual diagnostics reveal distinct aspects of forecast performance.
Bayesian model fuses multiple classifiers with explicit correlation modeling.
problem Combining outputs of multiple classifiers with explicit correlation.
method Hierarchical Bayesian model with correlated Dirichlet distribution.
result Fused classifier performance can be Bayes optimal even for highly correlated base classifiers.
Probabilistic-driven classification techniques extend the role of traditional approaches that output labels (usually integer numbers) only. Such techniques are more fruitful when dealing with problems where one is not interested in recognition/identification only, but also into monitoring the behavior of consumers and/…
The paper presents a probabilistic framework for SPD matrices in machine learning.
problem Machine learning on SPD matrices is fragmented; this paper aims to unify it.
method Unified probabilistic framework using Gaussian distributions and Bayes classifiers.
result Different SPD machine learning tools can be reinterpreted and extended using Gaussian distributions.
New framework uses conformal predictions for robust, scalable machine learning classification.
problem Developing robust and reliable machine learning models for classification.
method Introducing scalable classifiers linked to statistical order theory and probabilistic learning theory, defining a score function and conformal safety set.
result Demonstrated practical implications in cybersecurity for identifying DNS tunneling attacks.
Paper uses GMM and MAF for probabilistic classification, outperforming simpler models.
problem Classifying data with complex distributions.
method Density estimation using Gaussian Mixture Model and Masked Autoregressive Flow.
result Proposed classifiers outperform simpler models like linear discriminant analysis.
Accurate calibration of probabilistic predictive models learned is critical for many practical prediction and decision-making tasks. There are two main categories of methods for building calibrated classifiers. One approach is to develop methods for learning probabilistic models that are well-calibrated, ab initio. The…
In this paper we formulate a probabilistic model for class-specific discriminant subspace learning. The proposed model can naturally incorporate the multi-modal structure of the negative class, which is neglected by existing class-specific methods. Moreover, it can be directly used to define a class-specific probabilis…
Combines coarse learners for nonparametric probabilistic regression.
problem Predicting full probability density functions without strong assumptions.
method Combining gradient boosted forests trained on coarsened target values.
result Prediction intervals have high fidelity and provide valuable insights.
Unified approach to non-standard classification tasks.
problem Non-standard classification tasks like semi-supervised, positive-unlabelled, multi-positive-unlabelled and noisy-label learning.
method Probabilistic, unified approach training a classifier to predict label-distributions, then inferring class-distributions.
result Unified model for various non-standard classification tasks.
New method controls classifier guidance in diffusion models.
problem Improving classifier guidance in diffusion models.
method Cross-entropy control of classifier gradients.
result Effective guidance vectors with mean squared error O(dε). We define a generalized likelihood function based on uncertainty measures and show that maximizing such a likelihood function for different measures induces different types of classifiers. In the probabilistic framework, we obtain classifiers that optimize the cross-entropy function. In the possibilistic framework, we …
This paper introduces minimum-risk recalibration for probabilistic classifiers, improving their reliability and accuracy.
problem Improving the reliability and accuracy of probabilistic classifiers.
method Minimum-risk recalibration within the MSE decomposition framework, analyzing UMB method and label shift adaptation.
result The optimal number of bins for UMB scales with n1/3, resulting in a risk bound of approximately O(n−2/3). Two methods improve calibration of probabilistic classifiers, especially for multi-class problems.
problem Improving calibration of probabilistic classifiers, especially for multi-class problems.
method Two techniques: reduced calibration and class-wise calibration.
result Class-wise reduced calibration algorithms reduce prediction and per-class calibration errors.
Introduces Rashomon Capacity to measure predictive multiplicity in probabilistic classifiers.
problem Predictive multiplicity in classification models leading to unjustified decisions.
method Introduces Rashomon Capacity, a metric for probabilistic classifiers, and provides a rigorous derivation.
result Rashomon Capacity captures nuanced score variations and provides strategies for disclosing conflicting models.
New scoring rules compare probabilistic top lists in classification.
problem Evaluation of probabilistic top lists in classification.
method Elicitability through symmetric proper scoring rules.
result Brier score provides a well-suited metric for comparison.
Proposes a new framework for open set recognition using conditional probabilistic generative models.
problem Unknown samples can mislead traditional deep neural networks during testing.
method Conditional Probabilistic Generative Models (CPGM) that combine generative models with discriminative information.
result Significantly outperforms baselines on multiple benchmark datasets.
Classifiers based on probabilistic graphical models are very effective. In continuous domains, maximum likelihood is usually used to assess the predictions of those classifiers. When data is scarce, this can easily lead to overfitting. In any probabilistic setting, Bayesian averaging (BA) provides theoretically optimal…
We consider a discriminative learning (regression) problem, whereby the regression function is a convex combination of k linear classifiers. Existing approaches are based on the EM algorithm, or similar techniques, without provable guarantees. We develop a simple method based on spectral techniques and a `mirroring' tr…
Unified approach for fair classification with overlapping groups.
problem Ensuring fairness across multiple overlapping groups in prediction problems.
method Probabilistic population analysis leading to Bayes-optimal classifier, unifying existing methods.
result Outperforms baselines in fairness-performance tradeoff on real datasets.
Model misspecification is a long-standing enigma of the Bayesian inference framework as posteriors tend to get overly concentrated on ill-informed parameter values towards the large sample limit. Tempering of the likelihood has been established as a safer way to do updates from prior to posterior in the presence of mod…
Though neural network models demonstrate impressive performance, we do not understand exactly how these black-box models make individual predictions. This drawback has led to substantial research devoted to understand these models in areas such as robustness, interpretability, and generalization ability. In this paper,…
pRSL combines probabilistic rules to improve multi-label classification.
problem Modeling the structure between multi-label classes for better performance.
method Uses probabilistic propositional logic rules and belief propagation to combine predictions from multiple classifiers.
result pRSL achieves state-of-the-art performance on various benchmark datasets.
Paper improves deep learning for instance-level classification from label proportions.
problem Dealing with noisy pseudo-labeling and high-entropy class distributions in LLP.
method Introducing a two-stage training approach with constrained optimization and mixup strategy.
result Significant performance improvement in instance-level classification.
We investigate whether the bid/ask queue imbalance in a limit order book (LOB) provides significant predictive power for the direction of the next mid-price movement. We consider this question both in the context of a simple binary classifier, which seeks to predict the direction of the next mid-price movement, and a p…
SPPL simplifies probabilistic programming for exact inference.
problem Efficient exact inference in probabilistic models.
method SPPL translates probabilistic programs into sum-product expressions, leveraging new techniques for scalability.
result SPPL achieves up to 3500x speedups in exact inference.
This paper introduces a probabilistic framework for k-shot image classification. The goal is to generalise from an initial large-scale classification task to a separate task comprising new classes and small numbers of examples. The new approach not only leverages the feature-based representation learned by a neural net…
Proposes a probabilistic method for generating semantically-aware adversarial examples.
problem Generating adversarial examples that are difficult for humans to detect while preserving semantics.
method Embeds subjective understanding of semantics as a distribution into adversarial example generation.
result Achieves higher success rates in circumventing adversarial defense mechanisms.
We present a novel approach for supervised domain adaptation that is based upon the probabilistic framework of Gaussian processes (GPs). Specifically, we introduce domain-specific GPs as local experts for facial expression classification from face images. The adaptation of the classifier is facilitated in probabilistic…
SkewPNN uses probabilistic neural networks with skew-normal kernels to improve classification of imbalanced data.
problem Imbalanced data distribution leading to biased predictions for minority classes.
method Probabilistic neural networks with skew-normal kernel function and Bat optimization algorithm for hyperparameter tuning.
result SkewPNN and BA-SkewPNN outperform other methods in both balanced and imbalanced datasets.
A test detects unfairness in machine learning classifiers.
problem Detecting and mitigating algorithmic biases in machine learning.
method Optimal transport theory to quantify and mitigate bias.
result Proposes a statistical test for detecting unfair classifiers.
ProHOC detects OOD samples in class hierarchies, predicting them to correct internal nodes.
problem Binary OOD detection ignores semantic relationships between OOD and ID classes.
method Probabilistic hierarchical model using multi-depth networks trained for ID classification.
result ProHOC effectively classifies OOD samples to their correct internal nodes in class hierarchies.
DCK improves air quality index prediction with probabilistic spatial models.
problem Non-Gaussian, complex spatial structure of air quality index.
method Deep classifier kriging (DCK) for non-Gaussian, nonlinear spatial prediction.
result DCK outperforms conventional methods in predictive accuracy and uncertainty quantification.
ProSub uses angles in feature space to classify data as in- or out-of-distribution.
problem Open-set semi-supervised learning with unknown classes.
method Probabilistic approach based on angles in feature space, estimating conditional distributions of scores.
result ProSub achieves state-of-the-art performance on benchmark problems.
The study quantifies decision-making risks from suboptimal classifiers and proposes methods to reduce these risks.
problem Excess risk in decision-making from suboptimal probabilistic classifiers.
method Analytical expressions and upper/lower bounds for excess risk, calibration curve estimation, grouping loss estimator.
result Identifies regimes where recalibration alone or post-training is more effective.
New robustness metric helps select reliable classifiers.
problem Evaluating reliability of classifier predictions.
method Proposed new robustness metric for any classifier and feature type.
result Demonstrated ability to distinguish reliable from unreliable predictions.
Majority-of-Three is Optimal
problem Optimality of Voting Learners
method Majority vote of three classifiers
result Proves optimality for the simplest voting scheme
We develop and demonstrate a probabilistic method for classifying rare objects in surveys with the particular goal of building very pure samples. It works by modifying the output probabilities from a classifier so as to accommodate our expectation (priors) concerning the relative frequencies of different classes of obj…