Paper introduces a novel method for estimating model confidence in deep neural classifiers.
problem Reliable confidence estimation for deep neural classifiers in safety-critical applications.
method Proposes a novel target criterion (true class probability) and learns it from data with an auxiliary model.
result The proposed method outperforms strong baselines in various tasks and network architectures.
Proposes a new confidence criterion for deep neural networks to predict failures.
problem Predicting failures in deep neural networks.
method Introduces True Class Probability (TCP) as a new confidence criterion and proposes a learning scheme to estimate it.
result The proposed approach consistently outperforms existing methods in failure prediction.
Focal loss improves classification but not class-posterior probability estimation.
problem Improving class-posterior probability estimation from focal loss.
method Proved classification-calibration and derived a transformation to recover true class-posterior probabilities.
result A transformation of the confidence score from focal loss minimization allows recovery of true class-posterior probabilities.
Proposes a Dual Focal Loss method to improve semantic segmentation on unbalanced datasets.
problem Improving performance on unbalanced datasets in semantic segmentation.
method Adaptive Class Weight based Dual Focal Loss (DFL) function.
result DFL improves convergence and reduces class imbalance effect.
Tutorial on estimating SVM class probabilities.
problem Estimating class probabilities for SVM models.
method Compute implied posterior probabilities via isotonic regression.
result Calibrated implied posterior probabilities for SVMs.
AI can learn true probabilities if data and assumptions align.
problem Understanding when AI models can accurately represent true objective probabilities.
method Proved conditions under which AI can learn true probabilities.
result Conditions for learning true probabilities are identified.
New bounds for balanced classification improve understanding of imbalanced datasets.
problem Negligible size of the minority class in imbalanced datasets.
method Developed non-asymptotic and consistent bounds for balanced empirical risk minimization and balanced nearest neighbors estimates.
result Improved understanding of class-weighting benefits in real-world imbalanced classification settings.
Proposes a method to improve class-conditional conformal prediction for many classes.
problem Weak guarantees for specific classes in classification problems.
method Clusters classes with similar conformal scores and performs conformal prediction at the cluster level.
result Clustered conformal typically outperforms existing methods in class-conditional coverage and set size metrics.
In this work we investigate to which extent one can recover class probabilities within the empirical risk minimization (ERM) paradigm. The main aim of our paper is to extend existing results and emphasize the tight relations between empirical risk minimization and class probability estimation. Based on existing literat…
In this paper, we study the classification problem in which we have access to easily obtainable surrogate for true labels, namely complementary labels, which specify classes that observations do \textbf{not} belong to. Let Y and Yˉ be the true and complementary labels, respectively. We first model the annotati…
Paper analyzes proper losses and their performance in machine learning tasks.
problem Understanding the performance of estimators and forecasters in machine learning tasks.
method Analyzes surrogate regret and convergence rates for strictly proper losses.
result Strongly proper losses achieve the optimal convergence rate.
A generative model may generate utter nonsense when it is fit to maximize the likelihood of observed data. This happens due to "model error," i.e., when the true data generating distribution does not fit within the class of generative models being learned. To address this, we propose a model of active distribution lear…
A sequential classifier minimizes test samples for binary and multi-class classification.
problem Minimizing test samples for sequential classification with unknown distributions.
method Proposes a classifier for binary and multi-class problems, analyzing error probabilities and extending results.
result Significant advantage over non-sequential classifiers, achieving same exponents without rejection option.
Study examines how noise in training data affects classification of outliers.
problem Impact of label noise on classification of outlier observations.
method Investigates BCOPS algorithm with synthetic and real datasets.
result Noise in training data significantly impacts model performance for outlier classification.
In most classification tasks there are observations that are ambiguous and therefore difficult to correctly label. Set-valued classifiers output sets of plausible labels rather than a single label, thereby giving a more appropriate and informative treatment to the labeling of ambiguous instances. We introduce a framewo…
The problem is sequence prediction in the following setting. A sequence x1,..., xn,... of discrete-valued observations is generated according to some unknown probabilistic law (measure) mu. After observing each outcome, it is required to give the conditional probabilities of the next observation. The measure mu belongs…
We consider the problem of sequential sampling from a finite number of independent statistical populations to maximize the expected infinite horizon average outcome per period, under a constraint that the expected average sampling cost does not exceed an upper bound. The outcome distributions are not known. We construc…
A new method calculates optimal decisions from classifier outputs, improving predictions in drug discovery.
problem Finding optimal decisions from classifier outputs in fields like medicine.
method Develops a transducer that calculates probabilities from classifier outputs, enabling expected-utility maximization.
result Improves prediction accuracy in drug discovery problems, sometimes close to theoretical maximum.
New theory challenges traditional machine learning assumptions.
problem Traditional machine learning theories are critiqued.
method A new theory is proposed and discussed.
result Learning true probabilities is not equivalent to other learning goals.
Paper derives convergence rates for NPMLE in Hellinger distance using deep neural networks.
problem Difficulty in proving convergence of excess risk in nonparametric logistic regression.
method Unified approach for analyzing NPMLE, deriving convergence rates in Hellinger distance.
result Derives nearly optimal convergence rates for NPMLE with deep neural networks.
Paper analyzes risk bounds for in-context learning in multiclass classification.
problem Risk bounds for in-context learning in multiclass classification.
method Formalizes tasks as sequences of labeled examples and queries, estimates conditional class probabilities, establishes oracle inequality for KL divergence.
result ICL achieves minimax optimal rate for conditional probability estimation.
Study proves convergence of interest rate model approximations.
problem Investigating convergence of stochastic interest rate models.
method Developed analytical tools for true and truncated EM solutions, proving convergence in probability.
result True solution converges in probability to truncated EM solution as step size approaches zero.
The paper proposes a test to determine the number of latent classes in ordinal categorical data.
problem Determining the correct number of latent classes in latent class models with ordinal categorical data.
method The test statistic centers the largest singular value of a normalized residual matrix by a simple sample-size adjustment.
result The test statistic converges to zero under the null hypothesis and exceeds a fixed positive constant under an under-fitted alternative.
Bayesian method identifies causal DAG structure from non-Gaussian errors.
problem Learning causal structure from non-Gaussian errors in Bayesian networks.
method Bayesian hierarchical model with DAG prior for non-Gaussian errors.
result Posterior DAG selection consistency achieved under mild assumptions.
New methods protect privacy while providing accurate prediction sets.
problem Privacy-preserving conformal prediction for untrusted aggregators.
method Two LDP approaches: k-ary randomized response and binary search response.
result Finite-sample coverage guarantees and robust coverage under randomization.
The paper proposes a method to calibrate evidential clustering using bootstrapped finite mixture models.
problem Representing uncertainty in cluster membership using Dempster-Shafer mass functions.
method Constructing Dempster-Shafer mass functions by bootstrapping finite mixture models, computing confidence intervals, and calibrating the evidential partition.
result The proposed method calibrates the evidential partition such that the belief and plausibility degrees approximate the true probabilities with high confidence.
New nonparametric estimators improve causal effect estimation.
problem Estimation of causal effects with selection bias.
method Undersmoothing of the highly adaptive lasso for estimating the weighting mechanism.
result Asymptotic efficiency and convergence to nonparametric efficiency bound.
This paper studies active learning in the context of robust statistics. Specifically, we propose a variant of the Best Arm Identification problem for \emph{contaminated bandits}, where each arm pull has probability ε of generating a sample from an arbitrary contamination distribution instead of the true und…
We derive the fast convergence rates of a deep neural network (DNN) classifier with the rectified linear unit (ReLU) activation function learned using the hinge loss. We consider three cases for a true model: (1) a smooth decision boundary, (2) smooth conditional class probability, and (3) the margin condition (i.e., t…
We investigate a class of hierarchical mixtures-of-experts (HME) models where exponential family regression models with generalized linear mean functions of the form psi(ga+fx^Tfgb) are mixed. Here psi(...) is the inverse link function. Suppose the true response y follows an exponential family regression model with mea…
Undirected graphical models, or Markov networks, are a popular class of statistical models, used in a wide variety of applications. Popular instances of this class include Gaussian graphical models and Ising models. In many settings, however, it might not be clear which subclass of graphical models to use, particularly…
This paper considers the classification of linear subspaces with mismatched classifiers. In particular, we assume a model where one observes signals in the presence of isotropic Gaussian noise and the distribution of the signals conditioned on a given class is Gaussian with a zero mean and a low-rank covariance matrix.…
In safety-critical applications of machine learning, it is often important to abstain from making predictions on low confidence examples. Standard abstention methods tend to be focused on optimizing top-k accuracy, but in many applications, accuracy is not the metric of interest. Further, label shift (a shift in class …
The problem of pricing Bermudan options using Monte Carlo and a nonparametric regression is considered. We derive optimal non-asymptotic bounds for a lower biased estimate based on the suboptimal stopping rule constructed using some estimates of continuation values. These estimates may be of different nature, they may …
The paper develops bounds for predictive values in binary classification.
problem Lack of confidence intervals for positive and negative predictive values.
method Bi-criterion framework and distribution-free large deviation and uniform convergence bounds.
result New bounds for predictive values without relying on concentration inequalities.
A new framework for probabilistic learning using Maximum Probability Theorem.
problem Challenges in defining and quantifying model probabilities in probabilistic learning.
method Introduces a new probabilistic framework based on Maximum Probability Theorem, defining models as events with quantified probability measures.
result The probability of a model is invariant to reparameterization and depends solely on the likelihood function.
Robust portfolio optimization considers uncertainty in market probabilities.
problem Uncertainty in market probabilities in multiperiod portfolio selection.
method Robust mean-variance optimization using Wasserstein ball centered at empirical data.
result Numerical simulations show improved performance compared to other strategies.
Develops methods to improve reliability of deep learning for autonomous driving.
problem Safety concerns in deploying autonomous driving systems.
method Introduces a new criterion (true class probability) for estimating model confidence and learns it from data.
result Proposed method provides better failure prediction than current uncertainty measures.
Noise-Aware Conformal Prediction (NACP) calibrates CP for noisy labels.
problem Calibrating Conformal Prediction with noisy labels.
method Estimate conformal threshold from noisy labels using uniform noise coverage guarantee.
result Finite sample coverage guarantee for uniform noise remains effective in high-class tasks.
The paper proposes a method to estimate joint probability from unpaired data using entropic transport kernels.
problem Estimating joint probability from unpaired data with unknown internal ordering.
method Maximum-likelihood inference, entropic optimal transport kernels, EMML algorithm.
result The method can recover true density from empirical approximations as the number of blocks increases.
In cases of uncertainty, a multi-class classifier preferably returns a set of candidate classes instead of predicting a single class label with little guarantee. More precisely, the classifier should strive for an optimal balance between the correctness (the true class is among the candidates) and the precision (the ca…
New bounds quantify estimation error in kernel-based system identification with unknown hyperparameters.
problem Inaccurate error bounds for kernel-based system identification with unknown hyperparameters.
method Construct a high-probability set for true hyperparameters from marginal likelihood, then find worst-case posterior covariance.
result Proposed bounds contain true model with high probability and verified in simulations.
AdaDetectGPT improves text authorship detection with statistical guarantees.
problem Determining if text is authored by a human or an LLM.
method Adaptive learning of a witness function from training data to enhance logits-based detectors.
result AdaDetectGPT nearly uniformly improves text authorship detection, with up to 37% improvement.
Paper proves robust estimators' generalization guarantees without dimensionality issues.
problem Generalization guarantees for Wasserstein distributionally robust models.
method Analyzes and extends existing guarantees to broader classes of models and regularized versions.
result Generalization guarantees hold without dimensionality issues and cover distribution shifts.
Study optimizes tree-based models for better alignment of predicted scores and actual probabilities.
problem Traditional calibration metrics fail to align predicted scores with actual probabilities when score distributions deviate from the underlying data.
method Optimizes tree-based models (Random Forest, XGBoost) using Kullback-Leibler (KL) divergence to minimize the difference between predicted and true probability distributions.
result Optimized tree-based models yield superior alignment between predicted scores and actual probabilities without significant performance loss.
Estimates mean of random vector with near-optimal error in all directions.
problem Estimating the mean of a random vector with direction-dependent accuracy.
method Proves existence of an estimator with near-optimal error in all directions under certain conditions.
result The estimator satisfies the error bound for all directions, with probability 1-δ.
In many signal detection and classification problems, we have knowledge of the distribution under each hypothesis, but not the prior probabilities. This paper is aimed at providing theory to quantify the performance of detection via estimating prior probabilities from either labeled or unlabeled training data. The erro…
Bayesian method estimates coverage from sketching imperfect data.
problem Estimating coverage probabilities from compressed data.
method Bayesian nonparametric approach using Dirichlet process prior.
result Estimators accurately recover distinct counts and frequencies.