Paper develops proper, lower-bounded losses for weakly supervised classification.
problem Weakly supervised classification with corrupted labels.
method Representation theorem for proper losses, derived condition for lower-boundedness, generalized logit squeezing.
result Proper and lower-bounded losses for weak-label learning.
Paper analyzes proper losses and their performance in machine learning tasks.
problem Understanding the performance of estimators and forecasters in machine learning tasks.
method Analyzes surrogate regret and convergence rates for strictly proper losses.
result Strongly proper losses achieve the optimal convergence rate.
We study proper losses for discrete generative models without knowing the target distribution.
problem Evaluating generative models in the discrete setting without direct access to the target distribution.
method Define and construct black-box proper losses using statistical estimation theory.
result Black-box proper losses must be of polynomial form and involve more samples than the polynomial degree.
We study losses for binary classification and class probability estimation and extend the understanding of them from margin losses to general composite losses which are the composition of a proper loss with a link function. We characterise when margin losses can be proper composite losses, explicitly show how to determ…
This work broadens calibeating to various proper losses using Bregman divergence.
problem Calibration for a wide range of proper losses.
method Regret minimization and Bregman divergence approach.
result U-calibration results for a family of Tsallis losses with logarithmic regret and dimension independence.
This work generalizes calibeating for a broader range of proper losses using Bregman divergence.
problem Calibration for a wide range of proper losses beyond Brier and log loss.
method Regret minimization based on Bregman divergence for a family of proper losses.
result U-calibration results for a family of Tsallis losses with logarithmic regret and dimension independence.
Optimizing proper loss yields calibrated models under specific conditions.
problem Understanding when optimizing proper loss functions leads to calibrated predictions.
method Local optimality condition and Lipschitz functions.
result Predictors with local optimality are nearly calibrated and nearly locally optimal.
Study on proper learning under relaxed worst-case robust loss for VC classes.
problem Proper adversarially robust PAC learning under relaxed worst-case robust loss.
method Introduced a family of robust loss relaxations and showed their effectiveness for proper learnability.
result VC classes are properly PAC learnable with sample complexity close to standard PAC learning setup.
A new method learns proper multiclass losses and probabilities.
problem Learning proper multiclass losses for complex classification tasks.
method Extends monotonicity to multiclass problems using convex functions.
result Consistently outperforms natural multiclass baseline on up to 1,000 class datasets.
Optimal multiclass U-calibration error found to be Θ(√KT).
problem Online multiclass U-calibration with low regret for all bounded proper losses.
method Follow-the-Perturbed-Leader algorithm and lower bound construction.
result Optimal U-calibration error is Θ(√KT).
New algorithm reduces prediction errors across various loss functions.
problem Online forecasting algorithms' inability to adapt to different loss functions.
method Design of a novel Follow-the-Perturbed-Leader (FTPL) algorithm with self-concordant noise.
result Simultaneously achieves i l d e O ( T ) ilde O(\sqrt{T}) i l d e O ( T ) regret for bounded proper losses and O ( log T ) O(\log T) O ( log T ) regret for bounded smooth proper losses. The goal of online prediction with expert advice is to find a decision strategy which will perform almost as well as the best expert in a given pool of experts, on any sequence of outcomes. This problem has been widely studied and O ( T ) O(\sqrt{T}) O ( T ) and O ( log T ) O(\log{T}) O ( log T ) regret bounds can be achieved for convex losses (\cite{zin…
The paper decomposes probabilistic scores into reliability, uncertainty, and information loss.
problem Understanding the reliability and uncertainty of probabilistic predictions.
method Developed decomposition identities for proper losses, quantifying reliability, residual uncertainty, and information gain.
result A three-term identity for classification scores, revealing miscalibration, grouping term, and feature-level uncertainty.
The problem of bipartite ranking, where instances are labeled positive or negative and the goal is to learn a scoring function that minimizes the probability of mis-ranking a pair of positive and negative instances (or equivalently, that maximizes the area under the ROC curve), has been widely studied in recent years. …
Proposes measures for uncertainty quantification using proper scoring rules.
problem Uncertainty quantification for prediction tasks.
method Decomposes proper scoring rules into divergence and entropy components, tailoring uncertainty quantification to specific tasks.
result Flexibility in uncertainty quantification improves performance in selective prediction and active learning.
Paper proposes fitting loss functions to data using source functions from information geometry.
problem Choosing appropriate loss functions for machine learning models.
method Introduces source functions from information geometry to fit loss functions to the domain at hand.
result Significant improvements over state-of-the-art methods in model training.
New algorithms minimize dynamic regret for strongly convex losses.
problem Minimizing dynamic regret for strongly convex losses.
method Developed Strongly Adaptive algorithms exploiting KKT conditions.
result Achieved near optimal dynamic regret of O ( d 1 / 3 n 1 / 3 e x t T V [ u 1 : n ] 2 / 3 ∨ d ) O(d^{1/3} n^{1/3} ext{TV}[u_{1:n}]^{2/3} \vee d) O ( d 1/3 n 1/3 e x t T V [ u 1 : n ] 2/3 ∨ d ) . Improves generative models for cost-sensitive decisions.
problem Generative models lack awareness of decision costs.
method Integrates a decision loss into the training objective.
result Improves cost-sensitive forecast accuracy.
New method improves decision-making accuracy without complex calculations.
problem Improving decision-making accuracy in machine learning.
method Introducing a new measure called calibration decision loss ( C D L K \mathsf{CDL}_K CDL K ) for structured families of post-processing functions. result Proves upper and lower bounds for natural classes K K K of post-processing functions. Generative Cross-Entropy improves classification with fewer labels.
problem Limited sample efficiency of cross-entropy loss in data-scarce scenarios.
method Proposes Generative Cross-Entropy (GenCE), a new loss function that incorporates generative principles into a standard discriminative network.
result Generative Cross-Entropy outperforms traditional cross-entropy loss across various datasets and conditions.
We consider composite loss functions for multiclass prediction comprising a proper (i.e., Fisher-consistent) loss over probability distributions and an inverse link function. We establish conditions for their (strong) convexity and explore the implications. We also show how the separation of concerns afforded by using …
Optimizes predictions by recalibrating online forecasts with minimal error.
problem Tackles the challenge of recalibrating online predictions to be more accurate.
method Uses an imbalanced extension of the Blackwell approachability reduction framework to achieve ( ε , ε 2 ) (\varepsilon, \varepsilon^2) ( ε , ε 2 ) -recalibration. result Achieves ( ε , ε 2 ) (\varepsilon, \varepsilon^2) ( ε , ε 2 ) -recalibration for Lipschitz proper losses in T ≈ ε − 3 T \approx \varepsilon^{-3} T ≈ ε − 3 rounds. This work examines uncertainty sampling in binary classification using equivalent loss.
problem Lack of consensus on proper uncertainty definition and theoretical guarantees for active learning.
method Systematically examines uncertainty sampling via equivalent loss, proving its optimality.
result Established that uncertainty sampling optimizes against equivalent loss, providing theoretical guarantees.
The study of a machine learning problem is in many ways is difficult to separate from the study of the loss function being used. One avenue of inquiry has been to look at these loss functions in terms of their properties as scoring rules via the proper-composite representation, in which predictions are mapped to probab…
A new method decomposes subjective risk into epistemic and aleatoric uncertainties.
problem Uncertainty quantification in modeling decisions.
method Subjective risk decomposition using strictly proper loss.
result Recovery of classic uncertainty measures and new learning-theoretic connections.
Estimates proper calibration errors and refinement terms in probabilistic predictions.
problem Lack of a general estimator for proper calibration errors and refinement terms with known statistical properties.
method Proposes a method for consistent, asymptotically unbiased estimation of proper calibration errors and refinement terms.
result Proves the relation between refinement and f-divergences, implying information monotonicity in neural networks.
We study strictly proper scoring rules in the Reproducing Kernel Hilbert Space. We propose a general Kernel Scoring rule and associated Kernel Divergence. We consider conditions under which the Kernel Score is strictly proper. We then demonstrate that the Kernel Score includes the Maximum Mean Discrepancy as a special …
We analyze impermanent loss in AMMs and show G3Ms are simplest.
problem Understanding impermanent loss in automated market makers.
method Developed a general framework and analyzed Geometric Mean Market Makers (G3Ms).
result G3Ms have the simplest impermanent loss characteristics.
The paper argues that uncertainty quantification in ML is application-specific and proposes a flexible family of measures.
problem The need for proper uncertainty quantification in machine learning for safety-critical applications.
method A flexible family of uncertainty measures tailored to specific applications, using proper scoring rules to control characteristics.
result Different uncertainty measures are more suitable for different tasks (e.g., selective prediction, out-of-distribution detection, active learning).
A novel framework quantifies uncertainty using proper scores for various tasks.
problem Uncertainty quantification in machine learning for reliable applications.
method Proposes a general framework based on proper scores for epistemic, aleatoric uncertainty, and model calibration.
result Achieves state-of-the-art uncertainty estimation for large language models and generative models.
New algorithms achieve near-optimal cumulative loss in nonparametric online learning and games.
problem Fast rates of convergence in nonparametric online regression and classification.
method Randomized proper learning algorithms, hierarchical aggregation, multi-scale extension, stability proof.
result Achieved near-optimal cumulative loss bounds for real-valued and binary games.
SurvivalBoost improves prediction of event times in competing risks scenarios.
problem Predicting event times in scenarios with multiple possible outcomes.
method Developed a strictly proper censoring-adjusted scoring rule for stochastic optimization of competing risks.
result SurvivalBoost outperforms 12 state-of-the-art models across various metrics.
The concept of refinement from probability elicitation is considered for proper scoring rules. Taking directions from the axioms of probability, refinement is further clarified using a Hilbert space interpretation and reformulated into the underlying data distribution setting where connections to maximal marginal diver…
The last few years have seen a staggering number of empirical studies of the robustness of neural networks in a model of adversarial perturbations of their inputs. Most rely on an adversary which carries out local modifications within prescribed balls. None however has so far questioned the broader picture: how to fram…
Critiques binary classification evaluation methods, advocating for proper scoring rules.
problem The dominance of top-K metrics and fixed-threshold evaluations in machine learning.
method Introduces a decision-theoretic framework mapping evaluation metrics to their use cases, and implements a clipped Brier score variant.
result Demonstrates the clinical utility of proper scoring rules through a Python package, exttt{briertools}.
Deep models can fit noisy labels, but robustness and reliability are still issues.
problem Training deep models with noisy labels leads to unreliable uncertainty quantification.
method Analysis of conditional distribution over noisy labels and evaluation of robust loss functions.
result Strictly proper and robust loss functions preserve accuracy but do not guarantee reliability.
This paper argues against using calibration metrics for assessing posterior probabilities and proposes expected proper scoring rules instead.
problem The assessment of posterior probabilities generated by machine learning classifiers using calibration metrics is flawed and should be replaced with expected proper scoring rules.
method The paper reviews proper scoring rules from a practical perspective, explains why expected PSRs are a principled measure of posterior quality, and introduces a new calibration metric called calibration loss.
result Calibration loss is superior to expected calibration error and expected score divergence calibration metrics for assessing posterior probabilities.
Transformer models outperform LSTM in financial forecasting with MADL loss.
problem Optimizing loss functions for Transformer models in financial forecasting.
method Empirical experiments with MADL loss function on equity and cryptocurrency assets.
result Transformer models significantly outperform LSTM models in financial forecasting.
Proper learning is possible with labeled data, but unlabeled data can improve performance.
problem Problems that can only be learned improperly, like multiclass classification.
method Distributional regularization and worst-case performance evaluation.
result Proper learnability is possible under certain conditions involving unlabeled data.
Boosted DT classifiers become DP with new calibrated loss.
problem Making boosted DT classifiers differentially private.
method Crafted M α α α -loss and objective calibration. result Significantly outperforms random forests in DP settings.
New findings show second-order scoring rules can't accurately represent epistemic uncertainty.
problem Lack of epistemic uncertainty representation in second-order learners.
method Generalised second-order scoring rules introduced to prove theoretical limitations.
result No loss function incentivizes second-order learners to accurately represent epistemic uncertainty.
The paper introduces new measures for quantifying uncertainty in machine learning.
problem Uncertainty representation and quantification in machine learning.
method Proper scoring rules for aleatoric and epistemic uncertainty quantification.
result Established a natural bridge between credal set and second-order distribution representations of uncertainty.
This work explores alternative learning criteria beyond traditional risk.
problem The trade-offs of optimizing for average performance in machine learning.
method Survey and introduction of non-traditional learning criteria.
result Emphasizes the importance of considering the loss distribution for desirable performance.
New scoring rules improve probabilistic classification model evaluation.
problem Traditional scoring rules misalign with the preference for correct classifications.
method Introduces Penalized Brier Score (PBS) and Penalized Logarithmic Loss (PLL) to modify proper scoring rules.
result PBS and PLL better identify optimal checkpoints and early stopping points, leading to superior F1 scores.
New method estimates grouping loss in neural networks to improve confidence scores.
problem Improving confidence scores in neural networks to reflect true posterior probabilities.
method Proposed an estimator to approximate the grouping loss.
result Modern neural networks exhibit grouping loss, especially in distribution shifts.
We develop efficient algorithms to train ℓ 1 \ell_1 ℓ 1 -regularized linear classifiers with large dimensionality d d d of the feature space, number of classes k k k , and sample size n n n . Our focus is on a special class of losses that includes, in particular, the multiclass hinge and logistic losses. Our approach combines several…
This work improves diffusion models by estimating the optimal loss value for better training diagnostics.
problem The optimal loss value of diffusion models is unknown and not indicative of absolute data-fitting quality.
method Derive the optimal loss in closed form and develop effective estimators, including a stochastic variant.
result Unlocking the optimal loss as a metric for diagnosing training quality of diffusion models.
Focal loss improves classification but not class-posterior probability estimation.
problem Improving class-posterior probability estimation from focal loss.
method Proved classification-calibration and derived a transformation to recover true class-posterior probabilities.
result A transformation of the confidence score from focal loss minimization allows recovery of true class-posterior probabilities.