New findings show second-order scoring rules can't accurately represent epistemic uncertainty.
problem Lack of epistemic uncertainty representation in second-order learners.
method Generalised second-order scoring rules introduced to prove theoretical limitations.
result No loss function incentivizes second-order learners to accurately represent epistemic uncertainty.
The paper introduces new measures for quantifying uncertainty in machine learning.
problem Uncertainty representation and quantification in machine learning.
method Proper scoring rules for aleatoric and epistemic uncertainty quantification.
result Established a natural bridge between credal set and second-order distribution representations of uncertainty.
The paper argues that uncertainty quantification in ML is application-specific and proposes a flexible family of measures.
problem The need for proper uncertainty quantification in machine learning for safety-critical applications.
method A flexible family of uncertainty measures tailored to specific applications, using proper scoring rules to control characteristics.
result Different uncertainty measures are more suitable for different tasks (e.g., selective prediction, out-of-distribution detection, active learning).
The article reviews scoring rules for estimating and evaluating forecasts.
problem Evaluating probabilistic forecasts and estimating probability distributions.
method Mathematical foundations and characterization of scoring rules.
result Important families of scoring rules and their applications in statistics and machine learning.
New scoring rules improve probabilistic classification model evaluation.
problem Traditional scoring rules misalign with the preference for correct classifications.
method Introduces Penalized Brier Score (PBS) and Penalized Logarithmic Loss (PLL) to modify proper scoring rules.
result PBS and PLL better identify optimal checkpoints and early stopping points, leading to superior F1 scores.
This research adapts scoring rules for training survival models, improving predictive performance.
problem Training survival models with traditional methods struggles with censoring.
method Adapting scoring rules for survival analysis, creating a flexible framework for model training.
result Scoring rules can be successfully incorporated into model training, yielding competitive performance.
Study compares multivariate scoring rules for distribution forecasts.
problem Evaluating the discrimination ability of multivariate scoring rules.
method Simulation study comparing energy and variogram scores using historical data.
result Variogram score with p=0.5 outperforms other scores.
Prior-weighted logistic regression has become a standard tool for calibration in speaker recognition. Logistic regression is the optimization of the expected value of the logarithmic scoring rule. We generalize this via a parametric family of proper scoring rules. Our theoretical analysis shows how different members of…
Survival regression method improves log-likelihood scores.
problem Improper scoring rules in survival regression models.
method SurvivalMonotonic-net (SuMo-net) with monotonic neural networks.
result SuMo-net achieves state-of-the-art log-likelihood scores.
Study compares different scoring rules for machine-learned weather forecasts, finding scale-awareness improves forecast realism.
problem Improving the accuracy of machine-learned probabilistic weather forecasts.
method Comparison of scoring rules (CRPS, fair global energy score, graph energy score) and analysis of their impact on forecast field spectra.
result Scale-awareness improves forecast realism, particularly in the tropics.
In recent years, probabilistic forecasting is an emerging topic, which is why there is a growing need of suitable methods for the evaluation of multivariate predictions. We analyze the sensitivity of the most common scoring rules, especially regarding quality of the forecasted dependency structures. Additionally, we pr…
New concept of proper-calibeating extends classic calibrated forecasts to proper scoring rules.
problem Defining and extending calibrated forecasts to proper scoring rules.
method Extending the concepts of calibrated and calibeating forecasts to proper scoring rules and proving their properties.
result Proper-calibration always implies calibration, but proper-calibeating does not necessarily imply calibeating.
Critiques binary classification evaluation methods, advocating for proper scoring rules.
problem The dominance of top-K metrics and fixed-threshold evaluations in machine learning.
method Introduces a decision-theoretic framework mapping evaluation metrics to their use cases, and implements a clipped Brier score variant.
result Demonstrates the clinical utility of proper scoring rules through a Python package, exttt{briertools}.
Proposes measures for uncertainty quantification using proper scoring rules.
problem Uncertainty quantification for prediction tasks.
method Decomposes proper scoring rules into divergence and entropy components, tailoring uncertainty quantification to specific tasks.
result Flexibility in uncertainty quantification improves performance in selective prediction and active learning.
Generative networks minimize predictive scoring rules for probabilistic forecasting.
problem Evaluating and improving probabilistic forecasts using generative models.
method Training generative networks to minimize predictive-sequential scoring rules on temporal sequences.
result Our method outperforms adversarial approaches in probabilistic calibration.
Forecasts of multivariate probability distributions are required for a variety of applications. Scoring rules enable the evaluation of forecast accuracy, and comparison between forecasting methods. We propose a theoretical framework for scoring rules for multivariate distributions, which encompasses the existing quadra…
Proposes a score to compare rule-based algorithms' interpretability.
problem Lack of consensus on interpretability for predictive models.
method Defines a score with three terms: predictivity, stability, and simplicity, each quantified by simple formulas.
result Compares interpretability of rule-based and tree-based algorithms for regression and classification.
We study strictly proper scoring rules in the Reproducing Kernel Hilbert Space. We propose a general Kernel Scoring rule and associated Kernel Divergence. We consider conditions under which the Kernel Score is strictly proper. We then demonstrate that the Kernel Score includes the Maximum Mean Discrepancy as a special …
New scoring rules compare probabilistic top lists in classification.
problem Evaluation of probabilistic top lists in classification.
method Elicitability through symmetric proper scoring rules.
result Brier score provides a well-suited metric for comparison.
Advances rule-based multi-label classification using conformal prediction.
problem Improving accuracy and decision making in multi-label classification.
method Combines conformal prediction with rule-based learning to provide natural conformity scores and calibrate rule assessments.
result Calibrated conformity scores enhance prediction accuracy and decision making.
Signature kernel scoring rule improves weather forecasting by capturing temporal and spatial dependencies.
problem Lack of suitable scoring rules for probabilistic weather forecasting.
method Reframe weather variables as continuous paths using iterated integrals (signature kernels) to capture temporal and spatial dependencies.
result Signature kernel scoring rule outperforms conventional methods in weather forecasting, especially for long-term forecasts.
This paper explores how to choose scoring rules for estimating properties with parametric assumptions.
problem Indirect elicitation of properties with parametric assumptions.
method Developed a framework for choosing proper scoring rules for indirect elicitation, considering constraints and optimal solutions.
result The optimal estimation of the target property changes monotonically with the increase of each weight, and often setting some weights as zero yields the best configuration.
Study identifies regions where scoring rules reliably detect forecast errors.
problem Insufficient reliability of scoring rules in evaluating multivariate probabilistic forecasts.
method Systematic finite-sample analysis of proper scoring rules on synthetic and real-world data.
result Identified regions of reliability for scoring rules in time-series forecasting.
A new bias score method optimizes fairness in classification.
problem Ensuring fairness in binary classification under group constraints.
method Introducing bias scores and developing a post-hoc approach to adapt to fairness constraints.
result The method maintains high accuracy while ensuring fairness constraints.
There has been much recent interest in application of the pool-adjacent-violators (PAV) algorithm for the purpose of calibrating the probabilistic outputs of automatic pattern recognition and machine learning algorithms. Special cost functions, known as proper scoring rules form natural objective functions to judge the…
Paper introduces new metrics for evaluating model accuracy.
problem Improving model accuracy and calibration.
method Developed two second-order accuracy metrics with integral and numerical representations.
result Validates model calibration settings and reveals distortions.
It is the main goal of this article to address the bipartite ranking issue from the perspective of functional data analysis (FDA). Given a training set of independent realizations of a (possibly sampled) second-order random function with a (locally) smooth autocorrelation structure and to which a binary label is random…
We provide self-contained proof of a theorem relating probabilistic coherence of forecasts to their non-domination by rival forecasts with respect to any proper scoring rule. The theorem appears to be new but is closely related to results achieved by other investigators.
Study optimizes scoring rules for incentivizing agent's information gathering in online settings.
problem Optimizing incentives for agents to acquire information in online settings.
method Designing a sample-efficient algorithm that tailors the UCB algorithm to the strategic agent's model.
result Achieves sublinear T2/3-regret after T iterations, independent of the number of states. Fed-FEARE model extracts rules from multiple agencies' data securely.
problem Data privacy and secure rule extraction across multiple agencies.
method Federated F-score based ensemble tree model.
result Model performance measures significantly improved with federated learning.
Conditional forecasts improve performative prediction accuracy.
problem Performative predictions undermine standard forecasting methods.
method Condition forecasts on covariates to make them forecast-invariant.
result Proper scoring rules fail under conditioning, but two solutions are identified.
Generative models learn rules at different timescales, revealing a 'innovation window'.
problem Generative models' convergence to empirical training distribution rather than population distribution.
method Rule-valid synthetic tasks, analyzing τrule and τmem across training timescales. result The 'innovation window' widens with increasing dataset size and narrows with rule complexity.
Binary classification rules based on covariates typically depend on simple loss functions such as zero-one misclassification. Some cases may require more complex loss functions. For example, individual-level monitoring of HIV-infected individuals on antiretroviral therapy (ART) requires periodic assessment of treatment…
A dealer manages quotes and rejection rules to control slippage risk in FX markets.
problem Managing inventory risk and latency risk in OTC FX market making.
method Dynamic programming and adiabatic-quadratic approximation to optimize quotes and rejection rules.
result Developed a method to optimize quotes and rejection rules for managing slippage risk.
ProSMIN improves representation quality through probabilistic self-supervised learning.
problem Improving representation quality in self-supervised learning.
method ProSMIN uses two neural networks, online and target, to learn diverse representations through knowledge distillation and a modified scoring rule loss function.
result ProSMIN achieves superior accuracy and calibration on various downstream tasks.
Extends denoising and score estimation to energy models via Tweedie's formula.
problem Linking denoising and score estimation for a wider range of distributions.
method Derives a fundamental identity connecting energy score derivatives and scores.
result Establishes a new identity for energy scores analogous to Tweedie's formula.
New tree splitting criteria improve probabilistic predictions.
problem Improving tree-based nonparametric predictive distributions.
method Using proper scoring rules for tree splitting criteria.
result Trees with new splitting criteria produce better predictive distributions.
Improves generative models for cost-sensitive decisions.
problem Generative models lack awareness of decision costs.
method Integrates a decision loss into the training objective.
result Improves cost-sensitive forecast accuracy.
This article sets out to serve a dual purpose. On the one hand, we give an explicit description of the Lagrangian submanifolds of the nearly Kaehler 6-sphere which are ruled by circles of constant radius using Weierstrass formulae. On the other, we recognise all previous known examples of these Lagrangians as being rul…
A new screening rule improves lasso solving speed.
problem Efficiently solving lasso problems with high correlation.
method Uses second-order information from the Hessian to screen predictors.
result Outperforms alternatives on simulated and real data.
This work proposes optimal decision rules for hierarchical classifiers to better align with evaluation metrics.
problem Heuristic decision rules in hierarchical classification do not align with evaluation metrics.
method Derives optimal decision rules for various prediction settings, focusing on hierarchical hFβ scores. result Optimal decision rules enhance the performance and reliability of hierarchical classifiers.
Model selection for time series forecasting can be biased by the distribution of scores.
problem Model selection for probabilistic forecasting on time series data.
method Using proper scoring rules to aggregate scores across multiple time series.
result The mean score is immune to the skewness of the score distribution.
This paper proposes a new method to approximate posterior distributions using generative neural networks trained via scoring rule minimization.
problem Bayesian Likelihood-Free Inference for models with intractable likelihood.
method Approximate posterior with generative neural networks trained via scoring rule minimization, avoiding the instability of adversarial training.
result Scoring Rule minimization leads to better performance and uncertainty quantification compared to adversarial training.
DTOR explains anomalies with rule-based explanations.
problem Need to explain anomalies in data effectively.
method Applies Decision Tree Regressor to estimate anomaly scores and generate rule-based explanations.
result DTOR produces robust and consistent rule-based explanations.
A new score function improves explainability and reliability of AI systems.
problem Designing AI systems that are explainable, robust, and trustworthy.
method Integrates conformal prediction with explainable machine learning using a novel score function.
result The method achieves improved performance on target classes and satisfies conformal guarantees.
The paper is mainly devoted to systematic developments and applications of geometric aspects of second-order variational analysis that are revolved around the concept of parabolic regularity of sets. This concept has been known in variational analysis for more than two decades while being largely underinvestigated. We …
We present efficient differentiable implementations of second-order multi-hop reasoning using a large symbolic knowledge base (KB). We introduce a new operation which can be used to compositionally construct second-order multi-hop templates in a neural model, and evaluate a number of alternative implementations, with d…
Improving Bayesian filtering with strictly proper scoring rules
problem Bayesian filtering of partially and noisily observed dynamical systems
method Proper scoring ensemble filter (PSEF)
result Accurate approximation of challenging filtering distributions