The paper presents an R package for learning Bayesian networks from epidemiological data.
problem Learning Bayesian networks from messy, highly correlated datasets in epidemiology.
method The paper introduces an R package abn that implements multiple frequentist scoring rules for learning Bayesian networks from observational data, addressing data separation and adjustment issues.
result The package abn is robust and efficient for learning Bayesian networks from epidemiological data.
Unified taxonomy for ML uncertainty in physics, validated.
problem Uncertainty quantification in machine learning for physics.
method Unified taxonomy, principled validation tools.
result Illustrated validation tools with examples.
Scores measure certainty and doubt in classification predictions.
problem Quantitative uncertainty assessment in classification problems.
method Intuitive scores in Bayesian and frequentist frameworks.
result Measures assess and compare prediction quality and uncertainty.
A method for concept-based learning using probabilistic inference and expert rules.
problem Concept-based learning with limited training data.
method Divide images into patches, transform into embeddings, cluster, and use frequentist inference to find concepts.
result FI-CBL outperforms concept bottleneck model in small data scenarios.
In this paper, we derive a Bayesian model order selection rule by using the exponentially embedded family method, termed Bayesian EEF. Unlike many other Bayesian model selection methods, the Bayesian EEF can use vague proper priors and improper noninformative priors to be objective in the elicitation of parameter prior…
The article reviews scoring rules for estimating and evaluating forecasts.
problem Evaluating probabilistic forecasts and estimating probability distributions.
method Mathematical foundations and characterization of scoring rules.
result Important families of scoring rules and their applications in statistics and machine learning.
New scoring rules improve probabilistic classification model evaluation.
problem Traditional scoring rules misalign with the preference for correct classifications.
method Introduces Penalized Brier Score (PBS) and Penalized Logarithmic Loss (PLL) to modify proper scoring rules.
result PBS and PLL better identify optimal checkpoints and early stopping points, leading to superior F1 scores.
Study evaluates multivariate forecasting scoring rules and proposes new copula-based ones.
problem Evaluating and improving multivariate probabilistic forecasting methods.
method Analysis and comparison of existing scoring rules, development of copula-based scoring rules, simulation studies, and real data analysis.
result Proposed copula-based scoring rules provide strong distinction between models with correct and incorrect dependency structures.
This research adapts scoring rules for training survival models, improving predictive performance.
problem Training survival models with traditional methods struggles with censoring.
method Adapting scoring rules for survival analysis, creating a flexible framework for model training.
result Scoring rules can be successfully incorporated into model training, yielding competitive performance.
Study compares multivariate scoring rules for distribution forecasts.
problem Evaluating the discrimination ability of multivariate scoring rules.
method Simulation study comparing energy and variogram scores using historical data.
result Variogram score with p=0.5 outperforms other scores.
Prior-weighted logistic regression has become a standard tool for calibration in speaker recognition. Logistic regression is the optimization of the expected value of the logarithmic scoring rule. We generalize this via a parametric family of proper scoring rules. Our theoretical analysis shows how different members of…
New scoring rules for multivariate distributions and level sets.
problem Evaluating forecast accuracy for multivariate distributions and level sets.
method Theoretical framework for scoring rules, decomposition of multivariate scoring functions, numerical algorithm for computation.
result New scoring functions for multivariate distributions and level sets, including density and cumulative distribution level sets.
The study compares Bayesian and frequentist approaches in deep learning.
problem Comparing Bayesian and frequentist inference in deep learning.
method Conducts a comparative analysis of point and posterior estimators across various settings.
result Amortized point estimators generally outperform posterior inference, though posterior inference remains competitive in some low-dimensional problems.
Survival regression method improves log-likelihood scores.
problem Improper scoring rules in survival regression models.
method SurvivalMonotonic-net (SuMo-net) with monotonic neural networks.
result SuMo-net achieves state-of-the-art log-likelihood scores.
Study compares different scoring rules for machine-learned weather forecasts, finding scale-awareness improves forecast realism.
problem Improving the accuracy of machine-learned probabilistic weather forecasts.
method Comparison of scoring rules (CRPS, fair global energy score, graph energy score) and analysis of their impact on forecast field spectra.
result Scale-awareness improves forecast realism, particularly in the tropics.
New concept of proper-calibeating extends classic calibrated forecasts to proper scoring rules.
problem Defining and extending calibrated forecasts to proper scoring rules.
method Extending the concepts of calibrated and calibeating forecasts to proper scoring rules and proving their properties.
result Proper-calibration always implies calibration, but proper-calibeating does not necessarily imply calibeating.
Critiques binary classification evaluation methods, advocating for proper scoring rules.
problem The dominance of top-K metrics and fixed-threshold evaluations in machine learning.
method Introduces a decision-theoretic framework mapping evaluation metrics to their use cases, and implements a clipped Brier score variant.
result Demonstrates the clinical utility of proper scoring rules through a Python package, exttt{briertools}.
Proposes measures for uncertainty quantification using proper scoring rules.
problem Uncertainty quantification for prediction tasks.
method Decomposes proper scoring rules into divergence and entropy components, tailoring uncertainty quantification to specific tasks.
result Flexibility in uncertainty quantification improves performance in selective prediction and active learning.
Generative networks minimize predictive scoring rules for probabilistic forecasting.
problem Evaluating and improving probabilistic forecasts using generative models.
method Training generative networks to minimize predictive-sequential scoring rules on temporal sequences.
result Our method outperforms adversarial approaches in probabilistic calibration.
New findings show second-order scoring rules can't accurately represent epistemic uncertainty.
problem Lack of epistemic uncertainty representation in second-order learners.
method Generalised second-order scoring rules introduced to prove theoretical limitations.
result No loss function incentivizes second-order learners to accurately represent epistemic uncertainty.
Proposes a score to compare rule-based algorithms' interpretability.
problem Lack of consensus on interpretability for predictive models.
method Defines a score with three terms: predictivity, stability, and simplicity, each quantified by simple formulas.
result Compares interpretability of rule-based and tree-based algorithms for regression and classification.
We study strictly proper scoring rules in the Reproducing Kernel Hilbert Space. We propose a general Kernel Scoring rule and associated Kernel Divergence. We consider conditions under which the Kernel Score is strictly proper. We then demonstrate that the Kernel Score includes the Maximum Mean Discrepancy as a special …
New scoring rules compare probabilistic top lists in classification.
problem Evaluation of probabilistic top lists in classification.
method Elicitability through symmetric proper scoring rules.
result Brier score provides a well-suited metric for comparison.
We revisit the classical decision-theoretic problem of weighted expert voting from a statistical learning perspective. In particular, we examine the consistency (both asymptotic and finitary) of the optimal Nitzan-Paroush weighted majority and related rules. In the case of known expert competence levels, we give sharp …
Advances rule-based multi-label classification using conformal prediction.
problem Improving accuracy and decision making in multi-label classification.
method Combines conformal prediction with rule-based learning to provide natural conformity scores and calibrate rule assessments.
result Calibrated conformity scores enhance prediction accuracy and decision making.
Signature kernel scoring rule improves weather forecasting by capturing temporal and spatial dependencies.
problem Lack of suitable scoring rules for probabilistic weather forecasting.
method Reframe weather variables as continuous paths using iterated integrals (signature kernels) to capture temporal and spatial dependencies.
result Signature kernel scoring rule outperforms conventional methods in weather forecasting, especially for long-term forecasts.
New method uses GANs and proper scoring rules for robust scatter estimation.
problem Robust scatter estimation in statistics.
method General learning via classification framework based on proper scoring rules.
result Proposed robust scatter estimators achieve minimax rate under Huber's contamination model.
This paper explores how to choose scoring rules for estimating properties with parametric assumptions.
problem Indirect elicitation of properties with parametric assumptions.
method Developed a framework for choosing proper scoring rules for indirect elicitation, considering constraints and optimal solutions.
result The optimal estimation of the target property changes monotonically with the increase of each weight, and often setting some weights as zero yields the best configuration.
Study identifies regions where scoring rules reliably detect forecast errors.
problem Insufficient reliability of scoring rules in evaluating multivariate probabilistic forecasts.
method Systematic finite-sample analysis of proper scoring rules on synthetic and real-world data.
result Identified regions of reliability for scoring rules in time-series forecasting.
A new bias score method optimizes fairness in classification.
problem Ensuring fairness in binary classification under group constraints.
method Introducing bias scores and developing a post-hoc approach to adapt to fairness constraints.
result The method maintains high accuracy while ensuring fairness constraints.
There has been much recent interest in application of the pool-adjacent-violators (PAV) algorithm for the purpose of calibrating the probabilistic outputs of automatic pattern recognition and machine learning algorithms. Special cost functions, known as proper scoring rules form natural objective functions to judge the…
A new kernel-based nonconformity score improves multivariate prediction regions.
problem Tackling the challenge of compressing multivariate residual vectors into scalars while preserving geometric structure.
method Introducing a Multivariate Kernel Score (MKS) that decomposes into an anisotropic MMD, providing finite-sample coverage guarantees and convergence rates.
result The MKS produces prediction regions that explicitly adapt to geometric structure, reducing volume compared to ellipsoidal baselines.
We provide self-contained proof of a theorem relating probabilistic coherence of forecasts to their non-domination by rival forecasts with respect to any proper scoring rule. The theorem appears to be new but is closely related to results achieved by other investigators.
Study optimizes scoring rules for incentivizing agent's information gathering in online settings.
problem Optimizing incentives for agents to acquire information in online settings.
method Designing a sample-efficient algorithm that tailors the UCB algorithm to the strategic agent's model.
result Achieves sublinear T2/3-regret after T iterations, independent of the number of states. Fed-FEARE model extracts rules from multiple agencies' data securely.
problem Data privacy and secure rule extraction across multiple agencies.
method Federated F-score based ensemble tree model.
result Model performance measures significantly improved with federated learning.
Method finds optimal binary classification rules under weighted misclassification loss.
problem Optimal binary classification rules for resource-limited settings with cost-sensitive decisions.
method Ensemble learning to derive prediction scores and associated thresholds minimizing weighted misclassification loss.
result Jointly derived score and threshold outperforms methods that derive score first and threshold second.
Conditional forecasts improve performative prediction accuracy.
problem Performative predictions undermine standard forecasting methods.
method Condition forecasts on covariates to make them forecast-invariant.
result Proper scoring rules fail under conditioning, but two solutions are identified.
Generative models learn rules at different timescales, revealing a 'innovation window'.
problem Generative models' convergence to empirical training distribution rather than population distribution.
method Rule-valid synthetic tasks, analyzing τrule and τmem across training timescales. result The 'innovation window' widens with increasing dataset size and narrows with rule complexity.
Rule induction explains neural network predictions globally.
problem Understanding and explaining the behavior of trained models.
method Calculate feature importance, transform inputs, simplify space, fit rule induction model.
result Rule sets explain neural network predictions with 0.80 macro-averaged F-score.
ProSMIN improves representation quality through probabilistic self-supervised learning.
problem Improving representation quality in self-supervised learning.
method ProSMIN uses two neural networks, online and target, to learn diverse representations through knowledge distillation and a modified scoring rule loss function.
result ProSMIN achieves superior accuracy and calibration on various downstream tasks.
Extends denoising and score estimation to energy models via Tweedie's formula.
problem Linking denoising and score estimation for a wider range of distributions.
method Derives a fundamental identity connecting energy score derivatives and scores.
result Establishes a new identity for energy scores analogous to Tweedie's formula.
Improves generative models for cost-sensitive decisions.
problem Generative models lack awareness of decision costs.
method Integrates a decision loss into the training objective.
result Improves cost-sensitive forecast accuracy.
New tree splitting criteria improve probabilistic predictions.
problem Improving tree-based nonparametric predictive distributions.
method Using proper scoring rules for tree splitting criteria.
result Trees with new splitting criteria produce better predictive distributions.
This work proposes optimal decision rules for hierarchical classifiers to better align with evaluation metrics.
problem Heuristic decision rules in hierarchical classification do not align with evaluation metrics.
method Derives optimal decision rules for various prediction settings, focusing on hierarchical hFβ scores. result Optimal decision rules enhance the performance and reliability of hierarchical classifiers.
Model selection for time series forecasting can be biased by the distribution of scores.
problem Model selection for probabilistic forecasting on time series data.
method Using proper scoring rules to aggregate scores across multiple time series.
result The mean score is immune to the skewness of the score distribution.
Engression uses neural networks to learn conditional distributions, analyzing its error components.
problem Analyzing the error in neural-network-based engression.
method Engression decomposes excess risk into approximation, stochastic, and Monte Carlo errors.
result Established convergence rates under compositional smoothness assumptions.
This paper proposes a new method to approximate posterior distributions using generative neural networks trained via scoring rule minimization.
problem Bayesian Likelihood-Free Inference for models with intractable likelihood.
method Approximate posterior with generative neural networks trained via scoring rule minimization, avoiding the instability of adversarial training.
result Scoring Rule minimization leads to better performance and uncertainty quantification compared to adversarial training.
Robust Bayes-Assisted Conformal Prediction improves prediction set sizes.
problem Misspecification of Bayesian working model and prior misalignment.
method RoBAS (Robust Bayes-Assisted Shrinkage) framework with two instantiations.
result Proposed scores adapt to prior quality, reducing interval widths in shifted settings.