This paper argues against using calibration metrics for assessing posterior probabilities and proposes expected proper scoring rules instead.
problem The assessment of posterior probabilities generated by machine learning classifiers using calibration metrics is flawed and should be replaced with expected proper scoring rules.
method The paper reviews proper scoring rules from a practical perspective, explains why expected PSRs are a principled measure of posterior quality, and introduces a new calibration metric called calibration loss.
result Calibration loss is superior to expected calibration error and expected score divergence calibration metrics for assessing posterior probabilities.
New methods minimize GFlowNet training divergences for better sampling.
problem Training GFlowNets with KL divergence leads to biased and high-variance estimators.
method Design and implement efficient estimators for four divergence measures.
result Properly minimizing these divergences yields a provably correct and effective training scheme.
New divergences improve score-based methods for multi-modal distributions.
problem Blindness problem in score-based divergences for multi-modal distributions.
method Proposed a new family of divergences to mitigate blindness.
result Improved performance in density estimation compared to traditional approaches.
Study explores relationship between Hölder and FDPD divergences.
problem Understanding the relationship between Hölder and FDPD divergences.
method Intersection and generalization of divergence families, proving nonnegativity, deriving inequalities.
result Established ξ-Hölder divergence and derived inequalities. We study strictly proper scoring rules in the Reproducing Kernel Hilbert Space. We propose a general Kernel Scoring rule and associated Kernel Divergence. We consider conditions under which the Kernel Score is strictly proper. We then demonstrate that the Kernel Score includes the Maximum Mean Discrepancy as a special …
New optimal transport divergences derived from scoring functions.
problem Developing new divergences for optimal transport.
method Using scoring functions as cost functions in optimal transport.
result Comonotonic coupling is optimal for many new divergences.
Generative diffusion models are analyzed for their information dynamics.
problem Lack of a unified theoretical understanding of generative diffusion models.
method Integrated perspective connecting information-theoretic, dynamical, and thermodynamic aspects.
result Generative bandwidth is directly governed by the divergence of the score function's vector field.
Study on limits of LLM-based multi-agent planning reliability.
problem Reliability limits of LLM-based multi-agent planning.
method Modeling LLM-based multi-agent architecture as a decision network, showing dominance by centralized Bayes decision maker.
result Optimizing multi-agent directed acyclic graphs under communication budget is equivalent to choosing a constrained experiment.
SMT trains generative models by estimating mixture scores, outperforming existing methods.
problem Training one-step generative models efficiently and effectively.
method Score-of-Mixture Training (SMT) estimates the score of mixture distributions between real and fake samples.
result SMT/SMD outperform existing methods on CIFAR-10 and ImageNet 64x64 datasets.
Score matching fails to train VAEs robustly, revealing autoencoding loss insights.
problem Catastrophic failure of variational score matching on VAE models.
method Analysis of existing variational score matching objectives and their equivalence to autoencoding losses.
result Score matching methods fail to produce robust VAE models, predicting poor performance.
New bounds close the score matching gap for diffusion models.
problem The difference between sample quality and score matching loss in diffusion models.
method Theoretical analysis of score matching gap, developing tighter bounds for KL divergence, reverse KL divergence, and Wasserstein distance.
result The quality of score approximation impacts closing the score matching gap for low noise scales.
BaM improves BBVI by optimizing a score-based divergence, leading to faster convergence.
problem Slow convergence of black-box variational inference methods.
method Batch and match (BaM) approach based on a score-based divergence.
result BaM converges exponentially quickly to the target mean and covariance.
Improved hypothesis testing and change-point detection using diffusion-based methods.
problem Limited power of score-based hypothesis tests and change-point detection.
method Extending score-based Fisher divergence to diffusion-divergence by multiplying score functions with a matrix-valued function or weight matrix.
result Theoretical quantification and demonstration of optimal performance of diffusion-based algorithms.
Paper presents variational estimates for EBLVMs without structural assumptions.
problem Challenges in learning and evaluating EBLVMs due to intractable true posteriors and partition functions.
method Variational estimates of the score function and its gradient (VaES and VaGES) in a general EBLVM.
result The estimates can be applied to KSD and SM-based methods to learn EBLVMs and estimate Fisher divergence.
Estimates proper calibration errors and refinement terms in probabilistic predictions.
problem Lack of a general estimator for proper calibration errors and refinement terms with known statistical properties.
method Proposes a method for consistent, asymptotically unbiased estimation of proper calibration errors and refinement terms.
result Proves the relation between refinement and f-divergences, implying information monotonicity in neural networks.
M-FISHER detects and adapts to streaming data shifts with statistical validity and stability.
problem Detecting and adapting to distributional shifts in streaming data.
method Constructs an exponential martingale from non-conformity scores and applies Ville's inequality for detection. Fisher-preconditioned updates for adaptation.
result Establishes M-FISHER as a principled approach for robust, anytime-valid detection and geometrically stable adaptation.
Researchers establish bounds for SGMs' KL and Wasserstein divergences under various noise schedules.
problem Estimating the error between target and estimated distributions in SGMs.
method Established upper bounds for KL divergence and Wasserstein distance, incorporating target distribution properties and SGM hyperparameters.
result Optimal noise schedules identified for SGMs, improving generative quality.
Study optimizes tree-based models for better alignment of predicted scores and actual probabilities.
problem Traditional calibration metrics fail to align predicted scores with actual probabilities when score distributions deviate from the underlying data.
method Optimizes tree-based models (Random Forest, XGBoost) using Kullback-Leibler (KL) divergence to minimize the difference between predicted and true probability distributions.
result Optimized tree-based models yield superior alignment between predicted scores and actual probabilities without significant performance loss.
New geometric analysis shows L2 score error is flawed for diffusion models.
problem Score matching errors in diffusion models do not fully capture distributional quality.
method Decomposed score errors into gradient and solenoidal components, focusing on gradient's role in Fokker-Planck dynamics.
result Only gradient component affects marginal distributional quality; solenoidal component is structurally invisible.
A new machine learning model uses score matching to estimate probability densities efficiently.
problem Estimating probability density functions is challenging.
method Introduced a product Jacobi-Theta Boltzmann machine (pJTBM) and used score matching for efficient fitting.
result The pJTBM can fit probability densities more efficiently than the RTBM using score matching.
We compute the expected value of the Kullback-Leibler divergence to various fundamental statistical models with respect to canonical priors on the probability simplex. We obtain closed formulas for the expected model approximation errors, depending on the dimension of the models and the cardinalities of their sample sp…
Score matching errors are not sufficient for measuring diffusion model quality.
problem The L2 score matching error is not a reliable measure of diffusion model performance. method Decomposed score errors into gradient and solenoidal components and analyzed their geometric properties.
result Only the gradient component of the score error affects the marginal distributional quality.
A new, computationally friendly formula for a class of risk-averse preferences.
problem Characterizing a class of risk-averse preferences called uniformly weighted divergence preferences.
method Introducing a new formula that characterizes UWDP as the translation-invariant hull of state-independent expected utility.
result UWDP are the translation-invariant hull of state-independent expected utility over L0. This paper analyzes discrete diffusion models, deriving convergence bounds for their generated samples.
problem Theoretical guarantees for discrete-state diffusion models remain under-explored.
method Continuous Time Markov Chain (CTMC) framework and discrete-time sampling algorithm.
result Convergence bounds for KL divergence and TV distance are derived, showing linear dependence on dimension.
The paper analyzes how synthetic data training degrades diffusion models, providing bounds and characterizing different drift regimes.
problem The degradation of performance in diffusion models trained on synthetic data.
method Theoretical analysis of score-based diffusion models, focusing on the accumulated divergence between generated and target distributions.
result Upper and lower bounds on the accumulated divergence, providing the first lower bound for diffusion models.
We propose to interpret distribution model risk as sensitivity of expected loss to changes in the risk factor distribution, and to measure the distribution model risk of a portfolio by the maximum expected loss over a set of plausible distributions defined in terms of some divergence from an estimated distribution. The…
In this paper, we derive a useful lower bound for the Kullback-Leibler divergence (KL-divergence) based on the Hammersley-Chapman-Robbins bound (HCRB). The HCRB states that the variance of an estimator is bounded from below by the Chi-square divergence and the expectation value of the estimator. By using the relation b…
AR-CSM models use derivatives of univariate log-conditionals to estimate joint distributions efficiently.
problem Scalability and stability issues in training autoregressive models.
method Parameterize joint distribution using derivatives of univariate log-conditionals and introduce Composite Score Matching (CSM) for efficient training.
result AR-CSM models are more scalable and stable compared to previous score matching algorithms.
DSM on manifolds removes singularities and computes small-noise expansions.
problem DSM on manifolds with singular noise.
method Rao-Blackwellized score matching, nearest-point projection, intrinsic Riemannian score.
result Canonical target equals intrinsic Riemannian score up to a small correction.
Unified framework for robust, stable, and efficient density ratio estimation.
problem Density-chasm and support-chasm problems in density ratio estimation.
method Dequantified diffusion-Schrödinger bridge (D3RE) framework with DDBI and DSBI.
result Offers uniform approximation and bounded time scores in theory and empirical performance.
We extend the recently introduced theory of Lovasz-Bregman (LB) divergences (Iyer & Bilmes 2012) in several ways. We show that they represent a distortion between a "score" and an "ordering", thus providing a new view of rank aggregation and order based clustering with interesting connections to web ranking. We show ho…
We extend the recently introduced theory of Lovasz-Bregman (LB) divergences (Iyer & Bilmes, 2012) in several ways. We show that they represent a distortion between a 'score' and an 'ordering', thus providing a new view of rank aggregation and order based clustering with interesting connections to web ranking. We show h…
In statistical analysis, measuring a score of predictive performance is an important task. In many scientific fields, appropriate scores were tailored to tackle the problems at hand. A proper score is a popular tool to obtain statistically consistent forecasts. Furthermore, a mathematical characterization of the proper…
Improved inference-time alignment using Best-of-N and smoothing.
problem Reward overoptimization in Best-of-N (BoN) due to poor proxy reward models.
method Introduced Soft Best-of-N (SBoN) and analyzed its performance through KL divergence and regret analysis.
result Smoothing helps SBoN mitigate reward overoptimization, especially when proxy reward quality is low.
New method samples from posterior distributions efficiently.
problem Posterior sampling in score-based models is intractable.
method Annealed Langevin Monte Carlo with KL and Fisher divergence.
result Tractable sampling from distributions close to posterior and prior.
Optimal payoff choice constrained by Bregman-Wasserstein divergence.
problem Maximizing utility under a deviation constraint from a benchmark.
method Solving the problem using Bregman-Wasserstein divergence with a convex function φ.
result Provided the optimal payoff choice in this setting.
VT-DIS improves sampling from Boltzmann distributions with minimal overhead.
problem Bias in Monte Carlo estimates from score-based diffusion models.
method Variance-Tuned Diffusion Importance Sampling (VT-DIS) adapts noise covariance to correct bias.
result VT-DIS achieves effective sample sizes of 80%, 35%, and 3.5% on benchmarks, using less computational budget.
We unify f-divergences, Bregman divergences, surrogate loss bounds (regret bounds), proper scoring rules, matching losses, cost curves, ROC-curves and information. We do this by systematically studying integral and variational representations of these objects and in so doing identify their primitives which all are rela…
New findings show score matching's accuracy doesn't ensure numerical stability in diffusion sampling.
problem Numerical stability issues in diffusion sampling despite small forward-marginal error.
method Constructing a smooth score field with arbitrarily small forward-marginal L2 error, showing nonexplosive behavior and moments of every order. result Euler--Maruyama discretizations can converge in probability even when moments diverge, demonstrating failure of weak convergence.
Proximal Diffusion Models improve generative model efficiency.
problem Improving generative model efficiency and accuracy.
method Developed Proximal Diffusion Models using proximal maps instead of scores.
result Proximal Diffusion Models achieve faster convergence and higher accuracy.
New analysis of annealing paths in sampling and estimation.
problem Sampling from complex distributions and estimating normalization constants.
method Extending known results on Bregman divergence to quasi-arithmetic means under monotonic embedding.
result Analogous result for quasi-arithmetic means, highlighting the interplay between means, parametric families, and divergence functionals.
This paper generalizes beta divergence beyond its classical form associated with power variance functions of Tweedie models. Generalized form is represented by a compact definite integral as a function of variance function of the exponential dispersion model. This compact integral form simplifies derivations of many pr…
Bi-Lipschitz flows approximate a wide range of distributions.
problem Characterizing the expressivity of bi-Lipschitz normalizing flows.
method Linking score regularity to transport map bi-Lipschitzness via probability flow ODE.
result Gaussian pullbacks induced by bi-Lipschitz variance-preserving transport maps are L1-dense among all probability densities. Generative models need per-sample confidence scores to improve quality and stability.
problem Generative models produce unreliable outputs and lack confidence measures.
method Flow Matching with Confidence (FMwC) injects noise and integrates it through the network, providing per-sample confidence scores.
result The confidence score correlates with the velocity field's divergence, offering insights into generative processes.
New CUSUM algorithm detects changes in unnormalized models.
problem Change detection in models with unknown normalization constants.
method Score-based CUSUM (SCUSUM) algorithm based on Fisher divergence and Hyvärinen score.
result Asymptotic optimality of the SCUSUM algorithm demonstrated.
Proposes a new neural head for asymmetric representation learning.
problem Asymmetric representation learning in directed relations.
method Role-aware neural convex divergence head.
result Role-aware projections improve directional accuracy over plain ICNN-Bregman heads.
Robustifies elicitable functionals to handle small distribution misspecifications.
problem Determining uniquely optimal forecasts under distributional misspecification.
method Integrates statistical robustness into elicitable functionals using Kullback-Leibler divergence.
result Robust elicitable functionals admit unique solutions at the boundary of uncertainty regions.
Paper presents ERM with f-divergence regularization and its properties.
problem Minimizing empirical risk with f-divergence constraints. method Introduces normalization function and solves ERM-fDR via ODE. result Characterizes difference between empirical risks and provides numerical algorithm.