Paper proposes f-EBM for training deep EBMs using various f-divergences.
problem Training deep EBMs with intractable partition functions.
method Introduces f-EBM framework and optimization algorithm for any f-divergence.
result f-EBM outperforms contrastive divergence and other f-divergences.
We describe the underlying probabilistic interpretation of alpha and beta divergences. We first show that beta divergences are inherently tied to Tweedie distributions, a particular type of exponential family, known as exponential dispersion models. Starting from the variance function of a Tweedie model, we outline how…
Study compares statistical properties and power of divergence measures for credit risk monitoring.
problem Detecting distributional shifts in credit risk models.
method Derives statistical properties and chi-square benchmark values for Jensen-Shannon Divergence and Kullback-Leibler Divergence, demonstrating their applicability in credit risk monitoring.
result Jensen-Shannon Divergence and Kullback-Leibler Divergence follow chi-square distributions and reveal practical trade-offs in minimizing false positives vs. detecting changes.
For distributions P and Q with different supports or undefined densities, the divergence D(P∣∣Q) may not exist. We define a Spread Divergence D~(P∣∣Q) on modified P and Q and describe sufficient conditions for t…
Replacing MSE with f-divergence in diffusion models improves robustness under data contamination.
problem Improving robustness of diffusion models under data contamination.
method Replacing MSE with f-divergence in diffusion models.
result Empirical improvement in performance under data contamination.
Paper proposes f-DPG for aligning language models with preferences.
problem Aligning language models with user preferences.
method Uses f-divergence to approximate target distributions and minimizes a forward KL from it using DPG.
result Jensen-Shannon divergence often outperforms forward KL divergence, leading to significant improvements.
A method to compute divergences between decomposable models, useful in supervised learning.
problem Computing exact divergences between high-dimensional distributions is intractable.
method Proposes an approach to compute exact alpha-beta divergences between marginal and conditional distributions of decomposable models.
result Tractable computation of marginal and conditional alpha-beta divergences.
This paper introduces f-DPO, a generalized approach to Direct Preference Optimization using diverse divergence constraints.
problem Aligning large language models with human preferences while mitigating safety risks.
method Incorporates diverse divergence constraints to simplify the relationship between reward and optimal policy, eliminating the need for estimating the normalizing constant.
result Optimizes LLMs to align with human preferences more efficiently and under a broader set of divergence constraints.
This paper proposes the divergence triangle as a framework for joint training of generator model, energy-based model and inference model. The divergence triangle is a compact and symmetric (anti-symmetric) objective function that seamlessly integrates variational learning, adversarial learning, wake-sleep algorithm, an…
New loss functions based on f-divergences improve language model performance.
problem Improving multiclass classification and language modeling performance.
method Constructing new convex loss functions using f-divergences and deriving an operator for computation.
result The α-divergence loss function with α=1.5 performs well across various tasks. This paper introduces a new method to train normalizing flows using precision-recall divergences.
problem Training generative models with mode dropping and low-quality samples.
method Introduces PR-divergences and proposes a novel generative model to minimize precision-recall trade-offs.
result Normalizing flows can be trained to achieve specific precision-recall trade-offs using PR-divergences.
Estimates KL divergence with fairness considerations for sub-populations.
problem Fairly estimate KL divergence between distributions considering sub-populations.
method Proposes multi-group attribution for KL divergence estimation, derived from multi-calibration.
result Shows multi-group attribution provides better KL divergence estimates conditioned on sub-populations.
Probabilistic models are often trained by maximum likelihood, which corresponds to minimizing a specific f-divergence between the model and data distribution. In light of recent successes in training Generative Adversarial Networks, alternative non-likelihood training criteria have been proposed. Whilst not necessarily…
Fisher auto-encoders use Fisher divergence for more robust generative modeling.
problem Model uncertainty in generative models.
method Minimizing Fisher divergence between true and modeled joint distributions.
result Fisher auto-encoders can more accurately quantify model uncertainty.
Parametric adversarial divergences, which are a generalization of the losses used to train generative adversarial networks (GANs), have often been described as being approximations of their nonparametric counterparts, such as the Jensen-Shannon divergence, which can be derived under the so-called optimal discriminator …
This paper generalizes beta divergence beyond its classical form associated with power variance functions of Tweedie models. Generalized form is represented by a compact definite integral as a function of variance function of the exponential dispersion model. This compact integral form simplifies derivations of many pr…
Develops deep NMF models using β-divergences for feature extraction.
problem Inadequate evaluation metrics for deep NMF on diverse datasets.
method Introduces new deep NMF models using Kullback-Leibler divergence.
result Improves feature extraction quality across different types of data.
LLM safety alignment explained as divergence estimation.
problem Aligning large language models to avoid harmful outputs.
method Presented a theoretical framework showing alignment methods as divergence estimators.
result KLDO method improves safety alignment using compliance-refusal datasets.
Paper introduces symmetric divergence link models for probability distributions.
problem Symmetric divergence measures for probability distributions.
method Two general classes of link models: one for survival functions and another for cumulative probability distribution functions.
result Advantages of symmetric divergence measures over asymmetric measures for model averaging and feature assessment.
In this paper, we introduce new classes of divergences by extending the definitions of the Bregman divergence and the skew Jensen divergence. These new divergence classes (g-Bregman divergence and skew g-Jensen divergence) satisfy some properties similar to the Bregman or skew Jensen divergence. We show these g-diverge…
We speed up marginal inference by ignoring factors that do not significantly contribute to overall accuracy. In order to pick a suitable subset of factors to ignore, we propose three schemes: minimizing the number of model factors under a bound on the KL divergence between pruned and full models; minimizing the KL dive…
Rank-statistic method approximates f-divergences without density-ratio estimation.
problem Approximating f-divergences without explicit density-ratio estimation. method Mapping distribution rank histograms to discrete f-divergence and averaging over random projections. result The rank-statistic estimator is a lower bound of the true f-divergence and converges under mild conditions. The paper explores statistical and topological properties of sliced probability divergences.
problem Understanding the topological, statistical, and computational consequences of slicing divergences.
method Deriving theoretical properties of sliced probability divergences, including metric axioms preservation and weak continuity.
result Sliced divergences share similar topological properties and have stable sample complexity.
The paper introduces a new risk assessment framework using φ-divergence.
problem Assessing risk and decision-making in uncertain conditions.
method Introduces a novel framework called the φ-Divergence Quadrangle.
result Provides a more nuanced understanding of risk through φ-divergence.
Generative adversarial networks (GANs) are a family of generative models that do not minimize a single training criterion. Unlike other generative models, the data distribution is learned via a game between a generator (the generative model) and a discriminator (a teacher providing training signal) that each minimize t…
Introduces a new divergence measure for optimal transport.
problem Optimal transport distances and information divergences.
method Infimal convolution formulation of proximal optimal transport divergence.
result Establishes connections to dynamic formulations and partial differential equations.
New tools quantify deep generative models' performance.
problem Measuring the quality-diversity trade-off in deep generative models.
method Established non-asymptotic bounds on sample complexity and introduced frontier integrals.
result Smoothed estimators improve convergence rates of divergence frontiers.
Study introduces a variational approach for efficient KL divergence estimation in Dirichlet mixture models.
problem Efficient estimation of KL divergence in Dirichlet mixture models.
method Variational approach for a closed-form solution.
result Superior efficiency and accuracy compared to Monte Carlo methods.
Divergence functions play a key role as to measure the discrepancy between two points in the field of machine learning, statistics and signal processing. Well-known divergences are the Bregman divergences, the Jensen divergences and the f-divergences. In this paper, we show that the symmetric Bregman divergence can be …
We propose to interpret distribution model risk as sensitivity of expected loss to changes in the risk factor distribution, and to measure the distribution model risk of a portfolio by the maximum expected loss over a set of plausible distributions defined in terms of some divergence from an estimated distribution. The…
DM framework improves robustness and efficiency in latent-mixture models.
problem Efficient and robust inference in latent-mixture models.
method Divergence-minimization framework with monotonic convergence and robustness guarantees.
result DM yields consistent and asymptotically normal estimators under correct specification.
New method minimizes robust density power-based divergences for general parametric densities.
problem Computational complexity of minimizing DPD for general parametric densities.
method Stochastic approach to minimize DPD for general parametric density models.
result Proposed method can be applied to minimize other density power-based γ-divergences.
The paper analyzes the statistical properties of GANs using f-divergence.
problem Understanding the statistical behavior of GANs and comparing different f-divergences. method Asymptotic analysis of f-divergence GANs, including Kullback-Leibler divergence. result Asymptotically equivalent GANs with the same discriminator classes for correctly specified models.
AlphaNet improves supernets training with alpha-divergence.
problem Improving the uncertainty distillation in weight-sharing NAS.
method Proposes alpha-divergence for better uncertainty distillation in supernets.
result Significant improvements in model performance across various FLOPs regimes.
The family of f-divergences is ubiquitously applied to generative modeling in order to adapt the distribution of the model to that of the data. Well-definedness of f-divergences, however, requires the distributions of the data and model to overlap completely in every time step of training. As a result, as soon as the s…
Study explores relationship between Hölder and FDPD divergences.
problem Understanding the relationship between Hölder and FDPD divergences.
method Intersection and generalization of divergence families, proving nonnegativity, deriving inequalities.
result Established ξ-Hölder divergence and derived inequalities. Paper explores how generative models can be made more creative.
problem Limitation of generative models in diverging from original data distribution.
method Proposes a novel training objective called Bounded Adversarial Divergence (BAD) to enable creative divergence.
result Preliminary results suggest BAD can enable creative divergence in generative models.
E2M optimizes tensor density estimation by relaxing α-divergence to KL-divergence.
problem Analytical challenges in traditional α-divergence optimization for tensor-based density estimation. method E2M algorithm: relaxes optimization to KL-divergence, then applies tensor many-body approximation. result Flexible modeling of various low-rank structures and their mixtures.
Unified representation of density-power-based divergences simplifies estimation to M-estimation.
problem Outliers in density estimation.
method Define a norm-based Bregman density power divergence (NB-DPD) that reduces to M-estimation.
result NB-DPD connects and generalizes existing divergences, highlighting robustness properties.
New model learns better policies from expert demonstrations with higher efficiency.
problem Learning accurate policies from expert demonstrations with high efficiency.
method Generative adversarial imitation learning (GAIL) model that learns f-divergence automatically. result Learns better policies with higher data efficiency in physics-based control tasks.
New method for bidirectional generative modeling using adversarial gradient estimation.
problem Bidirectional generative modeling with various f-divergences. method Adversarial gradient estimation for f-divergence optimization. result Similar algorithms for different f-divergences with varying scaling. We review recent results about the maximal values of the Kullback-Leibler information divergence from statistical models defined by neural networks, including naive Bayes models, restricted Boltzmann machines, deep belief networks, and various classes of exponential families. We illustrate approaches to compute the max…
Paper shows robust generative learning with minimal assumptions on target distributions.
problem Learning generative models with minimal assumptions on target distributions.
method Lipschitz-regularized α-divergences with minimal assumptions. result Stable learning across various target distributions with minimal assumptions.
New ONMF model minimizes KL divergence for better sparse data modeling.
problem Clustering and data modeling with sparse vectors.
method Developed KL-ONMF algorithm based on alternating optimization.
result KL-ONMF outperforms Frobenius-norm ONMF for document classification and hyperspectral image unmixing.
Rényi Neural Processes replace KL divergence with Rényi divergence to improve NP performance.
problem Parameterization coupling in Neural Processes leads to prior misspecification.
method Propose Rényi Neural Processes (RNP) by replacing KL divergence with Rényi divergence.
result Significant performance improvements in real-world problems, including better log-likelihoods.
This paper improves active learning by using robust divergences for committee disagreement.
problem Active learning with high measurement costs.
method Query by committee with Bregman divergence (including Kullback-Leibler divergence as a special case).
result The proposed method is more robust and performs as well as or better than conventional methods.
A new objective function using Jensen-Shannon divergence improves generative learning from multiple data types.
problem Learning from multiple data types efficiently and accurately.
method Proposes a novel objective function using Jensen-Shannon divergence to approximate multimodal posteriors directly.
result The mmJSD objective optimizes an ELBO and improves generative learning tasks.
Unified view of KL-divergence and IPMs via DRE, with new DRM metrics.
problem Unified understanding of KL-divergence and IPMs.
method Unified representation via maximum likelihood density-ratio estimation (DRE).
result Unified form of IPMs and novel DRM metrics.