Improved bounds for estimating discrete distributions in KL divergence.
problem Estimating discrete distributions in KL divergence with accuracy.
method Used Laplace estimator and established concentration bounds.
result Deviation from mean scales as k / n \sqrt{k}/n k / n for n ≥ k n \ge k n ≥ k . New KL-divergence for Gaussian distributions based on Wasserstein geometry.
problem Computing KL-divergence for Gaussian distributions efficiently.
method Introducing WKL-divergence based on Wasserstein geometry.
result WKL-divergence evaluates to squared distance between points for Dirac measures.
Unified view of KL-divergence and IPMs via DRE, with new DRM metrics.
problem Unified understanding of KL-divergence and IPMs.
method Unified representation via maximum likelihood density-ratio estimation (DRE).
result Unified form of IPMs and novel DRM metrics.
The study tightens bounds on binomial probabilities and minimums using KL-divergence.
problem Tightening bounds on binomial probabilities and minimums of i.i.d. Binomials.
method Applied Sanov's theorem to derive upper and lower bounds on binomial tail probabilities and minimums, expressed in terms of KL-divergence.
result High probability upper and lower bounds on the minimum of i.i.d. Binomial random variables, finite sample, asymptotically tight.
Study introduces a variational approach for efficient KL divergence estimation in Dirichlet mixture models.
problem Efficient estimation of KL divergence in Dirichlet mixture models.
method Variational approach for a closed-form solution.
result Superior efficiency and accuracy compared to Monte Carlo methods.
Estimates KL divergence with fairness considerations for sub-populations.
problem Fairly estimate KL divergence between distributions considering sub-populations.
method Proposes multi-group attribution for KL divergence estimation, derived from multi-calibration.
result Shows multi-group attribution provides better KL divergence estimates conditioned on sub-populations.
Flow matching KL divergence bound derived for smooth distributions.
problem Estimating smooth distributions efficiently.
method Deterministic upper bound on KL divergence derived from flow-matching loss.
result Flow matching achieves nearly minimax-optimal efficiency under TV distance.
Forward-Euler fails for simulating Wasserstein gradient flows with KL divergence.
problem Simulating Wasserstein gradient flows with forward-Euler discretization fails for KL divergence.
method Forward-Euler discretization for Wasserstein gradient flows with KL divergence.
result Forward-Euler discretization can be incorrect for Wasserstein gradient flows with KL divergence.
Optimized α \alpha α -posteriors reduce KL divergence from true posterior in parametric misspecification.
problem Reduction of KL divergence from true posterior in parametric model misspecification.
method Derivation of Bernstein-von Mises theorem and optimization of α \alpha α -posteriors. result Optimized α \alpha α -posteriors minimize KL divergence from true posterior, especially in severe misspecification. Study improves density estimation for compact domains using h h h -lifted KL divergence.
problem Estimating probability density functions on compact domains.
method Introduced h h h -lifted Kullback--Leibler (KL) divergence for risk minimization. result Proved O ( 1 / n ) \mathcal{O}(1/{\sqrt{n}}) O ( 1/ n ) bound on estimation error. t-SNE algorithm's points remain bounded under gradient flow.
problem Understanding the boundedness of t-SNE points.
method Gradient flow of t-SNE with KL divergence, examining weak convergence assumptions.
result Points generated by t-SNE remain bounded under gradient flow.
Personalized sleep staging achieved with single-night data using KL-divergence regularization.
problem Improving automatic sleep staging accuracy with limited single-night data.
method KL-divergence regularization for transfer learning from a pretrained model to a personalized model.
result Personalized sleep staging accuracy of 79.6% with KL-divergence regularization.
The Dirichlet mechanism protects privacy while minimizing KL divergence.
problem Minimizing KL divergence while protecting sensitive data privacy.
method Using the exponential mechanism with the KL divergence loss function, resulting in the Dirichlet mechanism.
result Proved a probability tail bound on KL divergence and derived a lower bound for sample complexity.
New method learns disentangled signals without prior or model constraints.
problem Learning disentangled signals from data without prior or model constraints.
method Minimizes conditional KL divergence using a sequential algorithm to learn de-mixing flow models.
result Method learns self-sufficient signals that can reconstruct missing values.
Paper analyzes kNN estimator for KL divergence, proving its optimality.
problem Estimating KL divergence from identical samples.
method kNN estimator based on nearest neighbor distances.
result kNN method is asymptotically rate optimal for KL divergence estimation.
New method proves dimension-free convergence for ULD in KL divergence.
problem Polynomial scaling of existing convergence guarantees in high dimensions.
method Refined KL local error framework, focusing on tr(H) instead of d.
result First dimension-free KL divergence bounds for discretized ULD.
A1GM method improves efficiency in reconstructing missing data using KL divergence.
problem Efficiently reconstructing missing data in matrices.
method Fast non-gradient-based rank-1 NMF using KL divergence.
result A1GM outperforms gradient methods in efficiency with competitive reconstruction errors.
Study compares chi-squared divergence and KL-divergence posteriors for PAC-Bayesian bounds.
problem Investigates optimal posteriors for PAC-Bayesian bounds using chi-squared divergence.
method Analyzes bounds for three distance functions, derives FP equations for computation.
result Chi-squared divergence based posteriors have weaker bounds and worse test errors.
DAIS minimizes symmetrized KL divergence between initial and target distributions.
problem Optimizing over initial distributions in importance sampling.
method Differentiable annealed importance sampling (DAIS) minimizing symmetrized KL divergence.
result DAIS minimizes symmetrized KL divergence between initial and target distributions.
SRFE clarifies KL divergences without unifying learning frameworks.
problem Inductive biases of KL divergences and their limitations.
method Introducing SRFE, a log-moment-based functional of the likelihood ratio.
result SRFE recovers KL divergences as limits and reveals a mean-variance tradeoff.
Paper proposes a method to stabilize estimation of KL divergence using a discriminator in RKHS.
problem High variance and instability in estimating KL divergence using neural network discriminators.
method Developed a novel construction of the discriminator in RKHS, controlled its complexity, and proved the consistency of the estimator.
result Reduced variance and stabilized training of KL divergence estimates.
New method trains neural samplers to sample from multi-modal distributions efficiently.
problem Mode-seeking behavior of reverse KL divergence hinders effective sampling from multi-modal target distributions.
method Minimizing reverse diffusive KL divergence along diffusion trajectories of model and target densities.
result Demonstrated enhanced sampling performance across various multi-modal distributions.
Decentralized Bayesian learning reduces KL-divergence exponentially.
problem Efficiently learning posterior distributions in a decentralized setting.
method Decentralized Langevin dynamics in a non-convex setting.
result The algorithm converges to the target posterior distribution with exponential decrease in KL-divergence and polynomial decrease in error contributions.
EM algorithm converges in KL divergence for exponential families via mirror descent.
problem Lack of understanding of EM's non-asymptotic convergence properties.
method Viewing EM as a mirror descent algorithm, showing convergence rates in KL divergence.
result KL divergence rates for EM in exponential families, invariant to parametrization.
New samplers minimize KL divergence for constrained and non-Euclidean geometries.
problem Efficient sampling from constrained and non-Euclidean distributions.
method Stein Variational Mirror Descent and Mirrored Stein Variational Gradient Descent.
result New samplers converge more rapidly and accurately than prior methods.
We define a new method to estimate centroid for text classification based on the symmetric KL-divergence between the distribution of words in training documents and their class centroids. Experiments on several standard data sets indicate that the new method achieves substantial improvements over the traditional classi…
RHMC accelerates sampling from log-concave distributions.
problem Sampling from log-concave probability distributions efficiently.
method RHMC uses simulated Hamiltonian dynamics with random integration times.
result RHMC converges exponentially fast in KL divergence for log-concave distributions.
We speed up marginal inference by ignoring factors that do not significantly contribute to overall accuracy. In order to pick a suitable subset of factors to ignore, we propose three schemes: minimizing the number of model factors under a bound on the KL divergence between pruned and full models; minimizing the KL dive…
Learning word representations has garnered greater attention in the recent past due to its diverse text applications. Word embeddings encapsulate the syntactic and semantic regularities of sentences. Modelling word embedding as multi-sense gaussian mixture distributions, will additionally capture uncertainty and polyse…
KL-constrained API shows optimization issues and improved with regularization.
problem Optimization issues in KL-constrained API algorithms.
method Comparison of KL divergence as a constraint vs. regularizer, empirical evaluation.
result KL-constrained API is not guaranteed to converge and incurs linear regret.
Unified framework for unlearning in diffusion models using KL divergence and likelihood constraints.
problem Removing undesirable data or concepts while preserving utility of pretrained models.
method Constrained optimization framework based on reverse and forward KL divergences, and likelihood constraints.
result Our KL-constrained approach achieves superior retention-unlearning tradeoffs compared to weight-based baselines.
Improved KL divergence estimators for normalizing flows lead to faster convergence and better approximations.
problem Estimating KL divergences for normalizing flows efficiently and accurately.
method Path-gradient estimators for reverse and forward KL divergences.
result Path-gradient estimators lead to faster convergence and better approximation results.
Unified analysis of KL divergence using shifted composition for sampling.
problem Sampling from target distributions with KL divergence guarantees.
method Shifted composition rule applied to KL divergence, combining local error analysis and Girsanov's theorem.
result Unified KL guarantees for strongly log-concave, weakly log-concave, and log-Sobolev distributions.
We establish bounds on the KL divergence between two multivariate Gaussian distributions in terms of the Hamming distance between the edge sets of the corresponding graphical models. We show that the KL divergence is bounded below by a constant when the graphs differ by at least one edge; this is essentially the tighte…
Improved analysis for diffusion models reduces KL divergence error dependence on data dimension and discretization step size.
problem Analyze the convergence of diffusion-based generative models under minimal assumptions.
method Model the generation process as a composition of reverse ODE and noising steps, leveraging Wasserstein-type error control and noise addition.
result Achieved a linear dependence on data dimension and improved dependence on discretization step size for KL divergence error.
The paper explores how information geometry impacts classical CR inequalities.
problem Deriving and generalizing CR inequalities using information geometry.
method Examining Eguchi's theory and applying Amari-Nagoaka's theory to KL-divergence, and then extending to other divergences.
result Generalized CR inequalities derived from various divergences.
New method makes quality metrics scale-invariant for high-dimensional data.
problem Scale sensitivity in quality metrics affects the accuracy of data projections.
method Analytical and empirical investigation of stress and KL divergence; introduction of a scale-invariant technique.
result The proposed technique accurately captures expected behavior and makes metrics scale-invariant.
Paper bridges VAEs and KDEs for more flexible posterior estimation.
problem Limitations of Gaussian latent space in VAEs and challenges in KL-divergence estimation.
method Approximate posterior with KDEs and derive upper bound of KL-divergence in ELBO.
result Epanechnikov kernel minimizes KL-divergence upper bound asymptotically.
The variational autoencoder (VAE) is a powerful generative model that can estimate the probability of a data point by using latent variables. In the VAE, the posterior of the latent variable given the data point is regularized by the prior of the latent variable using Kullback Leibler (KL) divergence. Although the stan…
Causal KL improves on existing metrics for evaluating causal models.
problem Insufficient discrimination between causal models using edit-distance and KL divergence.
method Introducing Causal KL, an augmented KL divergence that considers causal relationships.
result Causal KL variants effectively distinguish between observationally equivalent models.
In this paper, we derive a useful lower bound for the Kullback-Leibler divergence (KL-divergence) based on the Hammersley-Chapman-Robbins bound (HCRB). The HCRB states that the variance of an estimator is bounded from below by the Chi-square divergence and the expectation value of the estimator. By using the relation b…
The paper addresses instability in KL divergence estimation using a neural network discriminator.
problem Unstable estimation of KL divergence due to discriminator complexity.
method Using a Reproducing Kernel Hilbert Space (RKHS) to control discriminator complexity.
result Theoretical bound on error probability of KL estimates based on discriminator complexity in RKHS.
Fuses posterior distributions from different datasets using KL divergence.
problem Combining information from multiple datasets with uncertainty.
method Mean field assumption, KL divergence, assign-and-average approach.
result Efficient non-parametric algorithm for fused model computation.
New ONMF model minimizes KL divergence for better sparse data modeling.
problem Clustering and data modeling with sparse vectors.
method Developed KL-ONMF algorithm based on alternating optimization.
result KL-ONMF outperforms Frobenius-norm ONMF for document classification and hyperspectral image unmixing.
New bounds close the score matching gap for diffusion models.
problem The difference between sample quality and score matching loss in diffusion models.
method Theoretical analysis of score matching gap, developing tighter bounds for KL divergence, reverse KL divergence, and Wasserstein distance.
result The quality of score approximation impacts closing the score matching gap for low noise scales.
Beta diffusion generates bounded data using multiplicative transitions.
problem Generating data within specific ranges.
method Integrates demasking and denoising with scaled and shifted beta distributions.
result KLUBs are more effective for optimizing beta diffusion compared to negative ELBOs.
New α \alpha α -divergence loss function improves neural density ratio estimation.
problem Optimization challenges in existing DRE methods, especially overfitting and high sample requirements.
method Derived α \alpha α -divergence loss function ( α \alpha α -Div) for neural density ratio estimation. result The α \alpha α -divergence loss function ( α \alpha α -Div) offers stable and effective optimization for DRE. While Generative Adversarial Networks (GANs) have empirically produced impressive results on learning complex real-world distributions, recent works have shown that they suffer from lack of diversity or mode collapse. The theoretical work of Arora et al. suggests a dilemma about GANs' statistical properties: powerful d…