Improved bounds for estimating discrete distributions in KL divergence.
problem Estimating discrete distributions in KL divergence with accuracy.
method Used Laplace estimator and established concentration bounds.
result Deviation from mean scales as k / n \sqrt{k}/n k / n for n ≥ k n \ge k n ≥ k . Paper proposes a method to stabilize estimation of KL divergence using a discriminator in RKHS.
problem High variance and instability in estimating KL divergence using neural network discriminators.
method Developed a novel construction of the discriminator in RKHS, controlled its complexity, and proved the consistency of the estimator.
result Reduced variance and stabilized training of KL divergence estimates.
New method improves text classification using KL divergence.
problem Multinomial text classification problem
method Centroid estimation based on symmetric KL divergence
result Substantial improvements over traditional classifiers
Unified view of KL-divergence and IPMs via DRE, with new DRM metrics.
problem Unified understanding of KL-divergence and IPMs.
method Unified representation via maximum likelihood density-ratio estimation (DRE).
result Unified form of IPMs and novel DRM metrics.
Paper shows how KL exponent is preserved via inf-projection for optimization problems.
problem Estimating the KL exponent for optimization problems.
method Shows KL exponent preservation via inf-projection for optimization problems.
result KL exponent is preserved via inf-projection for several important convex optimization models.
KL nearest neighbor estimator achieves near-minimax rates for differential entropy.
problem Estimating differential entropy with Hölder smoothness.
method Uniform upper and lower bounds on KL estimator performance.
result KL estimator is near-minimax rate-optimal without knowing smoothness.
Estimates KL divergence with fairness considerations for sub-populations.
problem Fairly estimate KL divergence between distributions considering sub-populations.
method Proposes multi-group attribution for KL divergence estimation, derived from multi-calibration.
result Shows multi-group attribution provides better KL divergence estimates conditioned on sub-populations.
Study introduces a variational approach for efficient KL divergence estimation in Dirichlet mixture models.
problem Efficient estimation of KL divergence in Dirichlet mixture models.
method Variational approach for a closed-form solution.
result Superior efficiency and accuracy compared to Monte Carlo methods.
Private KL distribution estimation improved with instance-optimality.
problem Minimizing KL divergence between true and estimated distributions.
method Construct minimax optimal private estimators, then focus on instance-optimality.
result Achieved instance-optimality up to constant factors for KL estimation.
Paper analyzes kNN estimator for KL divergence, proving its optimality.
problem Estimating KL divergence from identical samples.
method kNN estimator based on nearest neighbor distances.
result kNN method is asymptotically rate optimal for KL divergence estimation.
DPEs use KL divergence to approximate BNNs, improving uncertainty estimates for active learning.
problem Improving uncertainty estimates in active learning for visual classification.
method Regularized ensemble approach with KL divergence penalty for variational inference.
result DPEs steadily improve active learning performance with increased annotation budgets.
Study improves density estimation for compact domains using h h h -lifted KL divergence.
problem Estimating probability density functions on compact domains.
method Introduced h h h -lifted Kullback--Leibler (KL) divergence for risk minimization. result Proved O ( 1 / n ) \mathcal{O}(1/{\sqrt{n}}) O ( 1/ n ) bound on estimation error. New proof of Minkowski spacetime stability in exterior regions.
problem Stability of Minkowski spacetime in exterior regions.
method Unified treatment of decay of initial data, use of r p r^p r p -weighted estimates. result Reduced number of derivatives and simplified last slice treatment.
We establish bounds on the KL divergence between two multivariate Gaussian distributions in terms of the Hamming distance between the edge sets of the corresponding graphical models. We show that the KL divergence is bounded below by a constant when the graphs differ by at least one edge; this is essentially the tighte…
Paper analyzes inclusive KL inference using Wasserstein gradient flows.
problem Analyzing inclusive KL inference with mathematical tools.
method Gradient flows derived from PDE analysis.
result Unified view of existing sampling algorithms as inclusive-KL inference.
Theory for RLHF generalization under reward shift and clipped KL.
problem Theoretical understanding of RLHF generalization, especially with reward shift and clipped KL.
method Developed generalization theory for RLHF, accounting for reward shift and clipped KL.
result Presented generalization bounds for RLHF, suggesting generalization error from sampling, reward shift, and KL clipping.
The paper addresses instability in KL divergence estimation using a neural network discriminator.
problem Unstable estimation of KL divergence due to discriminator complexity.
method Using a Reproducing Kernel Hilbert Space (RKHS) to control discriminator complexity.
result Theoretical bound on error probability of KL estimates based on discriminator complexity in RKHS.
Improved KL divergence estimators for normalizing flows lead to faster convergence and better approximations.
problem Estimating KL divergences for normalizing flows efficiently and accurately.
method Path-gradient estimators for reverse and forward KL divergences.
result Path-gradient estimators lead to faster convergence and better approximation results.
Flow matching KL divergence bound derived for smooth distributions.
problem Estimating smooth distributions efficiently.
method Deterministic upper bound on KL divergence derived from flow-matching loss.
result Flow matching achieves nearly minimax-optimal efficiency under TV distance.
New estimator improves reliability of KL divergence estimation.
problem Estimating KL divergence reliably and efficiently.
method Proposes a new estimator using Reproducing Kernel Hilbert Space.
result Proposed estimator is consistent and more reliable for small datasets.
New algorithm minimizes inclusive KL for VI, improving accuracy.
problem Improving variational inference accuracy with KL(p||q).
method Markovian score climbing (MSC) using stochastic gradients.
result MSC converges to local optimum of inclusive KL without bias.
New method improves variational inference for better posterior approximation.
problem Challenges in minimizing inclusive KL divergence for amortized variational inference.
method Likelihood-tempered sequential Monte Carlo samplers to estimate inclusive KL gradient.
result SMC-Wake method fits variational distributions more accurately than existing methods.
Paper derives a new lower bound for KL-divergence using HCRB.
problem Estimating KL-divergence between distributions.
method Using Hammersley-Chapman-Robbins bound and information geometry.
result New lower bound for KL-divergence derived from HCRB.
Extends global stability of Minkowski spacetime to minimal decay assumptions.
problem Global stability of Minkowski spacetime under minimal decay assumptions.
method Uses r p r^p r p -weighted estimates instead of vectorfield method. result Proves global stability of Minkowski spacetime under minimal decay assumptions.
The paper develops new algorithms for KL-divergence NMF, proving convergence and performance.
problem Improving NMF for nonnegative data with KL divergence.
method Collect and analyze properties of KL objective function, propose and test new algorithms.
result Guaranteed non-increasing objective function for one proposed algorithm, global convergence.
Estimating entropy and mutual information consistently is important for many machine learning applications. The Kozachenko-Leonenko (KL) estimator (Kozachenko & Leonenko, 1987) is a widely used nonparametric estimator for the entropy of multivariate continuous random variables, as well as the basis of the mutual inform…
Paper analyzes KSG mutual information estimator for smooth distributions.
problem Analyzing the convergence rate of KSG estimator for smooth distributions.
method Adaptive recombination of KL entropy estimators analysis.
result Convergence rate of KSG estimator for smooth distributions is analyzed.
DAIS minimizes symmetrized KL divergence between initial and target distributions.
problem Optimizing over initial distributions in importance sampling.
method Differentiable annealed importance sampling (DAIS) minimizing symmetrized KL divergence.
result DAIS minimizes symmetrized KL divergence between initial and target distributions.
The Dirichlet mechanism protects privacy while minimizing KL divergence.
problem Minimizing KL divergence while protecting sensitive data privacy.
method Using the exponential mechanism with the KL divergence loss function, resulting in the Dirichlet mechanism.
result Proved a probability tail bound on KL divergence and derived a lower bound for sample complexity.
KL regularization helps RL algorithms by implicitly averaging q-values.
problem Understanding why KL regularization improves RL performance.
method An approximate value iteration scheme, studying KL and entropy regularization.
result Strong performance bound combining linear horizon dependency and averaging effect of estimation errors.
Paper analyzes and improves KL-regularized RL for LLMs with logarithmic regret.
problem Improving efficiency of RL fine-tuning for large language models.
method Optimism-based KL-regularized online contextual bandit algorithm with novel regret analysis.
result Achieves an O ( η log ( N R T ) ⋅ d R ) \mathcal{O}\big(η\log (N_{\mathcal R} T)\cdot d_{\mathcal R}\big) O ( η log ( N R T ) ⋅ d R ) logarithmic regret bound. Study investigates estimation error in EMHMM simulations.
problem Estimation error in Hidden Markov Models (HMMs) with EMHMM.
method Simulation study using variational Bayesian inference.
result KL divergence and L1-norm relate to estimation error and ground-truth HMM parameters.
Estimates Markov chains from samples, solving two related prediction and estimation problems.
problem Estimating an unknown Markov chain from its samples.
method Considered two problems: predicting conditional distribution and estimating transition matrix, using KL-divergence and various f f f -divergences. result Resolved estimation problem for all sufficiently smooth f f f -divergences, including KL-, L 2 L_2 L 2 , Chi-squared, Hellinger, and Alpha-divergences. KALE flow approximates KL divergence for distributions with disjoint support.
problem Approximating KL divergence for distributions with disjoint support.
method Relaxed KL gradient flow using RKHS, continuously interpolating between KL and MMD.
result Global convergence of KALE flow under sufficient smoothness assumptions.
We consider model-based reinforcement learning in finite Markov De- cision Processes (MDPs), focussing on so-called optimistic strategies. In MDPs, optimism can be implemented by carrying out extended value it- erations under a constraint of consistency with the estimated model tran- sition probabilities. The UCRL2 alg…
Paper bridges VAEs and KDEs for more flexible posterior estimation.
problem Limitations of Gaussian latent space in VAEs and challenges in KL-divergence estimation.
method Approximate posterior with KDEs and derive upper bound of KL-divergence in ELBO.
result Epanechnikov kernel minimizes KL-divergence upper bound asymptotically.
Improved VAE with optimal but intractable prior using density ratio trick.
problem Over-regularization with standard Gaussian prior in VAE.
method Introduced density ratio trick to estimate KL divergence without modeling aggregated posterior explicitly.
result VAE achieves high density estimation performance with implicit optimal prior.
New α \alpha α -divergence loss function improves neural density ratio estimation.
problem Optimization challenges in existing DRE methods, especially overfitting and high sample requirements.
method Derived α \alpha α -divergence loss function ( α \alpha α -Div) for neural density ratio estimation. result The α \alpha α -divergence loss function ( α \alpha α -Div) offers stable and effective optimization for DRE. Researchers establish bounds for SGMs' KL and Wasserstein divergences under various noise schedules.
problem Estimating the error between target and estimated distributions in SGMs.
method Established upper bounds for KL divergence and Wasserstein distance, incorporating target distribution properties and SGM hyperparameters.
result Optimal noise schedules identified for SGMs, improving generative quality.
MAE tackles KL Varnishing in VAEs by controlling latent space geometry.
problem KL Varnishing in VAEs with expressive decoders.
method Mutual posterior-divergence regularization to control latent space geometry.
result MAE achieves comparable or superior density estimation and meaningful representation learning.
New method learns disentangled signals without prior or model constraints.
problem Learning disentangled signals from data without prior or model constraints.
method Minimizes conditional KL divergence using a sequential algorithm to learn de-mixing flow models.
result Method learns self-sufficient signals that can reconstruct missing values.
FORE evaluates occupancy ratios without requiring Bellman completeness.
problem Offline reinforcement learning occupancy ratio estimation.
method Fitted occupancy-ratio evaluation (FORE) using adjoint Bellman recursion.
result FORE achieves convergence in KL without Bellman completeness.
fBNNs use stochastic processes for variational inference in neural networks.
problem Difficulties in specifying priors and posteriors in high-dimensional weight spaces.
method Maximize Evidence Lower Bound (ELBO) on stochastic processes, using spectral Stein gradient estimator.
result fBNNs provide reliable uncertainty estimates and extrapolate well with structured priors.
Improved KL convergence bounds for score diffusion models without restrictive assumptions.
problem Lack of comprehensive quantitative results for diffusion models, especially in non-regular scores and estimators.
method Score diffusion models with fixed step size from Ornstein-Uhlenbeck and kinetic semigroups, providing explicit and sharp KL convergence bounds.
result Explicit and sharp convergence bounds in KL applicable to any data distribution with finite Fisher information.
New proof of Kerr stability outside null cones.
problem Stability of Kerr spacetime in external regions.
method Unified treatment of initial data and r p r^p r p -weighted estimates. result Reduced number of derivatives and simplified last slice treatment.
Paper improves PAC-Bayes bounds using a better-than-KL divergence.
problem Estimating the generalization error of stochastic algorithms.
method Developed new PAC-Bayes bounds with a novel divergence.
result Achieved strictly tighter bounds than the KL divergence.
Estimation of density derivatives is a versatile tool in statistical data analysis. A naive approach is to first estimate the density and then compute its derivative. However, such a two-step approach does not work well because a good density estimator does not necessarily mean a good density-derivative estimator. In t…
New ONMF model minimizes KL divergence for better sparse data modeling.
problem Clustering and data modeling with sparse vectors.
method Developed KL-ONMF algorithm based on alternating optimization.
result KL-ONMF outperforms Frobenius-norm ONMF for document classification and hyperspectral image unmixing.