Paper derives a new lower bound for KL-divergence using HCRB.
problem Estimating KL-divergence between distributions.
method Using Hammersley-Chapman-Robbins bound and information geometry.
result New lower bound for KL-divergence derived from HCRB.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,878 papers · 148 categories
Trend · papers per month
48 results for “KL-divergence lower bound”
Tight Bounds on the Binomial CDF, and the Minimum of i.i.d Binomials, in terms of KL-Divergencemath.PR
The study tightens bounds on binomial probabilities and minimums using KL-divergence.
problem Tightening bounds on binomial probabilities and minimums of i.i.d. Binomials.
method Applied Sanov's theorem to derive upper and lower bounds on binomial tail probabilities and minimums, expressed in terms of KL-divergence.
result High probability upper and lower bounds on the minimum of i.i.d. Binomial random variables, finite sample, asymptotically tight.
Improved bounds for estimating discrete distributions in KL divergence.
problem Estimating discrete distributions in KL divergence with accuracy.
method Used Laplace estimator and established concentration bounds.
result Deviation from mean scales as for .
New bounds for sequential tests under power-one error levels.
problem Determining stopping times for sequential tests with power-one error levels.
method Proved two lower bounds for stopping times under specific conditions.
result Upper and lower bounds for sequential tests are shown to be tight.
We establish bounds on the KL divergence between two multivariate Gaussian distributions in terms of the Hamming distance between the edge sets of the corresponding graphical models. We show that the KL divergence is bounded below by a constant when the graphs differ by at least one edge; this is essentially the tighte…
Paper analyzes kNN estimator for KL divergence, proving its optimality.
problem Estimating KL divergence from identical samples.
method kNN estimator based on nearest neighbor distances.
result kNN method is asymptotically rate optimal for KL divergence estimation.
New bounds for Neyman-Pearson region using -divergences.
problem Bounding the Neyman-Pearson region for hypothesis testing.
method Establishing novel lower and upper bounds using -divergences.
result Best possible lower bound for the Neyman-Pearson boundary using hockey-stick -divergences.
The Dirichlet mechanism protects privacy while minimizing KL divergence.
problem Minimizing KL divergence while protecting sensitive data privacy.
method Using the exponential mechanism with the KL divergence loss function, resulting in the Dirichlet mechanism.
result Proved a probability tail bound on KL divergence and derived a lower bound for sample complexity.
Beta Diffusioncs.LG
Beta diffusion generates bounded data using multiplicative transitions.
problem Generating data within specific ranges.
method Integrates demasking and denoising with scaled and shifted beta distributions.
result KLUBs are more effective for optimizing beta diffusion compared to negative ELBOs.
New bounds close the score matching gap for diffusion models.
problem The difference between sample quality and score matching loss in diffusion models.
method Theoretical analysis of score matching gap, developing tighter bounds for KL divergence, reverse KL divergence, and Wasserstein distance.
result The quality of score approximation impacts closing the score matching gap for low noise scales.
Paper bridges VAEs and KDEs for more flexible posterior estimation.
problem Limitations of Gaussian latent space in VAEs and challenges in KL-divergence estimation.
method Approximate posterior with KDEs and derive upper bound of KL-divergence in ELBO.
result Epanechnikov kernel minimizes KL-divergence upper bound asymptotically.
This work improves bounds on Bayesian coreset quality.
problem Limitations of existing theoretical analysis of Bayesian coresets.
method Develops general upper and lower bounds on KL divergence.
result Demonstrates flexibility of new theoretical bounds in various models.
Record linkage involves merging records in large, noisy databases to remove duplicate entities. It has become an important area because of its widespread occurrence in bibliometrics, public health, official statistics production, political science, and beyond. Traditional linkage methods directly linking records to one…
Improved regret bounds for bandits with fixed expert advice using information theory.
problem Optimizing regret in bandit problems with fixed expert distributions.
method Information-theoretic analysis and KL-divergence measures.
result First regret bounds for EXP4 that can get arbitrarily close to zero under certain conditions.
A new upper bound for variational inference improves the efficiency of Bayesian deep learning.
problem Improving variational inference in Bayesian deep learning.
method Presented a new upper bound (EUBO) for evidence, derived from KL-divergence and log marginal likelihood, and used SGD for optimization.
result The new upper bound (EUBO) is tighter than previous methods and outperforms state-of-the-art results in Bayesian neural networks.
Improved BAI under DP reduces gap to constant.
problem Fixed-confidence BAI under global DP for Bernoulli distributions.
method New lower bound, stopping rule, and Top Two sampling rule.
result Reduces gap to a small multiplicative constant.
Private KL distribution estimation improved with instance-optimality.
problem Minimizing KL divergence between true and estimated distributions.
method Construct minimax optimal private estimators, then focus on instance-optimality.
result Achieved instance-optimality up to constant factors for KL estimation.
The natural gradient of ELBO vanishes in unconstrained optimization, simplifying learning.
problem The gap between evidence and ELBO has a vanishing natural gradient.
method Analyzes the Fisher-Rao gradient of ELBO and its implications for learning.
result Maximizing ELBO is equivalent to minimizing KL divergence, simplifying learning.
Flow matching KL divergence bound derived for smooth distributions.
problem Estimating smooth distributions efficiently.
method Deterministic upper bound on KL divergence derived from flow-matching loss.
result Flow matching achieves nearly minimax-optimal efficiency under TV distance.
Improved language models using ratio-matching and KL divergence.
problem Efficiently modeling discrete data with diffusion models.
method Introduced new theorems and a novel CTMC transition-rate matrix for ratio-matching and KL divergence.
result 10-15% improvement in perplexity and faster training steps.
We consider the problem of quantifying the quality of a model selection problem for a graphical model. We discuss this by formulating the problem as a detection problem. Model selection problems usually minimize a distance between the original distribution and the model distribution. For the special case of Gaussian di…
Variational Bayesian neural networks (BNNs) perform variational inference over weights, but it is difficult to specify meaningful priors and approximate posteriors in a high-dimensional weight space. We introduce functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund (ELBO) defi…
Study improves density estimation for compact domains using -lifted KL divergence.
problem Estimating probability density functions on compact domains.
method Introduced -lifted Kullback--Leibler (KL) divergence for risk minimization.
result Proved bound on estimation error.
New reweighted losses improve diffusion model training and image quality.
problem Training and improving diffusion models for image generation.
method Constructing a cascade of time-dependent variational lower bounds.
result Significant improvements in pixel-space image modeling quality.
Study compares chi-squared divergence and KL-divergence posteriors for PAC-Bayesian bounds.
problem Investigates optimal posteriors for PAC-Bayesian bounds using chi-squared divergence.
method Analyzes bounds for three distance functions, derives FP equations for computation.
result Chi-squared divergence based posteriors have weaker bounds and worse test errors.
Unified framework for understanding TVO and improving model learning.
problem Improving the tightness and efficiency of variational inference bounds.
method Exponential family interpretation and equal spacing in moment parameters.
result Unified framework and improved gradient estimator for TVO.
In-Context Learning as Nonparametric Conditional Probability Estimation: Risk Bounds and Optimalitystat.ML
Paper analyzes risk bounds for in-context learning in multiclass classification.
problem Risk bounds for in-context learning in multiclass classification.
method Formalizes tasks as sequences of labeled examples and queries, estimates conditional class probabilities, establishes oracle inequality for KL divergence.
result ICL achieves minimax optimal rate for conditional probability estimation.
New method proves dimension-free convergence for ULD in KL divergence.
problem Polynomial scaling of existing convergence guarantees in high dimensions.
method Refined KL local error framework, focusing on tr(H) instead of d.
result First dimension-free KL divergence bounds for discretized ULD.
Paper improves variational inference by tightening bounds using perturbation theory.
problem Improving variational inference's bias and KL divergence approximation.
method Revisits perturbation theory to derive corrections that tighten variational bounds.
result New bounds are tighter and more mass-covering, leading to higher likelihoods.
In this paper, we introduce Deep Probabilistic Ensembles (DPEs), a scalable technique that uses a regularized ensemble to approximate a deep Bayesian Neural Network (BNN). We do so by incorporating a KL divergence penalty term into the training objective of an ensemble, derived from the evidence lower bound used in var…
We investigate the use of alternative divergences to Kullback-Leibler (KL) in variational inference(VI), based on the Variational Dropout \cite{kingma2015}. Stochastic gradient variational Bayes (SGVB) \cite{aevb} is a general framework for estimating the evidence lower bound (ELBO) in Variational Bayes. In this work, …
t-SNE algorithm's points remain bounded under gradient flow.
problem Understanding the boundedness of t-SNE points.
method Gradient flow of t-SNE with KL divergence, examining weak convergence assumptions.
result Points generated by t-SNE remain bounded under gradient flow.
New algorithm clusters trajectories from multiple Markov chains with near-optimal error.
problem Clustering trajectories from multiple unknown Markov chains.
method Two-stage algorithm: spectral clustering followed by likelihood-based refinement.
result Achieves near-optimal clustering error with high probability.
KALE flow approximates KL divergence for distributions with disjoint support.
problem Approximating KL divergence for distributions with disjoint support.
method Relaxed KL gradient flow using RKHS, continuously interpolating between KL and MMD.
result Global convergence of KALE flow under sufficient smoothness assumptions.
SUMO provides unbiased log marginal likelihood estimation for latent variable models.
problem Biased estimates of log marginal likelihood in latent variable models.
method Randomized truncation of infinite series for unbiased estimation.
result Models trained with SUMO give better test-set likelihoods than standard methods.
Improved KL divergence estimators for normalizing flows lead to faster convergence and better approximations.
problem Estimating KL divergences for normalizing flows efficiently and accurately.
method Path-gradient estimators for reverse and forward KL divergences.
result Path-gradient estimators lead to faster convergence and better approximation results.
VI approximates complex densities faster than classical methods.
problem Approximating complex probability densities.
method Optimization of a family of probability density functions using KL divergence.
result VI converges faster than Markov Chain Monte Carlo.
Paper improves PAC-Bayes bounds using a better-than-KL divergence.
problem Estimating the generalization error of stochastic algorithms.
method Developed new PAC-Bayes bounds with a novel divergence.
result Achieved strictly tighter bounds than the KL divergence.
Adversarial perturbations dramatically decrease the accuracy of state-of-the-art image classifiers. In this paper, we propose and analyze a simple and computationally efficient defense strategy: inject random Gaussian noise, discretize each pixel, and then feed the result into any pre-trained classifier. Theoretically,…
We speed up marginal inference by ignoring factors that do not significantly contribute to overall accuracy. In order to pick a suitable subset of factors to ignore, we propose three schemes: minimizing the number of model factors under a bound on the KL divergence between pruned and full models; minimizing the KL dive…
Optimal Condition for Initialization Variance in Deep Neural Networks: An SGD Dynamics Perspectivestat.ML
Optimizes deep neural network initialization variance for better performance.
problem Improving deep neural network performance through optimal initialization variance.
method Using SGD dynamics and Fokker-Planck equations, we study the relationship between initialization and expected loss function.
result An optimal condition for initialization variance that leads to lower training loss and higher test accuracy.
VCL adds uncertainty to contrastive learning models.
problem Lack of uncertainty quantification in contrastive learning methods.
method VCL uses a decoder-free framework that maximizes ELBO with InfoNCE loss and KL divergence.
result VCL provides meaningful uncertainty estimates and matches deterministic baselines in accuracy.
DualVDT improves time-series forecasting with a novel dual reparametrized structure.
problem Time-series forecasting with improved performance and analytical rigor.
method Dual reparametrized variational mechanisms on VAE, latent score based generative model, reverse time stochastic differential equation, variational ancestral sampling, KL divergence reduction.
result Advanced performance in time-series forecasting with reduced KL divergence.
The paper examines VI for overparameterized BNNs, revealing a trade-off between likelihood and KL terms.
problem Critical issue in mean-field VI training for overparameterized BNNs.
method Theoretical and empirical study of overparameterized two-layer BNNs using VI.
result A trade-off between likelihood and KL terms in overparameterized regime, with KL scaling crucial.
Researchers establish bounds for SGMs' KL and Wasserstein divergences under various noise schedules.
problem Estimating the error between target and estimated distributions in SGMs.
method Established upper bounds for KL divergence and Wasserstein distance, incorporating target distribution properties and SGM hyperparameters.
result Optimal noise schedules identified for SGMs, improving generative quality.
Towards a Sharp Analysis of Offline Policy Learning for -Divergence-Regularized Contextual Banditscs.LG
Paper analyzes sample complexity for offline -divergence-regularized contextual bandits.
problem Lack of tight analyses for sample complexity in offline reinforcement learning.
method Novel pessimism-based analysis for reverse KL divergence, establishing sample complexity.
result Achieves sample complexity for reverse KL divergence, surpassing existing bounds.
Researchers estimate optimal PAC-Bayes bounds using Hamiltonian Monte Carlo.
problem Estimating tight PAC-Bayes bounds with restricted posterior families.
method Sampling from optimal Gibbs posterior using Hamiltonian Monte Carlo, estimating KL divergence, and proposing high-probability bounds.
result Significant tightness gaps in PAC-Bayes bounds, up to 5-6% in some cases.
Logit distance bounds representational similarity of models.
problem Approximating linear similarity when distributions are close.
method Defined a logit distance and proved its relationship to representational dissimilarity.
result Logit distance bounds representational similarity, providing nontrivial control in practice.