Unified representation of density-power-based divergences simplifies estimation to M-estimation.
problem Outliers in density estimation.
method Define a norm-based Bregman density power divergence (NB-DPD) that reduces to M-estimation.
result NB-DPD connects and generalizes existing divergences, highlighting robustness properties.
Estimates KL divergence with fairness considerations for sub-populations.
problem Fairly estimate KL divergence between distributions considering sub-populations.
method Proposes multi-group attribution for KL divergence estimation, derived from multi-calibration.
result Shows multi-group attribution provides better KL divergence estimates conditioned on sub-populations.
New α \alpha α -divergence loss function improves neural density ratio estimation.
problem Optimization challenges in existing DRE methods, especially overfitting and high sample requirements.
method Derived α \alpha α -divergence loss function ( α \alpha α -Div) for neural density ratio estimation. result The α \alpha α -divergence loss function ( α \alpha α -Div) offers stable and effective optimization for DRE. Improved bounds for estimating discrete distributions in KL divergence.
problem Estimating discrete distributions in KL divergence with accuracy.
method Used Laplace estimator and established concentration bounds.
result Deviation from mean scales as k / n \sqrt{k}/n k / n for n ≥ k n \ge k n ≥ k . New variational formula for Rényi divergences improves neural network estimation in high dimensions.
problem Estimating Rényi divergences in high-dimensional systems.
method Derive and apply a variational formula for Rényi divergences over various function spaces.
result Neural network estimators of Rényi divergences are consistent under certain conditions.
Paper analyzes kNN estimator for KL divergence, proving its optimality.
problem Estimating KL divergence from identical samples.
method kNN estimator based on nearest neighbor distances.
result kNN method is asymptotically rate optimal for KL divergence estimation.
Unified view of KL-divergence and IPMs via DRE, with new DRM metrics.
problem Unified understanding of KL-divergence and IPMs.
method Unified representation via maximum likelihood density-ratio estimation (DRE).
result Unified form of IPMs and novel DRM metrics.
Rank-statistic method approximates f f f -divergences without density-ratio estimation.
problem Approximating f f f -divergences without explicit density-ratio estimation. method Mapping distribution rank histograms to discrete f f f -divergence and averaging over random projections. result The rank-statistic estimator is a lower bound of the true f f f -divergence and converges under mild conditions. LLM safety alignment explained as divergence estimation.
problem Aligning large language models to avoid harmful outputs.
method Presented a theoretical framework showing alignment methods as divergence estimators.
result KLDO method improves safety alignment using compliance-refusal datasets.
In high-dimensional data, many sparse regression methods have been proposed. However, they may not be robust against outliers. Recently, the use of density power weight has been studied for robust parameter estimation and the corresponding divergences have been discussed. One of such divergences is the γ γ γ -divergence a…
The problem of f-divergence estimation is important in the fields of machine learning, information theory, and statistics. While several nonparametric divergence estimators exist, relatively few have known convergence properties. In particular, even for those estimators whose MSE convergence rates are known, the asympt…
Neural networks estimate statistical divergences with performance guarantees.
problem Estimating statistical divergences with theoretical performance guarantees.
method Parametrizing empirical variational form by a neural network and optimizing over parameter space.
result Established non-asymptotic absolute error bounds for neural estimators of four f \mathsf{f} f -divergences. We extend CS divergence to conditional distributions and show its advantages in time series data and sequential decision making.
problem Quantifying the closeness between conditional distributions.
method Developed and estimated a conditional Cauchy-Schwarz divergence using kernel density estimation.
result Conditional CS divergence outperforms previous methods in time series clustering and sequential decision making.
We extend a variational framework to estimate calibration errors for L p L_p L p divergences.
problem Ensuring predicted probabilities match observed class frequencies in machine learning.
method Extend variational framework to L p L_p L p divergences, separating over- and under-confidence. result Avoids overestimation and separates over- and under-confidence.
Unified framework for debiased machine learning using Riesz representer and Bregman divergence.
problem Estimating causal and structural parameters in machine learning.
method Generalized Riesz regression for fitting Riesz representer via Bregman divergence minimization.
result Automatic covariate balancing and Neyman orthogonality properties for debiased estimation.
Estimating divergences in a consistent way is of great importance in many machine learning tasks. Although this is a fundamental problem in nonparametric statistics, to the best of our knowledge there has been no finite sample exponential inequality convergence bound derived for any divergence estimators. The main cont…
E 2 ^2 2 M optimizes tensor density estimation by relaxing α α α -divergence to KL-divergence.
problem Analytical challenges in traditional α α α -divergence optimization for tensor-based density estimation. method E 2 ^2 2 M algorithm: relaxes optimization to KL-divergence, then applies tensor many-body approximation. result Flexible modeling of various low-rank structures and their mixtures.
New divergences improve estimation and GAN training performance.
problem Improving estimation and training in machine learning models.
method Function-space regularized Rényi divergences.
result New divergences reduce variance and improve training performance.
A density ratio is defined by the ratio of two probability densities. We study the inference problem of density ratios and apply a semi-parametric density-ratio estimator to the two-sample homogeneity test. In the proposed test procedure, the f-divergence between two probability densities is estimated using a density-r…
A new method optimizes a generalized Kullback-Leibler divergence for better simulation-based inference.
problem Optimizing likelihood functions when they are only known implicitly.
method Optimizes a generalized Kullback-Leibler divergence that accounts for normalization constants in unnormalized distributions.
result Unified approach that combines Neural Posterior Estimation and Neural Ratio Estimation.
Innovative inequalities for divergences with applications in PAC-Bayesian bounds and Monte Carlo.
problem Developing new inequalities for divergences.
method Introducing novel change of measure inequalities for f f f -divergences and α α α -divergences. result Applications in PAC-Bayesian bounds and Monte Carlo estimates.
Locally private mechanisms' output divergence bounds derived.
problem Bounding divergence between locally private mechanisms' outputs.
method Sharp upper bounds on divergence between input and output distributions.
result Established locally private versions of estimation risk bounds.
Study improves density estimation for compact domains using h h h -lifted KL divergence.
problem Estimating probability density functions on compact domains.
method Introduced h h h -lifted Kullback--Leibler (KL) divergence for risk minimization. result Proved O ( 1 / n ) \mathcal{O}(1/{\sqrt{n}}) O ( 1/ n ) bound on estimation error. Improved KL divergence estimators for normalizing flows lead to faster convergence and better approximations.
problem Estimating KL divergences for normalizing flows efficiently and accurately.
method Path-gradient estimators for reverse and forward KL divergences.
result Path-gradient estimators lead to faster convergence and better approximation results.
New divergences improve score-based methods for multi-modal distributions.
problem Blindness problem in score-based divergences for multi-modal distributions.
method Proposed a new family of divergences to mitigate blindness.
result Improved performance in density estimation compared to traditional approaches.
A new robust PCA estimator combining M-estimators and minimum divergence estimators.
problem Adverse effect of outlying observations in PCA for high-dimensional data.
method Minimum density power divergence estimator combined with a computationally efficient algorithm.
result High breakdown guarantee regardless of data dimension with theoretical support and practical applications.
Study introduces a variational approach for efficient KL divergence estimation in Dirichlet mixture models.
problem Efficient estimation of KL divergence in Dirichlet mixture models.
method Variational approach for a closed-form solution.
result Superior efficiency and accuracy compared to Monte Carlo methods.
Unified technique for sequential estimation of convex divergences.
problem Estimating convex divergences between distributions.
method Martingale methods and maximal inequalities for reverse submartingales.
result Valid time-uniform confidence sequences for arbitrary stopping times.
The paper provides estimates for eigenvalues of elliptic differential problems.
problem Computing eigenvalue estimates for elliptic differential problems.
method Analytical computation of eigenvalues for specific types of elliptic differential equations.
result Universal estimates of eigenvalues and gaps between consecutive eigenvalues are derived.
Proposes practical kernel tests for f f f -divergences with theoretical guarantees.
problem Two-sample testing and machine unlearning evaluation.
method Regularized f f f -divergence kernel tests, adaptive to hyperparameters. result Different f f f -divergences highlight localized differences. Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) are popular methods for training the weights of Restricted Boltzmann Machines. However, both methods use an approximate method for sampling from the model distribution. As a side effect, these approximations yield significantly different biases and…
Sharp bounds for high-probability estimation of discrete distributions.
problem Estimating discrete distributions with high probability under χ 2 χ^2 χ 2 -divergence. method Sharp upper and lower bounds for the classical Laplace estimator, and characterization of minimax high-probability risk for any estimator.
result Sharp bounds for high-probability estimation of discrete distributions can be achieved through a simple smoothing strategy.
The paper analyzes the statistical properties of GANs using f f f -divergence.
problem Understanding the statistical behavior of GANs and comparing different f f f -divergences. method Asymptotic analysis of f f f -divergence GANs, including Kullback-Leibler divergence. result Asymptotically equivalent GANs with the same discriminator classes for correctly specified models.
Proposes a new divergence measure for probability distributions.
problem Challenges in estimating divergences from empirical samples.
method Embeds data into RKHS, computes Jensen-Shannon divergence between covariance operators.
result Establishes RJSD as a lower bound on Jensen-Shannon divergence, enabling variational estimation.
New method minimizes robust density power-based divergences for general parametric densities.
problem Computational complexity of minimizing DPD for general parametric densities.
method Stochastic approach to minimize DPD for general parametric density models.
result Proposed method can be applied to minimize other density power-based γ-divergences.
Develops a direct debiased machine learning framework using Bregman divergence.
problem Reduces bias in machine learning estimates of causal effects or structural models.
method Neyman targeted estimation and generalized Riesz regression using Bregman divergence.
result Improves estimation of parameters of interest in causal models.
LDP is equivalent to contraction of E_γ-divergence, impacting privacy and utility.
problem Analyzing trade-offs between privacy and utility in estimation problems.
method Equivalence of LDP constraints to contraction coefficients of E_γ-divergence, using f-divergences and estimation-theoretic tools.
result LDP guarantees can be expressed in terms of contraction coefficients of arbitrary f-divergences.
This paper provides performance guarantees for neural estimation of statistical distances.
problem Developing performance guarantees for neural estimation of statistical distances.
method Non-asymptotic error bounds using function approximation theorems and empirical process theory.
result Established a fundamental tradeoff between approximation and estimation errors in neural estimation of statistical distances.
AlphaNet improves supernets training with alpha-divergence.
problem Improving the uncertainty distillation in weight-sharing NAS.
method Proposes alpha-divergence for better uncertainty distillation in supernets.
result Significant improvements in model performance across various FLOPs regimes.
The paper addresses instability in KL divergence estimation using a neural network discriminator.
problem Unstable estimation of KL divergence due to discriminator complexity.
method Using a Reproducing Kernel Hilbert Space (RKHS) to control discriminator complexity.
result Theoretical bound on error probability of KL estimates based on discriminator complexity in RKHS.
In this paper, we derive a useful lower bound for the Kullback-Leibler divergence (KL-divergence) based on the Hammersley-Chapman-Robbins bound (HCRB). The HCRB states that the variance of an estimator is bounded from below by the Chi-square divergence and the expectation value of the estimator. By using the relation b…
Paper proposes a method to stabilize estimation of KL divergence using a discriminator in RKHS.
problem High variance and instability in estimating KL divergence using neural network discriminators.
method Developed a novel construction of the discriminator in RKHS, controlled its complexity, and proved the consistency of the estimator.
result Reduced variance and stabilized training of KL divergence estimates.
We investigate the use of alternative divergences to Kullback-Leibler (KL) in variational inference(VI), based on the Variational Dropout \cite{kingma2015}. Stochastic gradient variational Bayes (SGVB) \cite{aevb} is a general framework for estimating the evidence lower bound (ELBO) in Variational Bayes. In this work, …
We show that the variational representations for f-divergences currently used in the literature can be tightened. This has implications to a number of methods recently proposed based on this representation. As an example application we use our tighter representation to derive a general f-divergence estimator based on t…
A new method for faster estimation of Wasserstein distance using Sinkhorn divergence.
problem Estimating the squared Wasserstein distance between probability distributions.
method Proposes a new estimator based on the Sinkhorn divergence with debiasing terms, and analyzes its sample complexity and computational efficiency.
result The proposed estimator allows higher regularization levels, leading to improved computational complexity and speedup in practice.
New method estimates covariance matrices without restrictive assumptions.
problem Estimating high-dimensional covariance matrices under restrictive assumptions.
method Distributionally robust covariance estimation problems with mild conditions.
result Robust estimators are efficient, consistent, and perform well.
A study on α \alpha α -GANs proving convergence and estimation guarantees.
problem Analyzing the convergence and estimation guarantees of α \alpha α -GANs. method Proved a correspondence between α \alpha α -GANs and f f f -divergences, and provided estimation bounds. result Estimation bounds indicate diverse GAN behavior as a function of α \alpha α . We consider nonparametric estimation of L 2 L_2 L 2 , Renyi- α α α and Tsallis- α α α divergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on cor…