Unified view of KL-divergence and IPMs via DRE, with new DRM metrics.
problem Unified understanding of KL-divergence and IPMs.
method Unified representation via maximum likelihood density-ratio estimation (DRE).
result Unified form of IPMs and novel DRM metrics.
Proposes ATM method to improve domain adaptation.
problem Mitigating distribution divergence between source and target domains.
method Adversarial Tight Match (ATM) method using Maximum Density Divergence (MDD).
result New state-of-the-art performance on domain adaptation benchmarks.
Study improves density estimation for compact domains using h h h -lifted KL divergence.
problem Estimating probability density functions on compact domains.
method Introduced h h h -lifted Kullback--Leibler (KL) divergence for risk minimization. result Proved O ( 1 / n ) \mathcal{O}(1/{\sqrt{n}}) O ( 1/ n ) bound on estimation error. Continuous-time PCD for MLE with explicit error bounds.
problem Maximum likelihood estimation of unnormalised densities.
method Continuous-time formulation as coupled SDEs, deriving UiT bounds.
result Explicit error bounds between PCD iterates and MLE solution.
We extend CS divergence to conditional distributions and show its advantages in time series data and sequential decision making.
problem Quantifying the closeness between conditional distributions.
method Developed and estimated a conditional Cauchy-Schwarz divergence using kernel density estimation.
result Conditional CS divergence outperforms previous methods in time series clustering and sequential decision making.
NDI learns from expert demonstrations by estimating occupancy measures.
problem Imitation Learning (IL) for complex systems.
method Density estimation of expert's occupancy measure followed by RL.
result NDI achieves state-of-the-art performance on control tasks.
Paper proposes f-EBM for training deep EBMs using various f-divergences.
problem Training deep EBMs with intractable partition functions.
method Introduces f-EBM framework and optimization algorithm for any f-divergence.
result f-EBM outperforms contrastive divergence and other f-divergences.
Unified framework for various probability distribution distances.
problem Handling diverse probability distribution distances in statistics.
method General framework covering density-based and distribution-function-based divergences.
result Unified approach to classical and modern statistical procedures.
Unified representation of density-power-based divergences simplifies estimation to M-estimation.
problem Outliers in density estimation.
method Define a norm-based Bregman density power divergence (NB-DPD) that reduces to M-estimation.
result NB-DPD connects and generalizes existing divergences, highlighting robustness properties.
New method calibrates reference distributions for bounded support.
problem Lack of principled method for bounded-support statistical reference distributions.
method Formulated maximum entropy on projective space of nonnegative measures.
result Prescribed acceptance region uniquely determines deformation parameter.
New method minimizes robust density power-based divergences for general parametric densities.
problem Computational complexity of minimizing DPD for general parametric densities.
method Stochastic approach to minimize DPD for general parametric density models.
result Proposed method can be applied to minimize other density power-based γ-divergences.
Unified framework for debiased machine learning using Riesz representer and Bregman divergence.
problem Estimating causal and structural parameters in machine learning.
method Generalized Riesz regression for fitting Riesz representer via Bregman divergence minimization.
result Automatic covariate balancing and Neyman orthogonality properties for debiased estimation.
Analyzed a generative model framework through Wasserstein Gradient Flow.
problem Generative modeling challenges.
method Wasserstein Gradient Flow (WGF) interpretation of Drifting Models (GMD).
result Different algorithms correspond to specific limiting points of WGFs on various divergences.
New robust learning framework for regression NNs using β-divergences.
problem Outliers and data contamination in regression NNs training.
method Proposes rRNet based on β-divergence for robust learning of regression NNs.
result rRNet achieves optimal 50% asymptotic breakdown point for all β ∈ (0, 1].
Maximum regularized likelihood estimators (MRLEs) are arguably the most established class of estimators in high-dimensional statistics. In this paper, we derive guarantees for MRLEs in Kullback-Leibler divergence, a general measure of prediction accuracy. We assume only that the densities have a convex parametrization …
New method uses SoS densities and α-divergences for efficient sequential transport maps.
problem Efficiently generating samples from approximated densities.
method Sequential transport maps using Sum-of-Squares (SoS) densities and α-divergences.
result Convex optimization problems with efficient semidefinite programming solutions.
The paper provides stability guarantees for non-parametric maximum likelihood estimation using statistical mechanics.
problem Non-parametric maximum likelihood estimation and Gaussian mixture models.
method Statistical mechanics analysis to establish stability guarantees for NPMLE.
result High probability upper bounds on the Kullback-Leibler divergence between NPMLE estimators and the true density.
Study explores relationship between Hölder and FDPD divergences.
problem Understanding the relationship between Hölder and FDPD divergences.
method Intersection and generalization of divergence families, proving nonnegativity, deriving inequalities.
result Established ξ ξ ξ -Hölder divergence and derived inequalities. Proposes a new flow model to better represent data on manifolds.
problem Flow models struggle to represent data on lower-dimensional manifolds accurately.
method Introduces a manifold prior that leverages spread divergence to improve model performance.
result Improves both sample and representation quality, identifies manifold intrinsic dimension.
A density ratio is defined by the ratio of two probability densities. We study the inference problem of density ratios and apply a semi-parametric density-ratio estimator to the two-sample homogeneity test. In the proposed test procedure, the f-divergence between two probability densities is estimated using a density-r…
New α \alpha α -divergence loss function improves neural density ratio estimation.
problem Optimization challenges in existing DRE methods, especially overfitting and high sample requirements.
method Derived α \alpha α -divergence loss function ( α \alpha α -Div) for neural density ratio estimation. result The α \alpha α -divergence loss function ( α \alpha α -Div) offers stable and effective optimization for DRE. In this paper, we consider an infinite dimensional exponential family, P \mathcal{P} P of probability densities, which are parametrized by functions in a reproducing kernel Hilbert space, H H H and show it to be quite rich in the sense that a broad class of densities on R d \mathbb{R}^d R d can be approximated arbitrarily well i…
Divergence estimators based on direct approximation of density-ratios without going through separate approximation of numerator and denominator densities have been successfully applied to machine learning tasks that involve distribution comparison such as outlier detection, transfer learning, and two-sample homogeneity…
Paper studies regularized KKL divergence for distributions with disjoint supports.
problem Inability of original KKL divergence to handle distributions with disjoint supports.
method Proposes a regularized variant of KKL divergence, derives bounds, and provides closed-form expression.
result Regularized KKL divergence is well-defined for all distributions and has finite-sample bounds.
Truncated densities are probability density functions defined on truncated domains. They share the same parametric form with their non-truncated counterparts up to a normalizing constant. Since the computation of their normalizing constants is usually infeasible, Maximum Likelihood Estimation cannot be easily applied t…
E 2 ^2 2 M optimizes tensor density estimation by relaxing α α α -divergence to KL-divergence.
problem Analytical challenges in traditional α α α -divergence optimization for tensor-based density estimation. method E 2 ^2 2 M algorithm: relaxes optimization to KL-divergence, then applies tensor many-body approximation. result Flexible modeling of various low-rank structures and their mixtures.
Persistently trained EBMs generate images and estimate complex densities.
problem Challenges in ML learning for energy-based models, especially non-convergence of MCMC.
method Introduce diffusion data, learn a joint EBM through persistent training with enhanced sampling.
result First simultaneous achievement of stability, post-training image generation, and superior out-of-distribution detection for image data.
Rank-statistic method approximates f f f -divergences without density-ratio estimation.
problem Approximating f f f -divergences without explicit density-ratio estimation. method Mapping distribution rank histograms to discrete f f f -divergence and averaging over random projections. result The rank-statistic estimator is a lower bound of the true f f f -divergence and converges under mild conditions. A method for eliciting expert beliefs using preferential questions and normalizing flows.
problem Eliciting high-dimensional probability distributions from noisy judgments.
method Normalizing flows based on preferential questions with a novel functional prior.
result The method allows for the inference of arbitrarily flexible densities from preferential judgments.
Adapts RKHS methods to estimate density ratios with optimal error.
problem Estimating density ratios from limited data.
method Minimizes regularized Bregman divergence in RKHS, with Lepskii type parameter choice.
result Adaptive minimax optimal error rate for quadratic loss.
Proposes a guaranteed regularization method for maximum likelihood estimation using gauge symmetry in Kullback-Leibler divergence.
problem Overfitting in maximum likelihood estimation.
method Introduces a regularization approach based on gauge symmetry in Kullback-Leibler divergence.
result The method provides a theoretically guaranteed optimal model without frequent hyperparameter tuning.
Score matching fails to train VAEs robustly, revealing autoencoding loss insights.
problem Catastrophic failure of variational score matching on VAE models.
method Analysis of existing variational score matching objectives and their equivalence to autoencoding losses.
result Score matching methods fail to produce robust VAE models, predicting poor performance.
New divergences improve score-based methods for multi-modal distributions.
problem Blindness problem in score-based divergences for multi-modal distributions.
method Proposed a new family of divergences to mitigate blindness.
result Improved performance in density estimation compared to traditional approaches.
Unified framework for robust, stable, and efficient density ratio estimation.
problem Density-chasm and support-chasm problems in density ratio estimation.
method Dequantified diffusion-Schrödinger bridge (D3RE) framework with DDBI and DSBI.
result Offers uniform approximation and bounded time scores in theory and empirical performance.
New framework improves experimental design using integral probability metrics.
problem Challenges in Bayesian Optimal Experimental Design (BOED) with KL divergence.
method Integrates integral probability metrics (IPMs) for stability and flexibility.
result IPM-based designs yield more robust and accurate credible sets.
Improved GANs estimate convergence rate for density estimation.
problem Improving the accuracy of density estimation with GANs.
method Proved an oracle inequality for JS divergence between GAN estimate and true density.
result JS-divergence rate of convergence is ( log n / n ) 2 β / ( 2 β + d ) (\log{n}/n)^{2β/(2β+ d)} ( log n / n ) 2 β / ( 2 β + d ) . A new approach to model rejection using density ratios.
problem Improving model performance through selective prediction.
method Optimization of a loss's risk with φ-divergence regularization to find an idealized data distribution.
result Model rejection can be made by comparing the density ratio of the idealized distribution to the actual data distribution.
New analysis of annealing paths in sampling and estimation.
problem Sampling from complex distributions and estimating normalization constants.
method Extending known results on Bregman divergence to quasi-arithmetic means under monotonic embedding.
result Analogous result for quasi-arithmetic means, highlighting the interplay between means, parametric families, and divergence functionals.
Probabilistic models are often trained by maximum likelihood, which corresponds to minimizing a specific f-divergence between the model and data distribution. In light of recent successes in training Generative Adversarial Networks, alternative non-likelihood training criteria have been proposed. Whilst not necessarily…
This work tackles sequential data learning challenges by improving neural network robustness to non-iid distribution shifts.
problem Sequential data learning challenges, particularly non-iid distribution shifts across batches.
method Cramér-Rao-based regularization using Fisher Information Matrix to adapt to sequential covariate shifts.
result Achieves 19% accuracy improvement over state-of-the-art methods.
Moser Flow generates models for complex geometries on manifolds without ODE solvers.
problem Learning generative models for complex geometries like spheres and tori.
method Moser Flow is a new class of continuous normalizing flows that parameterizes the model density as the divergence of a neural network.
result Moser Flow achieves significant improvements in density estimation, sample quality, and training complexity over existing methods.
Method uses normalizing flows to efficiently sample from complex target densities.
problem Sampling from complex target densities with zero values in regions of transformation.
method Normalizing flows to address exploding reverse Kullback-Leibler divergence.
result Demonstrated efficient sampling from multi-mode complex density function.
The variational autoencoder (VAE) is a powerful generative model that can estimate the probability of a data point by using latent variables. In the VAE, the posterior of the latent variable given the data point is regularized by the prior of the latent variable using Kullback Leibler (KL) divergence. Although the stan…
As a robust nonlinear similarity measure in kernel space, correntropy has received increasing attention in domains of machine learning and signal processing. In particular, the maximum correntropy criterion (MCC) has recently been successfully applied in robust regression and filtering. The default kernel function in c…
Generative adversarial networks (GANs) are successful deep generative models. GANs are based on a two-player minimax game. However, the objective function derived in the original motivation is changed to obtain stronger gradients when learning the generator. We propose a novel algorithm that repeats the density ratio e…
Paper analyzes robustness of MDPDE under INH setups.
problem Global reliability and breakdown behavior of MDPDE under INH.
method Asymptotic breakdown point analysis of MDPDE.
result Derives a theoretical lower bound for the asymptotic breakdown point.
Estimation of density derivatives is a versatile tool in statistical data analysis. A naive approach is to first estimate the density and then compute its derivative. However, such a two-step approach does not work well because a good density estimator does not necessarily mean a good density-derivative estimator. In t…
The need to estimate smooth probability distributions (a.k.a. probability densities) from finite sampled data is ubiquitous in science. Many approaches to this problem have been described, but none is yet regarded as providing a definitive solution. Maximum entropy estimation and Bayesian field theory are two such appr…