Study shows the corrected Akaike criterion is inadmissible for estimating Kullback-Leibler discrepancy.
problem Inadmissibility of the corrected Akaike information criterion for estimating Kullback-Leibler discrepancy.
method Loss estimation framework to demonstrate inadmissibility and provide improved estimators.
result Improved estimators of Kullback-Leibler discrepancy are provided and perform well in reduced-rank situations.
A new method for Gaussian filtering using gradient flows and Wasserstein metrics.
problem Approximating Gaussian and mixture-of-Gaussians filtering for complex systems.
method Variational approximation via gradient-flow representation on Wasserstein metric space.
result Competitive performance in posterior representation and parameter estimation for systems with multiplicative noise and multi-modal distributions.
New method uses kernel Stein discrepancy for measure transport without strict continuity constraints.
problem Minimizing Kullback-Leibler divergence for posterior approximation.
method Proposes minimizing kernel Stein discrepancy instead of Kullback-Leibler divergence.
result Demonstrates consistency and competitiveness of the new method.
Paper studies regularized KKL divergence for distributions with disjoint supports.
problem Inability of original KKL divergence to handle distributions with disjoint supports.
method Proposes a regularized variant of KKL divergence, derives bounds, and provides closed-form expression.
result Regularized KKL divergence is well-defined for all distributions and has finite-sample bounds.
A brain computer interface (BCI) is a system which provides direct communication between the mind of a person and the outside world by using only brain activity (EEG). The event-related potential (ERP)-based BCI problem consists of a binary pattern recognition. Linear discriminant analysis (LDA) is widely used to solve…
A new method improves density ratio estimation efficiency and accuracy.
problem Density ratio estimation trade-off between quality and efficiency.
method One-step Score-based Density Ratio Estimation (OS-DRE) combining analytic and solver-free approach.
result OS-DRE offers a favorable balance between estimation quality and inference efficiency.
Introduces Cauchy-Schwarz divergence for domain adaptation.
problem Evaluating discrepancy between source and target domains in unsupervised domain adaptation.
method Introduces Cauchy-Schwarz divergence as a measure for evaluating discrepancy between marginal and conditional distributions.
result CS divergence offers a tighter generalization error bound than Kullback-Leibler divergence.
The paper develops new algorithms for KL-divergence NMF, proving convergence and performance.
problem Improving NMF for nonnegative data with KL divergence.
method Collect and analyze properties of KL objective function, propose and test new algorithms.
result Guaranteed non-increasing objective function for one proposed algorithm, global convergence.
Bayesian Neural Networks improve accuracy and robustness.
problem Training and optimizing Bayesian Neural Networks (BNNs).
method Replace Kullback-Leibler divergence with Maximum Mean Discrepancy (MMD) in ELBO term.
result Higher accuracy on multiple benchmarks, including image classification tasks.
The paper proves sampling methods using discrete-time processes and information theory.
problem Proving convergence guarantees for diffusion-based sampling methods.
method Directly works with discrete-time stochastic processes and uses information theory.
result Discrepancy between sampling and comparison processes is bounded using information theory.
New framework improves experimental design using integral probability metrics.
problem Challenges in Bayesian Optimal Experimental Design (BOED) with KL divergence.
method Integrates integral probability metrics (IPMs) for stability and flexibility.
result IPM-based designs yield more robust and accurate credible sets.
Motivated by applications in hyperspectral imaging we investigate methods for approximating a high-dimensional non-negative matrix Y by a product of two lower-dimensional, non-negative matrices K and X. This so-called non-negative matrix factorization is based…
In this paper we extend the setting of the online prediction with expert advice to function-valued forecasts. At each step of the online game several experts predict a function, and the learner has to efficiently aggregate these functional forecasts into a single forecast. We adapt basic mixable (and exponentially conc…
We develop a method to combine Markov chain Monte Carlo (MCMC) and variational inference (VI), leveraging the advantages of both inference approaches. Specifically, we improve the variational distribution by running a few MCMC steps. To make inference tractable, we introduce the variational contrastive divergence (VCD)…
We propose a framework for solving high-dimensional Bayesian inference problems using \emph{structure-exploiting} low-dimensional transport maps or flows. These maps are confined to a low-dimensional subspace (hence, lazy), and the subspace is identified by minimizing an upper bound on the Kullback--Leibler divergence …
Score matching errors are not sufficient for measuring diffusion model quality.
problem The L2 score matching error is not a reliable measure of diffusion model performance. method Decomposed score errors into gradient and solenoidal components and analyzed their geometric properties.
result Only the gradient component of the score error affects the marginal distributional quality.
Develops a KL-divergence-based deep learning method for survival analysis with short data.
problem Challenges of training deep learning models with limited survival data.
method Kullback-Leibler (KL) divergence to integrate external and internal data.
result Proposed model achieves better performance and higher robustness.
New geometric analysis shows L2 score error is flawed for diffusion models.
problem Score matching errors in diffusion models do not fully capture distributional quality.
method Decomposed score errors into gradient and solenoidal components, focusing on gradient's role in Fokker-Planck dynamics.
result Only gradient component affects marginal distributional quality; solenoidal component is structurally invisible.
SHAP Distance assesses semantic fidelity of synthetic tabular data.
problem Semantic fidelity of synthetic tabular data is not well evaluated.
method SHAP Distance, defined as cosine distance between global SHAP attribution vectors.
result SHAP Distance detects semantic discrepancies overlooked by standard measures.
We conduct non-asymptotic analysis on the mean-field variational inference for approximating posterior distributions in complex Bayesian models that may involve latent variables. We show that the mean-field approximation to the posterior can be well-approximated relative to the Kullback-Leibler divergence discrepancy m…
Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using …
Synthesizes sensor likelihoods to enforce accuracy constraints in uncertain systems.
problem Designing sensing architectures for systems with uncertain or unavailable sensor models and accuracy requirements.
method Inverts the design flow, synthesizing measurement likelihoods that minimize Kullback-Leibler divergence from the prior while enforcing an accuracy bound.
result The method synthesizes a maximum-entropy posterior and induced likelihood, accommodating various discrepancy metrics.
Optimizes kernel discrepancies by selecting subsets efficiently.
problem Improving kernel discrepancies for QMC methods.
method Introduces a novel subset selection algorithm for kernel discrepancies.
result Efficiently generates low-discrepancy samples from various distributions.
New dispersion indices based on inaccuracy and divergence introduced for information measures.
problem Measuring variability in uncertainty measures.
method Introducing new dispersion indices based on Kerridge inaccuracy and Kullback-Leibler divergence.
result Properties, bounds, and examples of new dispersion indices presented.
Rényi divergence is related to Rényi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by Rényi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its or…
New discrepancy function compares discrete probability measures considering space geometry.
problem Comparing discrete probability measures in a geometrically meaningful way.
method Proposes the Fourier Discrepancy Function, proving convexity, differentiability, and providing gradient formula.
result Proves the Fourier Discrepancy is convex, twice differentiable, and provides an explicit gradient formula.
This paper introduces localized discrepancy theories for unsupervised domain adaptation.
problem Improving generalization bounds for unsupervised domain adaptation.
method Localized discrepancies defined on the hypothesis space after localization, leading to smaller and asymmetric values.
result Improved generalization bounds and sample complexity reduction.
The problem of filtering information from large correlation matrices is of great importance in many applications. We have recently proposed the use of the Kullback-Leibler distance to measure the performance of filtering algorithms in recovering the underlying correlation matrix when the variables are described by a mu…
Develops a direct debiased machine learning framework using Bregman divergence.
problem Reduces bias in machine learning estimates of causal effects or structural models.
method Neyman targeted estimation and generalized Riesz regression using Bregman divergence.
result Improves estimation of parameters of interest in causal models.
A new method optimizes a generalized Kullback-Leibler divergence for better simulation-based inference.
problem Optimizing likelihood functions when they are only known implicitly.
method Optimizes a generalized Kullback-Leibler divergence that accounts for normalization constants in unnormalized distributions.
result Unified approach that combines Neural Posterior Estimation and Neural Ratio Estimation.
Regularizes f-divergences with MMD to analyze Wasserstein flows.
problem Limitations of f-divergences in measures' support. method Rewriting MMD regularization as Moreau envelope in RKHS, analyzing gradients.
result Analysis of Wasserstein flows of MMD-regularized f-divergences. The article introduces practical estimators for kernel discrepancies.
problem Estimating kernel discrepancies accurately and efficiently.
method Presented various estimators for MMD, HSIC, and KSD, including V-statistics, U-statistics, and incomplete U-statistics. Stressed the importance of kernel bandwidth and introduced adaptive estimators.
result Adaptive estimators combining multiple estimators with various kernels address the problem of kernel selection.
One of the challenges in training generative models such as the variational auto encoder (VAE) is avoiding posterior collapse. When the generator has too much capacity, it is prone to ignoring latent code. This problem is exacerbated when the dataset is small, and the latent dimension is high. The root of the problem i…
New error bound for diffusion models without dimensionality constraints.
problem Error estimation in diffusion generative models without dimensionality constraints.
method Derive dimension-free error bound using a smooth test functional.
result Explicit, dimension-free bound on generated vs true data distributions.
We present a derivation of the Kullback Leibler (KL)-Divergence (also known as Relative Entropy) for the von Mises Fisher (VMF) Distribution in d-dimensions.
Mamba improves time series forecasting by quantifying uncertainty.
problem Mamba forecasts have high mean errors in benchmarks.
method Dual-network framework for probabilistic forecasting.
result Predictive uncertainty reduced significantly for both synthetic and real-world data.
SnapMMD forecasts cell differentiation outcomes from snapshot data.
problem Forecasting cell differentiation outcomes from limited snapshot data.
method SnapMMD learns dynamics by directly fitting the joint distribution of state measurements and observation time with MMD loss, allowing for unknown and state-dependent volatilities.
result SnapMMD delivers accurate forecasts and an R2-style statistic for diagnosing fit.
Paper tackles backwards-compatible data adaptation for confounded covariate and label shifts.
problem Adapt covariates to predict labels confounded with covariate shifts.
method Proposes confounded shift framework based on minimizing divergence between source and target conditional distributions, conditioning on confounders.
result Demonstrates approach on synthetic and real datasets, achieving backwards-compatible data adaptation.
MPMC generates low-discrepancy points using graph neural networks.
problem Generating efficient low-discrepancy point sets.
method Leveraging Graph Neural Networks to model geometric properties.
result Achieves state-of-the-art performance in generating low-discrepancy points.
Sliced kernelized Stein discrepancy improves goodness-of-fit tests and model learning in high dimensions.
problem The curse-of-dimensionality in kernelized Stein discrepancy (KSD).
method Sliced Stein discrepancy and its scalable variants using optimal one-dimensional projections.
result Significantly outperforms KSD and baselines in goodness-of-fit tests and improves model learning.
Much of machine learning relies on comparing distributions with discrepancy measures. Stein's method creates discrepancy measures between two distributions that require only the unnormalized density of one and samples from the other. Stein discrepancies can be combined with kernels to define kernelized Stein discrepanc…
Paper calculates KL divergence for isotropic Gaussian-Markov fields.
problem Measuring divergence between isotropic Gaussian-Markov fields.
method Derives closed-form KL divergence expressions.
result Develops new similarity measures in image processing.
This paper defines the notion of class discrepancy for families of functions. It shows that low discrepancy classes admit small offline and streaming coresets. We provide general techniques for bounding the class discrepancy of machine learning problems. As corollaries of the general technique we bound the discrepancy …
Paper analyzes sparse aggregation in GLMs with Kullback-Leibler risk bounds.
problem Sparse aggregation in GLMs for parameter approximation.
method Exponential weighted aggregation scheme with Kullback-Leibler risk bounds.
result Sharp oracle inequality for Kullback-Leibler risk with leading constant 1 and minimax-optimal rate of aggregation.
Semi-parametric framework for nonlinear system identification
problem Nonlinear system identification
method Orthogonal Gaussian process regression
result Interpretable models from incomplete physics
TMDA aligns subdomain data distribution discrepancies across domains using manifold representations.
problem Transfer learning challenges due to domain divergence.
method TMDA uses low-dimensional manifolds to represent subdomains and aligns local data distribution discrepancies across domains using M3D.
result TMDA is a promising method for various transfer learning tasks.
The performance of standard learning procedures has been observed to differ widely across groups. Recent studies usually attribute this loss discrepancy to an information deficiency for one group (e.g., one group has less data). In this work, we point to a more subtle source of loss discrepancy---feature noise. Our mai…
New partition designs reduce star discrepancy in high-dimensional sampling.
problem Improving the expected star discrepancy in high-dimensional sampling.
method Developed non-equal volume partitions to achieve lower expected star discrepancy.
result Explicit upper bounds for expected star discrepancy under non-equal volume partitions.