New divergence measures improve KL approximation.
problem Improving KL divergence approximation without AC condition.
method Introduced -geodesical skew divergence.
result Properties of -geodesical skew divergence studied.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Parameter Divergence”
In high-dimensional data, many sparse regression methods have been proposed. However, they may not be robust against outliers. Recently, the use of density power weight has been studied for robust parameter estimation and the corresponding divergences have been discussed. One of such divergences is the -divergence a…
Study on geometric Jensen-Shannon divergence for Gaussian measures in Hilbert space.
problem Computing divergence between Gaussian measures in infinite-dimensional Hilbert space.
method Closed form expression and regularization for divergence calculation.
result Closed form expression and regularization for Geometric Jensen-Shannon divergence.
Rényi divergence is related to Rényi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by Rényi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its or…
Proposes practical kernel tests for -divergences with theoretical guarantees.
problem Two-sample testing and machine unlearning evaluation.
method Regularized -divergence kernel tests, adaptive to hyperparameters.
result Different -divergences highlight localized differences.
Improved Bayesian inference using power priors with historical data.
problem Improving Bayesian inference with historical data.
method Generalized power priors that adapt to the parameter of Amari's -divergence.
result Improved performance through appropriate choices of the parameter.
Distances are fundamental primitives whose choice significantly impacts the performances of algorithms in machine learning and signal processing. However selecting the most appropriate distance for a given task is an endeavor. Instead of testing one by one the entries of an ever-expanding dictionary of {\em ad hoc} dis…
Privacy amplification improved through contraction coefficients and -divergence.
problem Improving privacy guarantees in iterative algorithms.
method Using contraction coefficients derived from -divergence to determine differential privacy parameters.
result Tighter bounds on differential privacy parameters of iterative algorithms.
We investigate the use of alternative divergences to Kullback-Leibler (KL) in variational inference(VI), based on the Variational Dropout \cite{kingma2015}. Stochastic gradient variational Bayes (SGVB) \cite{aevb} is a general framework for estimating the evidence lower bound (ELBO) in Variational Bayes. In this work, …
Proposes a new measure to evaluate stability of statistical parameters under distributional shifts.
problem Difficulty in transferring knowledge across data sets due to distributional changes.
method Introduces a measure of instability quantifying sensitivity of statistical parameters to Kullback-Leibler divergence and directional shifts.
result The proposed measure can elucidate the type of shifts a parameter is sensitive to and improve estimation accuracy under shifted distributions.
Develops a direct debiased machine learning framework using Bregman divergence.
problem Reduces bias in machine learning estimates of causal effects or structural models.
method Neyman targeted estimation and generalized Riesz regression using Bregman divergence.
result Improves estimation of parameters of interest in causal models.
This paper introduces a novel approach for learning to rank (LETOR) based on the notion of monotone retargeting. It involves minimizing a divergence between all monotonic increasing transformations of the training scores and a parameterized prediction function. The minimization is both over the transformations as well …
Automates VI divergence selection for efficient few-shot learning.
problem Efficiently selecting divergence measures for VI to improve performance.
method Meta-learning algorithm to learn optimal divergence metric and variational parameter initialization.
result Meta-learning approach outperforms standard VI methods across various tasks.
This paper introduces a variational approximation framework using direct optimization of what is known as the {\it scale invariant Alpha-Beta divergence} (sAB divergence). This new objective encompasses most variational objectives that use the Kullback-Leibler, the R{é}nyi or the gamma divergences. It also gives access…
We present the very first robust Bayesian Online Changepoint Detection algorithm through General Bayesian Inference (GBI) with -divergences. The resulting inference procedure is doubly robust for both the parameter and the changepoint (CP) posterior, with linear time and constant space complexity. We provide a const…
AES uses α-divergence to select informative points for BO, improving optimization performance.
problem Optimizing complex functions with limited evaluations.
method AES uses α-divergence to select points based on dependency with global maximum.
result AES outperforms other information-based acquisition functions in various experiments.
Rank-statistic method approximates -divergences without density-ratio estimation.
problem Approximating -divergences without explicit density-ratio estimation.
method Mapping distribution rank histograms to discrete -divergence and averaging over random projections.
result The rank-statistic estimator is a lower bound of the true -divergence and converges under mild conditions.
Matrix SMD converges to unique solution minimizing Bregman divergence.
problem High-dimensional multi-output classification and matrix completion problems.
method Stochastic Mirror Descent with matrix parameters and matrix mirror functions.
result Matrix SMD converges exponentially to the unique solution minimizing Bregman divergence.
This paper addresses the estimation of the latent dimensionality in nonnegative matrix factorization (NMF) with the β-divergence. The β-divergence is a family of cost functions that includes the squared Euclidean distance, Kullback-Leibler and Itakura-Saito divergences as special cases. Learning the model order is impo…
New method uses KL-divergence to create non-informative priors for multivariate Gaussian.
problem Handling hyperparameters for non-informative limits in multivariate Gaussian conjugate priors.
method Using scaled KL-divergence between multivariate Gaussians to construct Wishart and normal-Wishart conjugate priors.
result Forming non-informative priors without violating Wishart shape parameter restrictions.
A novel stepwise VI method using vine copulas for complex latent dependence.
problem Modeling complex latent dependence structures in probabilistic models.
method Stepwise estimation of vine copula parameters using Rényi divergence and a stopping criterion.
result Our method outperforms mean-field VI and is more parsimonious in complex applications.
BaM improves BBVI by optimizing a score-based divergence, leading to faster convergence.
problem Slow convergence of black-box variational inference methods.
method Batch and match (BaM) approach based on a score-based divergence.
result BaM converges exponentially quickly to the target mean and covariance.
We focus on the maximum regularization parameter for anisotropic total-variation denoising. It corresponds to the minimum value of the regularization parameter above which the solution remains constant. While this value is well know for the Lasso, such a critical value has not been investigated in details for the total…
Paper calculates KL divergence for isotropic Gaussian-Markov fields.
problem Measuring divergence between isotropic Gaussian-Markov fields.
method Derives closed-form KL divergence expressions.
result Develops new similarity measures in image processing.
We develop a method to combine Markov chain Monte Carlo (MCMC) and variational inference (VI), leveraging the advantages of both inference approaches. Specifically, we improve the variational distribution by running a few MCMC steps. To make inference tractable, we introduce the variational contrastive divergence (VCD)…
This paper proposes a more efficient training method for energy-based models.
problem The computational burden and validity trade-off in Contrastive Divergence training.
method Introducing Diffusion Contrastive Divergence (DCD) to replace Langevin dynamics with diffusion processes.
result The proposed DCDs are more computationally efficient and handle gradient terms better than Contrastive Divergence.
EM optimizes tensor density estimation by relaxing -divergence to KL-divergence.
problem Analytical challenges in traditional -divergence optimization for tensor-based density estimation.
method EM algorithm: relaxes optimization to KL-divergence, then applies tensor many-body approximation.
result Flexible modeling of various low-rank structures and their mixtures.
To ensure stability of learning, state-of-the-art generalized policy iteration algorithms augment the policy improvement step with a trust region constraint bounding the information loss. The size of the trust region is commonly determined by the Kullback-Leibler (KL) divergence, which not only captures the notion of d…
AIS algorithm improves heavy-tailed distribution estimation.
problem Inconsistent estimators and slow convergence in AIS for heavy-tailed distributions.
method Adapts Student-t proposal distributions by matching escort moments and minimizing α-divergence.
result Improves estimation accuracy for heavy-tailed distributions.
There has been a growing interest in mutual information measures due to their wide range of applications in Machine Learning and Computer Vision. In this paper, we present a generalized structured regression framework based on Shama-Mittal divergence, a relative entropy measure, which is introduced to the Machine Learn…
Model financial markets using information theory with a single parameter.
problem Capture the complexity of financial markets with a simple model.
method Derive an idealized model based on four information-theoretic assumptions, minimizing surprisal and divergence.
result The model uses squared radial Ornstein-Uhlenbeck processes for state variables and their sums.
Local mass perspective on Bayesian inference
problem Measuring distributional discrepancy in Bayesian inference
method Introducing Mass Index and Regularised Extended KL
result Proving inequalities for comparing local small-ball masses
Unified technique for sequential estimation of convex divergences.
problem Estimating convex divergences between distributions.
method Martingale methods and maximal inequalities for reverse submartingales.
result Valid time-uniform confidence sequences for arbitrary stopping times.
Bayesian surrogate models reduce uncertainty in high-dimensional design optimisation problems.
problem Uncertainty in high-dimensional inputs for complex computational models.
method Variational Bayesian inference for constructing statistical surrogates with Gaussian process priors and KL divergence for approximation.
result The RDVGP surrogate provides accurate and versatile approximations for robust structural optimisation.
The paper presents methods to improve uncertainty calibration in Bayesian Neural Networks.
problem Uncalibrated Bayesian Neural Networks often lead to overconfidence.
method The paper uses alpha-divergences from Information Geometry for calibration.
result Calibration using alpha-divergences provides better uncertainty estimates and is more efficient.
Neural networks estimate statistical divergences with performance guarantees.
problem Estimating statistical divergences with theoretical performance guarantees.
method Parametrizing empirical variational form by a neural network and optimizing over parameter space.
result Established non-asymptotic absolute error bounds for neural estimators of four -divergences.
A study on -GANs proving convergence and estimation guarantees.
problem Analyzing the convergence and estimation guarantees of -GANs.
method Proved a correspondence between -GANs and -divergences, and provided estimation bounds.
result Estimation bounds indicate diverse GAN behavior as a function of .
We study -divergence contraction and its privacy implications.
problem Analyzing privacy in data processing and algorithms.
method Generalizing Dobrushin's coefficient to -divergence and deriving contraction coefficients.
result Local differential privacy can be expressed in terms of -divergence contraction, leading to precise sample size reductions.
Study compares chi-squared divergence and KL-divergence posteriors for PAC-Bayesian bounds.
problem Investigates optimal posteriors for PAC-Bayesian bounds using chi-squared divergence.
method Analyzes bounds for three distance functions, derives FP equations for computation.
result Chi-squared divergence based posteriors have weaker bounds and worse test errors.
The empirical NTK diverges from the NTK in classification problems during overtraining.
problem The divergence of empirical NTK from NTK in classification problems during overtraining.
method Demonstrated strictly positive definiteness of NTKs for FCNs and ResNets. Proved divergence of neural network parameters during training with cross-entropy loss.
result The empirical NTK does not uniformly converge to the NTK across all times on the training samples as the network width increases.
Improved UDA framework using -divergence measures.
problem Addressing distribution shifts in machine learning.
method Refined -divergence-based discrepancy and -domain discrepancy.
result Novel target error and sample complexity bounds.
Generative adversarial networks (GANs) are a family of generative models that do not minimize a single training criterion. Unlike other generative models, the data distribution is learned via a game between a generator (the generative model) and a discriminator (a teacher providing training signal) that each minimize t…
Stochastic variational inference (SVI) plays a key role in Bayesian deep learning. Recently various divergences have been proposed to design the surrogate loss for variational inference. We present a simple upper bound of the evidence as the surrogate loss. This evidence upper bound (EUBO) equals to the log marginal li…
We study 1-parameter families in the space of -invariant, unit volume metrics on a given compact, connected, almost-effective homogeneous space . In particular, we focus on diverging sequences, i.e. which are not contained in any compact subset of , and we prove some structu…
Deep nonlinear models pose a challenge for fitting parameters due to lack of knowledge of the hidden layer and the potentially non-affine relation of the initial and observed layers. In the present work we investigate the use of information theoretic measures such as mutual information and Kullback-Leibler (KL) diverge…
Improved privacy bounds enhance deep learning training efficiency.
problem Enhancing privacy guarantees in deep learning models.
method Deriving optimal DP parameters using -divergences.
result Significantly reduces the number of iterations needed for training deep learning models.
Gaussian mixture models (GMM) are powerful parametric tools with many applications in machine learning and computer vision. Expectation maximization (EM) is the most popular algorithm for estimating the GMM parameters. However, EM guarantees only convergence to a stationary point of the log-likelihood function, which c…
Differentially private statistical inference using -divergence.
problem Achieving differential privacy without altering data generation.
method Sampling from a generalised posterior minimizing -divergence.
result More precise inference with broader applicability.