arXiv research

The paper explores how information geometry impacts classical CR inequalities.

New divergence measures improve KL approximation.

Warped product affects divergences in information geometry.

New measures generalize existing ones, linking information and risk.

New dispersion indices based on inaccuracy and divergence introduced for information measures.

AES uses α-divergence to select informative points for BO, improving optimization performance.

The study explores generalized divergences and exponential families with a focus on sufficient conditions and laws of large numbers.

The paper introduces a new method for estimating optimal policies in dynamic treatment regimes using information geometry.

We unify f-divergences, Bregman divergences, surrogate loss bounds (regret bounds), proper scoring rules, matching losses, cost curves, ROC-curves and information. We do this by systematically studying integral and variational representations of these objects and in so doing identify their primitives which all are rela…

A recently introduced canonical divergence $\mathcal{D}$ for a dual structure $(\mathrm{g},\nabla,\nabla^*)$ is discussed in connection to other divergence functions. Finally, open problems concerning symmetry properties are outlined.

New principle controls graph-informed adversarial discrepancies.

A one-to-one correspondence is drawn between law invariant risk measures and divergences, which we define as functionals of pairs of probability measures on arbitrary standard Borel spaces satisfying a few natural properties. Divergences include many classical information divergence measures, such as relative entropy a…

Optimal transport and information geometry both study geometric structures on spaces of probability distributions. Optimal transport characterizes the cost-minimizing movement from one distribution to another, while information geometry originates from coordinate-invariant properties of statistical inference. Their con…

We review recent results about the maximal values of the Kullback-Leibler information divergence from statistical models defined by neural networks, including naive Bayes models, restricted Boltzmann machines, deep belief networks, and various classes of exponential families. We illustrate approaches to compute the max…

The paper develops sum-of-squares relaxations for computing $f$-divergences.

Automated feature selection is important for text categorization to reduce the feature size and to speed up the learning process of classifiers. In this paper, we present a novel and efficient feature selection framework based on the Information Theory, which aims to rank the features with their discriminative capacity…

Exponential families and mixture families are parametric probability models that can be geometrically studied as smooth statistical manifolds with respect to any statistical divergence like the Kullback-Leibler (KL) divergence or the Hellinger divergence. When equipping a statistical manifold with the KL divergence, th…

In this work, we study generalized entropies and information geometry in a group-theoretical framework. We explore the conditions that ensure the existence of some natural properties and at the same time of a group-theoretical structure for a large class of entropies. In addition, a method for defining new entropies, u…

Paper introduces symmetric divergence link models for probability distributions.

In Riemannian geometry geodesics are integral curves of the Riemannian distance gradient. We extend this classical result to the framework of Information Geometry. In particular, we prove that the rays of level-sets defined by a pseudo-distance are generated by the sum of two tangent vectors. By relying on these vector…

It is well-known that there are a number of relations between theoretical finance theory and information theory. Some of these relations are exact and some are approximate. In this paper we will explore some of these relations and determine under which conditions the relations are exact. It turns out that portfolio the…

Paper studies statistical manifolds with logarithmic divergences.

We study the logarithmic $L^{(α)}$-divergence which extrapolates the Bregman divergence and corresponds to solutions to novel optimal transport problems. We show that this logarithmic divergence is equivalent to a conformal transformation of the Bregman divergence, and, via an explicit affine immersion, is equivalent t…

Researchers study the geometric properties of a specific type of stable processes.

The paper explores geometry of probability measures and barycenter maps.

Develops information geometry for Lévy processes in finance.

Technical report on f-divergences and f-GAN training properties.

Information theoretic measures (e.g. the Kullback Liebler divergence and Shannon mutual information) have been used for exploring possibly nonlinear multivariate dependencies in high dimension. If these dependencies are assumed to follow a Markov factor graph model, this exploration process is called structure discover…

The paper solves portfolio selection using Rényi divergence and optimization.

There has been a growing interest in mutual information measures due to their wide range of applications in Machine Learning and Computer Vision. In this paper, we present a generalized structured regression framework based on Shama-Mittal divergence, a relative entropy measure, which is introduced to the Machine Learn…

We propose a modified $χ^β$-divergence, give some of its properties, and show that this leads to the definition of a generalized Fisher information. We give generalized Cramér-Rao inequalities, involving this Fisher information, an extension of the Fisher information matrix, and arbitrary norms and power of the estimat…

$f$-divergences are a general class of divergences between probability measures which include as special cases many commonly used divergences in probability, mathematical statistics and information theory such as Kullback-Leibler divergence, chi-squared divergence, squared Hellinger distance, total variation distance e…

We investigate connections between information-theoretic and estimation-theoretic quantities in vector Poisson channel models. In particular, we generalize the gradient of mutual information with respect to key system parameters from the scalar to the vector Poisson channel model. We also propose, as another contributi…

The paper explores how information theory aids in statistical learning models.

Rényi divergence is related to Rényi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by Rényi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its or…

The paper presents methods to improve uncertainty calibration in Bayesian Neural Networks.

Information divergence functions play a critical role in statistics and information theory. In this paper we show that a non-parametric f-divergence measure can be used to provide improved bounds on the minimum binary classification probability of error for the case when the training and test data are drawn from the sa…

We study strictly proper scoring rules in the Reproducing Kernel Hilbert Space. We propose a general Kernel Scoring rule and associated Kernel Divergence. We consider conditions under which the Kernel Score is strictly proper. We then demonstrate that the Kernel Score includes the Maximum Mean Discrepancy as a special …

We study $E_γ$-divergence contraction and its privacy implications.

New method uses KL-divergence to create non-informative priors for multivariate Gaussian.

Computing approximate nearest neighbors in high dimensional spaces is a central problem in large-scale data mining with a wide range of applications in machine learning and data science. A popular and effective technique in computing nearest neighbors approximately is the locality-sensitive hashing (LSH) scheme. In thi…

We study the geometry of probability distributions with respect to a generalized family of Csiszár $f$-divergences. A member of this family is the relative $α$-entropy which is also a Rényi analog of relative entropy in information theory and known as logarithmic or projective power divergence in statistics. We apply E…

Model financial markets using information theory with a single parameter.

Information geometry offers new tools for statistical analysis.

RényiCL uses Rényi divergence for robust contrastive learning with stronger data augmentations.

Develops a new divergence framework that combines $f$-divergences and IPMs.

We describe Information Forests, an approach to classification that generalizes Random Forests by replacing the splitting criterion of non-leaf nodes from a discriminative one -- based on the entropy of the label distribution -- to a generative one -- based on maximizing the information divergence between the class-con…

In this paper, we derive a useful lower bound for the Kullback-Leibler divergence (KL-divergence) based on the Hammersley-Chapman-Robbins bound (HCRB). The HCRB states that the variance of an estimator is bounded from below by the Chi-square divergence and the expectation value of the estimator. By using the relation b…