arXiv research

New insights into natural exponential families improve regret bounds for bandit problems.

Exponential family distributions are highly useful in machine learning since their calculation can be performed efficiently through natural parameters. The exponential family has recently been extended to the t-exponential family, which contains Student-t distributions as family members and thus allows us to handle noi…

The paper proposes a method to train time-varying generative models using natural gradients.

We study online learning under logarithmic loss with regular parametric models. Hedayati and Bartlett (2012b) showed that a Bayesian prediction strategy with Jeffreys prior and sequential normalized maximum likelihood (SNML) coincide and are optimal if and only if the latter is exchangeable, and if and only if the opti…

Efficiently learns exponential family distributions with i.i.d. samples.

Exponential family extensions of principal component analysis (EPCA) have received a considerable amount of attention in recent years, demonstrating the growing need for basic modeling tools that do not assume the squared loss or Gaussian distribution. We extend the EPCA model toolbox by presenting the first exponentia…

NatPN provides fast, accurate uncertainty estimation for exponential family distributions.

A nonparametric family of conditional distributions is introduced, which generalizes conditional exponential families using functional parameters in a suitable RKHS. An algorithm is provided for learning the generalized natural parameter, and consistency of the estimator is established in the well specified case. In ex…

Exponential families are a particular class of statistical manifolds which are particularly important in statistical inference, and which appear very frequently in statistics. For example, the set of normal distributions, with mean μ and deviation σ, form a 2-dimensional exponential family. In this paper, we show that …

We propose a robust estimator to improve maximum likelihood in probabilistic models.

We consider the design of prediction market mechanisms known as automated market makers. We show that we can design these mechanisms via the mold of \emph{exponential family distributions}, a popular and well-studied probability distribution template used in statistics. We give a full development of this relationship a…

Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to \emph{minimal} exponential-family (EF) approximations. In this paper, we extend their application to estimate \emph{structured} approximations such as mixtures of E…

EFDA extends LDA to non-Gaussian models using exponential families.

We investigate penalized maximum log-likelihood estimation for exponential family distributions whose natural parameter resides in a reproducing kernel Hilbert space. Key to our approach is a novel technique, doubly dual embedding, that avoids computation of the partition function. This technique also allows the develo…

Chentsov's theorem characterizes the Fisher information metric on statistical models as essentially the only Riemannian metric that is invariant under sufficient statistics. This implies that each statistical model is naturally equipped with a geometry, so Chentsov's theorem explains why many statistical properties can…

Exponential family plays an important role in information geometry. In arXiv:1811.01394, we introduced a method to construct an exponential family $\mathcal{P}=\{p_θ\}_{θ\inΘ}$ on a homogeneous space $G/H$ from a pair $(V,v_0)$. Here $V$ is a representation of $G$ and $v_0$ is an $H$-fixed vector in $V$. Then the follo…

Information geometry applies concepts in differential geometry to probability and statistics and is especially useful for parameter estimation in exponential families where parameters are known to lie on a Riemannian manifold. Connections between the geometric properties of the induced manifold and statistical properti…

Introduces Legendre bundle for dually flat manifolds and quantum field theories.

New method for fast inference in diffusion models.

Recently much attention has been paid to deep generative models, since they have been used to great success for variational inference, generation of complex data types, and more. In most all of these settings, the goal has been to find a particular member of that model family: optimized parameters index a distribution …

The paper introduces natural α-embeddings for item representations.

A method for converting NIW parameters for better estimation.

Paper introduces kernel deformed exponential families for sparse continuous attention.

Paper studies statistical manifolds with logarithmic divergences.

Exponential families and mixture families are parametric probability models that can be geometrically studied as smooth statistical manifolds with respect to any statistical divergence like the Kullback-Leibler (KL) divergence or the Hellinger divergence. When equipping a statistical manifold with the KL divergence, th…

New convergence results for NGVI with various step sizes and sample sizes.

QBVI uses natural gradients for efficient Bayesian learning.

A new black-box optimizer using implicit natural gradient.

The study explores generalized divergences and exponential families with a focus on sufficient conditions and laws of large numbers.

Generalizes moment-matching for exponential families with conditioning or hidden data.

Correspondence found between exponential families and affine Grassmannians.

We consider three different approaches to define natural Riemannian metrics on polytopes of stochastic matrices. First, we define a natural class of stochastic maps between these polytopes and give a metric characterization of Chentsov type in terms of invariance with respect to these maps. Second, we consider the Fish…

We provide a classification of graphical models according to their representation as subfamilies of exponential families. Undirected graphical models with no hidden variables are linear exponential families (LEFs), directed acyclic graphical models and chain graphs with no hidden variables, including Bayesian networks …

Constructing exponential families from statistical manifolds.

New research shows fixed-budget best-arm identification cannot match static oracle performance.

Thompson Sampling has been demonstrated in many complex bandit models, however the theoretical guarantees available for the parametric multi-armed bandit are still limited to the Bernoulli case. Here we extend them by proving asymptotic optimality of the algorithm using the Jeffreys prior for 1-dimensional exponential …

New Thompson sampling algorithm reduces regret for exponential family bandits.

Moment polytope of toric exponential families is a projection of a simplex.

In this paper we investigate the family of functions representable by deep neural networks (DNN) with rectified linear units (ReLU). We give an algorithm to train a ReLU DNN with one hidden layer to *global optimality* with runtime polynomial in the data size albeit exponential in the input dimension. Further, we impro…

We propose a novel approach for density estimation with exponential families for the case when the true density may not fall within the chosen family. Our approach augments the sufficient statistics with features designed to accumulate probability mass in the neighborhood of the observed points, resulting in a non-para…

The versatility of exponential families, along with their attendant convexity properties, make them a popular and effective statistical model. A central issue is learning these models in high-dimensions, such as when there is some sparsity pattern of the optimal parameter. This work characterizes a certain strong conve…

Extends likelihood ratio exponential families to analyze various optimization methods.

Researchers improve NCE by addressing its flat loss landscape issues.

The paper introduces a new method for graph embedding using exponential family distributions.

Efficient method for learning continuous exponential families beyond Gaussian.

Score matching offers efficient estimation for certain distributions.

This paper shows any Kähler metric can be a Fisher information metric.

New tensor framework connects Fisher information, hypergraphs, and multi-observable correlations.