arXiv research

Gibbs sampler contracts entropy under strong log-concavity, improving mixing time.

We prove a generalization of the fundamental inequality of Guivarc'h relating entropy, drift and critical exponent to Gibbs measures on geometrically finite quotients of CAT(-1) metric spaces. For random walks with finite superexponential moment, we show that the equality is achieved if and only if the Gibbs density is…

Modified Gibbs-Helmholtz equation geometric models for thermodynamics.

The paper extends entropy maximization to multiscale settings and applies it to neural networks.

Proposes a link between randomness and compression in deep learning.

Decentralized learning achieves centralized performance via Gibbs measures.

The study examines entropy and pressure at infinity in negatively curved manifolds, linking them to strong positive recurrence.

Study risk-sensitive market making with entropy regularization for better quote control.

We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier obtained by a risk-sensitive perturbation of the weights of a learned classifier. Entropy-SGD works by optimizing the bound's p…

We prove the eventological $H$-theorem that complements the Boltzmann H-theorem from statistical mechanics and serves as a mathematical excuse (mathematically no less convincing than the Boltzmann H-theorem for the second law of thermodynamics) for what can be called "the second law of eventology", which justifies the …

Paper introduces a new measure combining entropy and Gini index.

PAC-Bayesian bounds for MLPs with cross entropy loss validated.

Novel analysis of EFP for finite-sum problems in neural networks.

PAC-Bayes bounds for Gibbs posteriors derived via singular learning theory.

In this article we consider macrocanonical models for texture synthesis. In these models samples are generated given an input texture image and a set of features which should be matched in expectation. It is known that if the images are quantized, macrocanonical models are given by Gibbs measures, using the maximum ent…

We derive generalization and excess risk bounds for neural nets using a family of complexity measures based on a multilevel relative entropy. The bounds are obtained by introducing the notion of generated hierarchical coverings of neural nets and by using the technique of chaining mutual information introduced in Asadi…

Bayesian model predicts circular data with fast Gibbs sampling.

In this paper, we present a new class of Markov decision processes (MDPs), called Tsallis MDPs, with Tsallis entropy maximization, which generalizes existing maximum entropy reinforcement learning (RL). A Tsallis MDP provides a unified framework for the original RL problem and RL with various types of entropy, includin…

We study approximations of non-Gaussian stationary processes having long range correlations with microcanonical models. These models are conditioned by the empirical value of an energy vector, evaluated on a single realization. Asymptotic properties of maximum entropy microcanonical and macrocanonical processes and the…

In this paper, we propose a novel maximum causal Tsallis entropy (MCTE) framework for imitation learning which can efficiently learn a sparse multi-modal policy distribution from demonstrations. We provide the full mathematical analysis of the proposed framework. First, the optimal solution of an MCTE problem is shown …

The cornerstone of Boltzmann-Gibbs ($BG$) statistical mechanics is the Boltzmann-Gibbs-Jaynes-Shannon entropy $S_{BG} \equiv -k\int dx f(x)\ln f(x)$, where $k$ is a positive constant and $f(x)$ a probability density function. This theory has exibited, along more than one century, great success in the treatment of syste…

Statistical mechanics explains income and wealth distribution in developed economies.

This review explores entropy applications in data analysis and machine learning.

Study non-negative curvature Markov chains, proving entropy contraction.

The Bregman divergence (Bregman distance, Bregman measure of distance) is a certain useful substitute for a distance, obtained from a well-chosen function (the "Bregman function"). Bregman functions and divergences have been extensively investigated during the last decades and have found applications in optimization, o…

We define On-Average KL-Privacy and present its properties and connections to differential privacy, generalization and information-theoretic quantities including max-information and mutual information. The new definition significantly weakens differential privacy, while preserving its minimalistic design features such …

Geometric study of thermodynamics using cotangent bundles.

McDiarmid's inequality under dependence via approximate tensorization of entropy

This paper presents a new approach, called perturb-max, for high-dimensional statistical inference that is based on applying random perturbations followed by optimization. This framework injects randomness to maximum a-posteriori (MAP) predictors by randomly perturbing the potential function for the input. A classic re…

A plethora of natural, artificial and social systems exist which do not belong to the Boltzmann-Gibbs (BG) statistical-mechanical world, based on the standard additive entropy $S_{BG}$ and its associated exponential BG factor. Frequent behaviors in such complex systems have been shown to be closely related to $q$-stati…

The Gibbs sampler is one of the most popular algorithms for inference in statistical models. In this paper, we introduce a herding variant of this algorithm, called herded Gibbs, that is entirely deterministic. We prove that herded Gibbs has an $O(1/T)$ convergence rate for models with independent variables and for ful…

Gibbs sampling is a Markov chain Monte Carlo method that is often used for learning and inference on graphical models. Minibatching, in which a small random subset of the graph is used at each iteration, can help make Gibbs sampling scale to large graphical models by reducing its computational cost. In this paper, we p…

Study speculative trading using RL with exploratory framework.

We prove a large deviation principle for a sequence of point processes defined by Gibbs probability measures on a Polish space. This is obtained as a consequence of a more general Laplace principle for the non-normalized Gibbs measures. We consider three main applications: Conditional Gibbs measures on compact spaces, …

We develop a framework for approximating collapsed Gibbs sampling in generative latent variable cluster models. Collapsed Gibbs is a popular MCMC method, which integrates out variables in the posterior to improve mixing. Unfortunately for many complex models, integrating out these variables is either analytically or co…

The pairwise influence matrix of Dobrushin has long been used as an analytical tool to bound the rate of convergence of Gibbs sampling. In this work, we use Dobrushin influence as the basis of a practical tool to certify and efficiently improve the quality of a discrete Gibbs sampler. Our Dobrushin-optimized Gibbs samp…

New decision-theoretic characterization separates belief and decision posteriors.

Personal income distribution in the USA has a well-defined two-class structure. The majority of population (97-99%) belongs to the lower class characterized by the exponential Boltzmann-Gibbs ("thermal") distribution, whereas the upper class (1-3% of population) has a Pareto power-law ("superthermal") distribution. By …

We review a simple model of closed economy, where the economic agents make money transactions and a saving criterion is present. We observe the Gibbs distribution for zero saving propensity, and non-Gibbs distributions otherwise. While the exact solution in the case of zero saving propensity is already known to be give…

For large scale on-line inference problems the update strategy is critical for performance. We derive an adaptive scan Gibbs sampler that optimizes the update frequency by selecting an optimum mini-batch size. We demonstrate performance of our adaptive batch-size Gibbs sampler by comparing it against the collapsed Gibb…

New Gibbs sampling method improves MCMC efficiency.

The notion of Berman-Gibbs stability was originally introduced by Robert Berman for $\mathbb{Q}$-Fano varieties $X$. We show that the pair $(X, -K_X)$ is K-stable (resp. K-semistable) provided that $X$ is Berman-Gibbs stable (resp. semistable).

Souriau studies Gibbs states for symplectic manifolds with group actions.

New model estimates Gibbs free energies using machine learning and isobaric-isothermal flows.

Study on Metropolis-within-Gibbs schemes for high-dimensional Bayesian models.

Introduces HMC method for sampling Gibbs densities.

New method improves uncertainty quantification in latent variable models.

DiGS improves sampling from multi-modal distributions.