Paper describes profiles of multivariate normal distributions and novel estimators for mutual information.
problem Estimating mutual information for complex distributions.
method Analytical description of profiles, introduction of Bend and Mix Models, Monte Carlo estimation.
result Bend and Mix Models accurately estimate mutual information profiles and provide Bayesian estimates.
Neural estimator improves mutual information estimation in high dimensions.
problem Estimating mutual information in high dimensions is challenging.
method Parametrizing conditional densities with normalizing flows and using block autoregressive structure.
result Improved mutual information estimation on benchmark tasks.
Paper benchmarks mutual info estimators on diverse distributions.
problem Evaluating mutual information estimators on complex, real-world distributions.
method Constructs a diverse family of known-ground truth distributions, proposes a benchmark platform.
result Highlights differences in classical and neural estimators' performance across various conditions.
New estimator improves mutual information estimation.
problem Estimating mutual information in data science and machine learning.
method Proposes a new estimator that uses a preliminary estimate of the data distribution.
result A preliminary estimate helps in estimating mutual information more accurately.
A new method estimates mutual information using neural classifiers.
problem Estimating mutual information for high-dimensional data is challenging.
method Trains a classifier to estimate joint distribution probability.
result Demonstrates high accuracy and reduces variance compared to variational methods.
InfoBridge uses diffusion bridges to estimate mutual information accurately.
problem Estimating mutual information between random variables.
method Formulated mutual information estimation as a domain transfer problem using diffusion bridge models.
result Demonstrated unbiased estimator for various data types.
New method estimates mutual information using normalizing flows.
problem Mutual information estimation in high-dimensional data.
method Normalizing flows to map data to target distributions with known MI.
result Theoretical guarantees and practical advantages demonstrated.
Estimates conditional mutual information using a minmax formulation.
problem Estimating conditional mutual information in high dimensions.
method Uses a minmax optimization problem to train a neural network.
result Improves estimation accuracy compared to existing methods.
Hybrid method improves mutual information estimation from samples.
problem Estimating mutual information from joint distributions is challenging.
method Proposes a hybrid method combining discriminative and generative approaches.
result Hybrid methods yield tighter variational bounds on mutual information.
This paper improves DNN generalization by accurately estimating mutual information.
problem Intractability of estimating mutual information in DNNs.
method Introduces a probabilistic representation of DNNs to accurately estimate mutual information.
result Derives a tighter generalization bound than previous relaxations.
Deep learning model estimates mutual information with low bias and variance.
problem Estimating mutual information between continuous variables.
method Supervised deep learning approach using Linfoot informational correlation as labels.
result Lower bias and variance compared to other methods.
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, an…
Novel mutual information bound improves statistical inference rates.
problem Improving statistical inference rates in Bayesian nonparametrics.
method Introduces a novel mutual information bound.
result Improved contraction rates for fractional posteriors.
Several methods of estimating the mutual information of random variables have been developed in recent years. They can prove valuable for novel approaches to learning statistically independent features. In this paper, we use one of these methods, a mutual information neural estimation (MINE) network, to present a proof…
Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confide…
Proposes MIGE for accurate MI gradient estimation in high-dimensional settings.
problem Intractability of MI in continuous and high-dimensional settings.
method Score estimation of implicit distributions for gradient estimation of MI.
result MIGE provides tight and smooth gradient estimation of MI in high-dimensional settings.
Review of information plane analyses in neural networks, highlighting mixed results and methodological challenges.
problem Understanding the relationship between information-theoretic compression and neural network performance.
method Literature review and detailed analysis of information quantity estimation methods.
result Information plane compression is not necessarily information-theoretic but compatible with geometric compression.
Exploration is a difficult challenge in reinforcement learning and is of prime importance in sparse reward environments. However, many of the state of the art deep reinforcement learning algorithms, that rely on epsilon-greedy, fail on these environments. In such cases, empowerment can serve as an intrinsic reward sign…
The conditional mutual information I(X;Y|Z) measures the average information that X and Y contain about each other given Z. This is an important primitive in many learning problems including conditional independence testing, graphical model inference, causal strength estimation and time-series problems. In several appl…
A method to improve image synthesis diversity using mutual information.
problem Mode collapse in conditional GANs for multimodal image synthesis.
method Explicitly estimate and maximize mutual information between latent code and output image.
result Prevents mode collapse and encourages synthesis of diverse images.
Reshef et al. recently proposed a new statistical measure, the "maximal information coefficient" (MIC), for quantifying arbitrary dependencies between pairs of stochastic quantities. MIC is based on mutual information, a fundamental quantity in information theory that is widely understood to serve this need. MIC, howev…
A neural network approach for feature selection using mutual information.
problem Feature ranking and selection leading to sub-optimal solutions for class separability.
method Stochastic mutual information gradient estimation for dimensionality reduction.
result The network projects features onto an output space maximizing mutual information with class labels.
New method estimates spin system mutual information using neural networks.
problem Estimating mutual information in spin systems.
method Monte Carlo sampling enhanced by autoregressive neural networks.
result Area law satisfied for temperatures away from critical temperature.
Mutual information is widely applied to learn latent representations of observations, whilst its implication in classification neural networks remain to be better explained. We show that optimising the parameters of classification neural networks with softmax cross-entropy is equivalent to maximising the mutual informa…
Multivariate pattern analyses approaches in neuroimaging are fundamentally concerned with investigating the quantity and type of information processed by various regions of the human brain; typically, estimates of classification accuracy are used to quantify information. While a extensive and powerful library of method…
Estimating mutual information is an important statistics and machine learning problem. To estimate the mutual information from data, a common practice is preparing a set of paired samples {(xi,yi)}i=1n∼i.i.d.p(x,y). However, in many situations, it…
A new multi-label CPC method improves mutual information estimation and representation learning.
problem Underestimation of mutual information in contrastive predictive coding.
method Introducing a multi-label classification problem to overcome the logm bound in mutual information estimation. result The new method exceeds the logm bound and leads to better mutual information estimation and improved unsupervised representation learning. Estimates latent dimensionality for prediction tasks using mutual information.
problem Estimating the latent dimensionality needed for accurate prediction.
method Formulates the problem as an Information Bottleneck question and uses neural mutual information estimators with a hybrid critic to preserve latent geometry.
result The hybrid critic method provides a more accurate estimation of task-relevant dimensionality.
MINIMALIST maximizes mutual information for likelihood estimation from simulated data.
problem Learning model parameters from likelihood functions that cannot be computed.
method Maximizes mutual information between simulated data and model parameters using neural networks.
result Different methods aiming at the same optimal energy form can be directly benchmarked.
The paper proposes a method to learn structured representations from unlabeled data using mutual information maximization.
problem Learning structured representations from unlabeled data.
method Adversarial maximization of mutual information between a structured latent variable and a target variable.
result The proposed method outperforms current baselines in document hashing and yields highly compressed interpretable representations.
In this paper, we suggest a framework to make use of mutual information as a regularization criterion to train Auto-Encoders (AEs). In the proposed framework, AEs are regularized by minimization of the mutual information between input and encoding variables of AEs during the training phase. In order to estimate the ent…
KSG mutual information estimator, which is based on the distances of each sample to its k-th nearest neighbor, is widely used to estimate mutual information between two continuous random variables. Existing work has analyzed the convergence rate of this estimator for random variables whose densities are bounded away fr…
In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex lo…
MINDE estimates Mutual Information using neural diffusion models.
problem Estimating Mutual Information between random variables.
method Score-based diffusion models to estimate Kullback Leibler divergence.
result MINDE is more accurate than existing methods, especially for challenging distributions.
Paper refines InfoNCE for accurate mutual information estimation.
problem Indirect connection of InfoNCE to mutual information estimation.
method Introduces InfoNCE-anchor with an auxiliary anchor class for consistent density ratio estimation.
result InfoNCE-anchor yields a plug-in MI estimator with significantly reduced bias.
The identification of relevant features, i.e., the driving variables that determine a process or the properties of a system, is an essential part of the analysis of data sets with a large number of variables. A mathematical rigorous approach to quantifying the relevance of these features is mutual information. Mutual i…
In this paper, we investigate the problem of learning disentangled representations. Given a pair of images sharing some attributes, we aim to create a low-dimensional representation which is split into two parts: a shared representation that captures the common information between the images and an exclusive representa…
Framework generates multimodal datasets with known MI for benchmarking.
problem Benchmarking mutual information estimators and SSL techniques.
method Flow-based generative model and structured causal framework.
result Regression performance improves with increasing MI between modalities.
We study the problem of using i.i.d. samples from an unknown multivariate probability distribution p to estimate the mutual information of p. This problem has recently received attention in two settings: (1) where p is assumed to be Gaussian and (2) where p is assumed only to lie in a large nonparametric smooth…
Improved Monte-Carlo models by constraining mutual information between latent and observable variables.
problem Training density models leads to latent variables being useless.
method Weave tighter Monte-Carlo bounds with mutual information constraints.
result Improved training of models with continuous and discrete latent variables.
New method removes unwanted information from representations efficiently.
problem Learning representations that are uninformative about a target variable.
method Adversarial training with a novel proxy metric for mutual information, leading to an analytically computable approximation.
result Our method effectively removes unwanted information with limited time budget.
Proposes a robust VIB approach using soft labels and mutual info estimation.
problem Improving robustness of VIB to adversarial perturbations.
method Refines categorical class information with soft labels from a reference network, relaxes Gaussian posterior assumption.
result Significantly outperforms benchmarked models on MNIST and CIFAR-10.
Proposes CLUB for reliable MI minimization in high dimensions.
problem Estimating and minimizing mutual information in high-dimensional spaces.
method Contrastive Log-ratio Upper Bound (CLUB) for MI minimization.
result CLUB provides reliable estimation of mutual information.
The richness in the content of various information networks such as social networks and communication networks provides the unprecedented potential for learning high-quality expressive representations without external supervision. This paper investigates how to preserve and extract the abundant information from graph-s…
Estimating entropy and mutual information consistently is important for many machine learning applications. The Kozachenko-Leonenko (KL) estimator (Kozachenko & Leonenko, 1987) is a widely used nonparametric estimator for the entropy of multivariate continuous random variables, as well as the basis of the mutual inform…
Tensor networks reveal limitations for efficient text description but suggest potential for images.
problem Efficiently describing large text and image data sets using tensor networks.
method Investigation of mutual information scaling, introduction of mutual information estimators, and use of autoregressive and convolutional neural networks.
result Text data cannot be efficiently described by 1D tensor networks, while images may be better described by 2D tensor networks.
A new contrastive MI estimator improves efficiency and tightness.
problem Efficient and tight mutual information estimation.
method Contrastive Fenchel-Legendre optimization.
result The FLO estimator is tight and converges under stochastic gradient descent.
LMI approximates mutual information in high dimensions using learned low-dimensional representations.
problem Estimating mutual information between high-dimensional variables is challenging due to sample size limitations.
method Developed a method called latent MI (LMI) approximation that applies a nonparametric MI estimator to low-dimensional representations learned by a simple model architecture.
result LMI can approximate MI well for variables with >10^3 dimensions if their dependence structure has low intrinsic dimensionality.