Neural network estimates mutual information for ICA.
problem Learning statistically independent features from noisy data.
method Minimizing mutual information estimated by a MINE network in a differentiable encoder network.
result Qualitatively equal solution to FastICA for blind-source-separation.
Neural estimator improves mutual information estimation in high dimensions.
problem Estimating mutual information in high dimensions is challenging.
method Parametrizing conditional densities with normalizing flows and using block autoregressive structure.
result Improved mutual information estimation on benchmark tasks.
Paper benchmarks mutual info estimators on diverse distributions.
problem Evaluating mutual information estimators on complex, real-world distributions.
method Constructs a diverse family of known-ground truth distributions, proposes a benchmark platform.
result Highlights differences in classical and neural estimators' performance across various conditions.
A new method estimates mutual information using neural classifiers.
problem Estimating mutual information for high-dimensional data is challenging.
method Trains a classifier to estimate joint distribution probability.
result Demonstrates high accuracy and reduces variance compared to variational methods.
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, an…
Softmax cross-entropy optimizes mutual information in neural networks.
problem Understanding the relationship between mutual information and classification neural networks.
method Demonstrated that optimizing softmax cross-entropy maximizes mutual information between inputs and labels.
result Softmax cross-entropy can approximate mutual information and highlight relevant image regions.
Review of information plane analyses in neural networks, highlighting mixed results and methodological challenges.
problem Understanding the relationship between information-theoretic compression and neural network performance.
method Literature review and detailed analysis of information quantity estimation methods.
result Information plane compression is not necessarily information-theoretic but compatible with geometric compression.
This paper improves DNN generalization by accurately estimating mutual information.
problem Intractability of estimating mutual information in DNNs.
method Introduces a probabilistic representation of DNNs to accurately estimate mutual information.
result Derives a tighter generalization bound than previous relaxations.
Paper describes profiles of multivariate normal distributions and novel estimators for mutual information.
problem Estimating mutual information for complex distributions.
method Analytical description of profiles, introduction of Bend and Mix Models, Monte Carlo estimation.
result Bend and Mix Models accurately estimate mutual information profiles and provide Bayesian estimates.
Estimates conditional mutual information using a minmax formulation.
problem Estimating conditional mutual information in high dimensions.
method Uses a minmax optimization problem to train a neural network.
result Improves estimation accuracy compared to existing methods.
New method estimates spin system mutual information using neural networks.
problem Estimating mutual information in spin systems.
method Monte Carlo sampling enhanced by autoregressive neural networks.
result Area law satisfied for temperatures away from critical temperature.
New estimator reduces bias and variance issues in mutual information estimation.
problem Difficulty in using variational MI estimators due to bias/variance tradeoffs and self-consistency issues.
method Developed a new estimator based on a unified perspective of variational approaches, focusing on variance reduction.
result Empirical results show improved bias-variance trade-offs compared to existing estimators.
A neural network approach for feature selection using mutual information.
problem Feature ranking and selection leading to sub-optimal solutions for class separability.
method Stochastic mutual information gradient estimation for dimensionality reduction.
result The network projects features onto an output space maximizing mutual information with class labels.
Deep learning model estimates mutual information with low bias and variance.
problem Estimating mutual information between continuous variables.
method Supervised deep learning approach using Linfoot informational correlation as labels.
result Lower bias and variance compared to other methods.
Improves data efficiency for estimating mutual information.
problem Estimating mutual information from limited data.
method Developed a Data-Efficient MINE Estimator (DEMINE) and Meta-DEMINE.
result Significantly improved data efficiency in estimating mutual information.
Estimates latent dimensionality for prediction tasks using mutual information.
problem Estimating the latent dimensionality needed for accurate prediction.
method Formulates the problem as an Information Bottleneck question and uses neural mutual information estimators with a hybrid critic to preserve latent geometry.
result The hybrid critic method provides a more accurate estimation of task-relevant dimensionality.
New estimator improves reliability of KL divergence estimation.
problem Estimating KL divergence reliably and efficiently.
method Proposes a new estimator using Reproducing Kernel Hilbert Space.
result Proposed estimator is consistent and more reliable for small datasets.
Exploration is a difficult challenge in reinforcement learning and is of prime importance in sparse reward environments. However, many of the state of the art deep reinforcement learning algorithms, that rely on epsilon-greedy, fail on these environments. In such cases, empowerment can serve as an intrinsic reward sign…
Tensor networks reveal limitations for efficient text description but suggest potential for images.
problem Efficiently describing large text and image data sets using tensor networks.
method Investigation of mutual information scaling, introduction of mutual information estimators, and use of autoregressive and convolutional neural networks.
result Text data cannot be efficiently described by 1D tensor networks, while images may be better described by 2D tensor networks.
Study introduces a benchmark suite for evaluating neural MI estimators on real-world unstructured datasets.
problem Lack of comprehensive evaluation methods for neural MI estimators on real-world unstructured datasets.
method Developed a benchmark suite using same-class sampling and a binary symmetric channel trick.
result Showed accurate manipulation of true MI values of real-world datasets.
New neural network approach using mutual information.
problem Training neural networks for imbalanced datasets.
method Converts neural network classifiers to mutual information evaluators.
result New form of softmax leads to better classification accuracy, especially for imbalanced datasets.
MINIMALIST maximizes mutual information for likelihood estimation from simulated data.
problem Learning model parameters from likelihood functions that cannot be computed.
method Maximizes mutual information between simulated data and model parameters using neural networks.
result Different methods aiming at the same optimal energy form can be directly benchmarked.
Estimates copula density for complex data distributions.
problem Estimating copula density from observed data.
method Neural network-based copula density neural estimation (CODINE).
result Novel approach capable of modeling complex distributions.
MINDE estimates Mutual Information using neural diffusion models.
problem Estimating Mutual Information between random variables.
method Score-based diffusion models to estimate Kullback Leibler divergence.
result MINDE is more accurate than existing methods, especially for challenging distributions.
Proposes a robust VIB approach using soft labels and mutual info estimation.
problem Improving robustness of VIB to adversarial perturbations.
method Refines categorical class information with soft labels from a reference network, relaxes Gaussian posterior assumption.
result Significantly outperforms benchmarked models on MNIST and CIFAR-10.
New method removes unwanted information from representations efficiently.
problem Learning representations that are uninformative about a target variable.
method Adversarial training with a novel proxy metric for mutual information, leading to an analytically computable approximation.
result Our method effectively removes unwanted information with limited time budget.
Unified variational bounds for mutual information, addressing high-dimensional challenges.
problem Estimating and optimizing Mutual Information (MI) in high dimensions is challenging.
method Unified framework of variational lower bounds parameterized by neural networks, trading off bias and variance.
result Unified bounds flexibly trade off bias and variance, improving estimation and representation learning.
Proposes a new method to measure epistemic uncertainty in Bayesian neural networks.
problem Measuring epistemic uncertainty in Bayesian neural networks for out-of-distribution detection.
method Proposes measuring disagreement between logits and their pre-softmax counterparts as an epistemic uncertainty measure.
result Proposed epistemic uncertainty scores outperform mutual information and equal predictive entropy performance.
Novel approach uses neural networks to enhance CI testing for feature selection.
problem Challenges in implementing Markov blanket feature selection due to CI testing limitations.
method Two-step approach: feature mapping followed by CI testing using k-NN. result The method boosts CI testing performance, leading to improved feature selection.
InfoAtlas speeds up MI estimation for real-time data analysis.
problem Efficiently measuring statistical dependency between high-dimensional datasets.
method Directly infers mutual information in a single forward pass using a pretrained model.
result Matches state-of-the-art accuracy with 100x speedup.
A method to improve image synthesis diversity using mutual information.
problem Mode collapse in conditional GANs for multimodal image synthesis.
method Explicitly estimate and maximize mutual information between latent code and output image.
result Prevents mode collapse and encourages synthesis of diverse images.
This paper proposes a method to learn graph representations without supervision.
problem Learning high-quality graph representations without external supervision.
method Graphical Mutual Information (GMI) to measure graph and hidden representation correlation.
result The proposed method outperforms state-of-the-art unsupervised counterparts and sometimes supervised ones.
We find the maximum mutual information for neural networks and its key determinants.
problem Understanding the maximum mutual information in neural architectures.
method Derived closed-form expression for maximum mutual information across neural network families.
result Maximum mutual information stems from a generalized formula and is influenced by network width and statistical invariances.
Optimizes experimental designs for intractable models using mutual information bounds.
problem Finding optimal experimental designs for models with intractable data-generating distributions.
method Maximizes mutual information lower bounds parametrized by neural networks, updating network parameters and designs simultaneously.
result Framework enables experimental design for various tasks including parameter estimation and model discrimination.
New estimator improves mutual information estimation.
problem Estimating mutual information in data science and machine learning.
method Proposes a new estimator that uses a preliminary estimate of the data distribution.
result A preliminary estimate helps in estimating mutual information more accurately.
This study quantifies the scalability of k-Sliced Mutual Information (k-SMI) with dimension.
problem Understanding how SMI and its estimation rates depend on the ambient dimension.
method Developed k-SMI framework and derived bounds on MC estimates, established optimal convergence rates, and provided asymptotic results.
result Sharp bounds and optimal convergence rates for k-SMI estimation, revealing interplay with dimension and sample size.
New methods estimate point-wise dependency from neural MI models.
problem Estimating point-wise dependency between different events.
method Developed two methods: Probabilistic Classifier and Density-Ratio Fitting.
result Demonstrated effectiveness in MI estimation, self-supervised representation learning, and cross-modal retrieval.
Softmax emerges naturally in neural networks as a measure of conditional mutual information.
problem The artificial nature of softmax in neural networks.
method Information-theoretic perspective to derive log-softmax and evaluate conditional mutual information.
result Training deterministic neural networks through log-softmax maximises conditional mutual information.
InfoBridge uses diffusion bridges to estimate mutual information accurately.
problem Estimating mutual information between random variables.
method Formulated mutual information estimation as a domain transfer problem using diffusion bridge models.
result Demonstrated unbiased estimator for various data types.
New protocol makes neural MI estimators reliable in high-dimensional data.
problem Accurate estimation of mutual information in high-dimensional, undersampled data.
method Developed a practical protocol for neural MI estimators, incorporating statistical consistency checks, bias correction, and confidence intervals.
result Neural MI estimators can be made reliable when dependencies admit a low-dimensional latent representation.
New method estimates mutual information using normalizing flows.
problem Mutual information estimation in high-dimensional data.
method Normalizing flows to map data to target distributions with known MI.
result Theoretical guarantees and practical advantages demonstrated.
Neural networks help create summary statistics for complex models.
problem Creating summary statistics for models with intractable likelihood functions.
method Infomax learning with neural networks to maximize mutual information.
result Improves performance of approximate Bayesian computation and neural likelihood methods.
Maximizes mutual information to improve graph neural networks performance.
problem Loss of information between nodes in GNNs aggregation and iteration schemes.
method Explores mutual information maximization in the aggregation and iteration scheme of GNNs.
result Improves state-of-the-art performance on graph tasks.
Hybrid method improves mutual information estimation from samples.
problem Estimating mutual information from joint distributions is challenging.
method Proposes a hybrid method combining discriminative and generative approaches.
result Hybrid methods yield tighter variational bounds on mutual information.
A new differential entropy estimator for neural networks training.
problem Lack of effective differential entropy estimators for neural network training.
method KNIFE: a fully parameterized, differentiable kernel-based estimator of differential entropy.
result KNIFE effectively estimates differential entropy and improves neural network training.
In this paper, we present a new approach to interpret deep learning models. By coupling mutual information with network science, we explore how information flows through feedforward networks. We show that efficiently approximating mutual information allows us to create an information measure that quantifies how much in…
New method uses neural networks to optimize experimental designs for complex models.
problem Designing experiments for complex, intractable models with high computational cost.
method Neural mutual information estimation for mutual information maximization.
result Optimal experimental designs and posterior inference can be jointly determined.
New bounds improve neural network generalization through slicing.
problem Difficulty in evaluating mutual information in high dimensions for neural networks.
method Slicing the parameter space and using disintegrated mutual information and k-sliced mutual information.
result Slicing improves generalization and offers significant computational and statistical advantages.