MIM learns useful representations with high mutual information.
problem Learning useful representations for downstream tasks.
method Symmetric Jensen-Shannon divergence and mutual information regularizer in an encoder/decoder framework.
result MIM learns high mutual information representations without posterior collapse.
Dynamic learning rate set using Mutual Information for neural networks.
problem Optimizing learning rate in deep neural networks.
method Mutual Information is used to dynamically adjust the learning rate during training cycles. Two approaches are tested: relative change and relative to a reference measure.
result Mutual Information can effectively set dynamic learning rates, leading to competitive performance.
New method uses mutual info and network science to explain deep learning models.
problem Interpreting deep neural networks for understanding their decision-making process.
method Coupling mutual information with network science to quantify information flow in deep learning models.
result Proposed NIF technique for codifying information flow in deep learning models.
Improved bounds on learning algorithms' performance using conditional mutual information.
problem Bounding the generalization error of learning algorithms.
method Introducing conditional mutual information and disintegrated mutual information to tighten bounds.
result New bounds are tighter than previous ones, especially for noisy, iterative algorithms.
MIM learns joint distributions with mutual information and low divergence.
problem Learning joint distributions over observations and latent variables.
method Probabilistic auto-encoder with three design principles: low divergence, high mutual information, and low marginal entropy.
result MIM learns representations with high mutual information, consistent encoding and decoding distributions, effective latent clustering, and comparable data log likelihood to VAE.
Paper proposes mutual information learning for deep learning classifiers.
problem Overfitting in deep learning models.
method Mutual information learning framework to train classifiers.
result MILCs achieve better generalization than conditional entropy classifiers.
Neural network estimates mutual information for ICA.
problem Learning statistically independent features from noisy data.
method Minimizing mutual information estimated by a MINE network in a differentiable encoder network.
result Qualitatively equal solution to FastICA for blind-source-separation.
Wasserstein dependency measure improves unsupervised representation learning.
problem Incomplete representations from mutual information maximization.
method Wasserstein dependency measure using Wasserstein distance instead of KL divergence.
result Improved results on tasks with high mutual information.
Machine learning categorizes mutual funds for better investment strategies.
problem Identifying similar mutual funds for diversified investment applications.
method Machine learning to learn and reproduce a well-regarded categorization system.
result Machine learning can learn and reproduce a categorization system that is widely regarded.
The paper proposes a method to learn structured representations from unlabeled data using mutual information maximization.
problem Learning structured representations from unlabeled data.
method Adversarial maximization of mutual information between a structured latent variable and a target variable.
result The proposed method outperforms current baselines in document hashing and yields highly compressed interpretable representations.
The paper tackles adversarial robustness by maximizing worst-case mutual information.
problem Training robust machine learning models against adversarial inputs is challenging.
method Develops a notion of representation vulnerability and an unsupervised learning method to maximize worst-case mutual information.
result Proves a lower bound on minimum adversarial risk and supports robustness of representations.
New mutual information framework improves contrastive learning for vision tasks.
problem Maximizing mutual information for better unsupervised learning representations.
method Reformulated mutual information as a lower bound, introducing new negative sampling strategies.
result Improved representations outperform previous methods in various vision tasks.
New estimator improves mutual information estimation.
problem Estimating mutual information in data science and machine learning.
method Proposes a new estimator that uses a preliminary estimate of the data distribution.
result A preliminary estimate helps in estimating mutual information more accurately.
The paper proposes a model to learn disentangled representations using mutual information.
problem Learning disentangled representations from shared and exclusive attributes.
method Mutual information maximization for shared attributes and minimization for disentanglement.
result The proposed model outperforms state-of-the-art models in representation disentanglement.
MSRL learns a representation maximizing mutual info with response variables.
problem Learning sufficient representations for complex, multi-dimensional data.
method Variational mutual information, deep neural networks, generalized Dudley's inequality.
result MSRL achieves consistent and accurate representation learning.
The paper introduces submodular information measures for machine learning applications.
problem Generalizing information-theoretic measures to non-random variables.
method Developing combinatorial information measures based on submodular functions.
result Submodular mutual information is submodular in one argument for certain submodular functions.
Proposes a framework to maximize mutual information in VAE models for better latent code representation.
problem Lack of explicit measurement of the quality of learned representations in VAE models.
method Variational Mutual Information Maximization Framework for VAE.
result Maximizes mutual information between latent codes and observations, improving latent code representation.
Paper benchmarks mutual info estimators on diverse distributions.
problem Evaluating mutual information estimators on complex, real-world distributions.
method Constructs a diverse family of known-ground truth distributions, proposes a benchmark platform.
result Highlights differences in classical and neural estimators' performance across various conditions.
Combines chaining and mutual information methods for tighter generalization bounds.
problem Bounding generalization error of learning algorithms, especially in deep learning.
method Integrates chaining and mutual information methods to create a new generalization bound.
result Example shows significant improvement over existing bounds.
The mutual information is a core statistical quantity that has applications in all areas of machine learning, whether this is in training of density models over multiple data modalities, in maximising the efficiency of noisy transmission channels, or when learning behaviour policies for exploration by artificial agents…
A new regularizer boosts long-range dependency in sequence data.
problem Improving long-range dependency in sequence data models.
method Developed a mutual information regularizer to enhance sequence learning.
result The approach increases mutual information and likelihood on holdout data.
Maximizes mutual information to improve graph neural networks performance.
problem Loss of information between nodes in GNNs aggregation and iteration schemes.
method Explores mutual information maximization in the aggregation and iteration scheme of GNNs.
result Improves state-of-the-art performance on graph tasks.
DIM learns deep representations by maximizing mutual information, outperforming unsupervised methods.
problem Learning useful representations from unlabeled data.
method Maximizing mutual information between input and output of a deep neural network encoder, incorporating prior knowledge.
result DIM outperforms other unsupervised learning methods and competes with fully-supervised learning on classification tasks.
The study bounds generalization error using mutual information and proposes methods to control it.
problem Understanding and controlling the generalization capability of learning algorithms.
method Derives upper bounds on generalization error using mutual information, proposes regularization methods.
result Provides theoretical guidelines for balancing data fit and generalization.
A new method estimates mutual information using neural classifiers.
problem Estimating mutual information for high-dimensional data is challenging.
method Trains a classifier to estimate joint distribution probability.
result Demonstrates high accuracy and reduces variance compared to variational methods.
Empowerment-driven exploration improves performance in sparse reward environments.
problem Challenges in reinforcement learning, especially in sparse reward environments.
method Formulate empowerment as mutual information between states and actions, estimate using Mutual Information Neural Estimator and forward dynamics model.
result Empowerment-driven agents significantly improve performance on Montezuma's Revenge.
Deep learning model estimates mutual information with low bias and variance.
problem Estimating mutual information between continuous variables.
method Supervised deep learning approach using Linfoot informational correlation as labels.
result Lower bias and variance compared to other methods.
Proposes mutual information for regression without prior knowledge.
problem Regression tasks without prior model knowledge.
method Mutual information learning formulation, SGD convergence analysis.
result High dimensionality can be beneficial with a threshold.
A new mutual information lower bound for multimodal regression active learning.
problem Lack of effective acquisition functions for multimodal regression active learning.
method Introduces a Two-Index framework for separating epistemic and aleatoric sources of uncertainty, deriving MI-LB as a closed-form approximation.
result MI-LB consistently outperforms baselines on multimodal regression tasks.
We find the maximum mutual information for neural networks and its key determinants.
problem Understanding the maximum mutual information in neural architectures.
method Derived closed-form expression for maximum mutual information across neural network families.
result Maximum mutual information stems from a generalized formula and is influenced by network width and statistical invariances.
New method learns diverse solutions in reinforcement learning without gradient bias.
problem Lack of diverse solutions in reinforcement learning tasks.
method Maximizes state-action-based mutual information directly, using variational lower bound.
result Successfully learns an infinite set of diverse solutions.
Study shows mutual information can reward structure learning agents without expert systems.
problem Designing rewards for structure learning agents in natural language environments.
method Revisited Information Theory of unsupervised induction of phrase-structure grammars, using random sets of linguistic samples.
result Empirical evidence that simulated semantic structures can be distinguished from random ones by mutual information among their constituents.
This paper proposes a method to learn graph representations without supervision.
problem Learning high-quality graph representations without external supervision.
method Graphical Mutual Information (GMI) to measure graph and hidden representation correlation.
result The proposed method outperforms state-of-the-art unsupervised counterparts and sometimes supervised ones.
The study identifies influential factors in VAE representations.
problem Identifying and supervising influential factors in VAE representations without manual intervention.
method Mutual information as a necessary indicator to discover influential factors; design an algorithm for calculating mutual information.
result Mutual information influences the lower bound of VAE's reconstruction error and downstream classification task.
Method learns to select features for each example to maximize mutual information with the response.
problem Model interpretability and feature selection for better understanding of predictions.
method Instancewise feature selection using mutual information maximization.
result Effective in synthetic and real data sets, improving both quantitative and qualitative understanding.
Many machine learning problems can be characterized by mutual contamination models. In these problems, one observes several random samples from different convex combinations of a set of unknown base distributions. It is of interest to decontaminate mutual contamination models, i.e., to recover the base distributions ei…
Proposes a framework to improve VAE latent codes using mutual information.
problem Lack of explicit measure for VAE latent variable quality.
method Variational Mutual Information Maximization Framework.
result Improves relationships between latent codes and observations.
A new reinforcement learning method uses mutual information to encourage agents to control their environment.
problem Learning from internal drives instead of external rewards.
method Formulate an intrinsic objective as mutual information between goal states and controllable states, derive a surrogate objective for efficient optimization.
result Demonstrated the efficacy of the approach in robotic tasks.
Novel mutual information bound improves statistical inference rates.
problem Improving statistical inference rates in Bayesian nonparametrics.
method Introduces a novel mutual information bound.
result Improved contraction rates for fractional posteriors.
InfoBridge uses diffusion bridges to estimate mutual information accurately.
problem Estimating mutual information between random variables.
method Formulated mutual information estimation as a domain transfer problem using diffusion bridge models.
result Demonstrated unbiased estimator for various data types.
A novel method integrates feature and topology views for unsupervised graph representation learning.
problem Lack of mutual information across feature and topology views in graph representation learning.
method Proposes a multi-view representation learning module and a common representation learning module using mutual information maximization and reconstruction loss minimization.
result Demonstrates effectiveness in integrating feature and topology views, achieving comparable or better performance than supervised methods.
STDGI learns node representations for spatio-temporal graphs via mutual information maximization.
problem Challenges in learning node representations for spatio-temporal graphs due to structural changes over time.
method STDGI is a fully unsupervised approach based on mutual information maximization that exploits both spatial and temporal dynamics.
result STDGI's learned node representations improve spatio-temporal auto-regressive forecasting models.
Efficiently learns Gaussian tree models with near-optimal sample complexity.
problem Learning tree-structured Gaussian distributions efficiently.
method Conditional mutual information tester for Gaussian variables, near-optimal sample complexity.
result Near-optimal sample complexity for structure learning of Gaussian tree models.
New model learns symmetry transformations from complex data.
problem Learning symmetry transformations in complex domains like chemical space.
method Two latent subspaces, deep information bottleneck, continuous mutual information regularizer.
result Model outperforms state-of-the-art methods on artificial and molecular datasets.
New method removes unwanted information from representations efficiently.
problem Learning representations that are uninformative about a target variable.
method Adversarial training with a novel proxy metric for mutual information, leading to an analytically computable approximation.
result Our method effectively removes unwanted information with limited time budget.
A new method estimates mutual information with fewer paired samples using optimal transport.
problem Estimating mutual information from limited paired samples.
method Semi-supervised Squared-loss Mutual Information (SMI) estimation using optimal transport.
result The method can estimate mutual information with fewer paired samples.
A new multi-label CPC method improves mutual information estimation and representation learning.
problem Underestimation of mutual information in contrastive predictive coding.
method Introducing a multi-label classification problem to overcome the logm bound in mutual information estimation. result The new method exceeds the logm bound and leads to better mutual information estimation and improved unsupervised representation learning. Proposes MIGE for accurate MI gradient estimation in high-dimensional settings.
problem Intractability of MI in continuous and high-dimensional settings.
method Score estimation of implicit distributions for gradient estimation of MI.
result MIGE provides tight and smooth gradient estimation of MI in high-dimensional settings.