Proposes a framework to maximize mutual information in VAE models for better latent code representation.
problem Lack of explicit measurement of the quality of learned representations in VAE models.
method Variational Mutual Information Maximization Framework for VAE.
result Maximizes mutual information between latent codes and observations, improving latent code representation.
The paper proposes a method to learn structured representations from unlabeled data using mutual information maximization.
problem Learning structured representations from unlabeled data.
method Adversarial maximization of mutual information between a structured latent variable and a target variable.
result The proposed method outperforms current baselines in document hashing and yields highly compressed interpretable representations.
The paper proves mutual information measurement is statistically limited.
problem Measuring mutual information from finite data is difficult.
method Proves statistical limitations on any method of measuring mutual information.
result Any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N ).
The paper tackles adversarial robustness by maximizing worst-case mutual information.
problem Training robust machine learning models against adversarial inputs is challenging.
method Develops a notion of representation vulnerability and an unsupervised learning method to maximize worst-case mutual information.
result Proves a lower bound on minimum adversarial risk and supports robustness of representations.
Maximizes mutual information to improve graph neural networks performance.
problem Loss of information between nodes in GNNs aggregation and iteration schemes.
method Explores mutual information maximization in the aggregation and iteration scheme of GNNs.
result Improves state-of-the-art performance on graph tasks.
A novel method integrates feature and topology views for unsupervised graph representation learning.
problem Lack of mutual information across feature and topology views in graph representation learning.
method Proposes a multi-view representation learning module and a common representation learning module using mutual information maximization and reconstruction loss minimization.
result Demonstrates effectiveness in integrating feature and topology views, achieving comparable or better performance than supervised methods.
MINIMALIST maximizes mutual information for likelihood estimation from simulated data.
problem Learning model parameters from likelihood functions that cannot be computed.
method Maximizes mutual information between simulated data and model parameters using neural networks.
result Different methods aiming at the same optimal energy form can be directly benchmarked.
QAInfomax improves reading comprehension by maximizing mutual information, achieving state-of-the-art performance.
problem Distractor sentences in question answering datasets are hard to distinguish from relevant ones.
method QAInfomax regularizes reading comprehension models to learn mutual information among passages, questions, and answers.
result QAInfomax achieves state-of-the-art performance on Adversarial-SQuAD dataset.
Information-maximization clustering learns a probabilistic classifier in an unsupervised manner so that mutual information between feature vectors and cluster assignments is maximized. A notable advantage of this approach is that it only involves continuous optimization of model parameters, which is substantially easie…
AMI framework improves text generation by optimizing mutual information between source and target.
problem Previous MI approaches ignored the backward network, leading to loose variational bounds.
method AMI is a saddle point optimization framework that iteratively promotes and demotes generated instances.
result AMI significantly outperforms baselines on various text generation tasks.
A method to improve image synthesis diversity using mutual information.
problem Mode collapse in conditional GANs for multimodal image synthesis.
method Explicitly estimate and maximize mutual information between latent code and output image.
result Prevents mode collapse and encourages synthesis of diverse images.
Enhances exploration in hierarchical networks using mutual information.
problem Limited exploration in hierarchical Deep Q-Networks.
method Adversarial Soft Actor-Critic with mutual information optimization.
result Improves hierarchical network exploration through mutual information maximization.
STDGI learns node representations for spatio-temporal graphs via mutual information maximization.
problem Challenges in learning node representations for spatio-temporal graphs due to structural changes over time.
method STDGI is a fully unsupervised approach based on mutual information maximization that exploits both spatial and temporal dynamics.
result STDGI's learned node representations improve spatio-temporal auto-regressive forecasting models.
Proposes a framework to improve VAE latent codes using mutual information.
problem Lack of explicit measure for VAE latent variable quality.
method Variational Mutual Information Maximization Framework.
result Improves relationships between latent codes and observations.
Reshef et al. recently proposed a new statistical measure, the "maximal information coefficient" (MIC), for quantifying arbitrary dependencies between pairs of stochastic quantities. MIC is based on mutual information, a fundamental quantity in information theory that is widely understood to serve this need. MIC, howev…
Maximizes mutual info across views for better image representations.
problem Improving image representation learning through multiple views.
method Maximizing mutual information between features from multiple views.
result ImageNet accuracy of 68.1% using linear evaluation, significantly outperforming prior methods.
InfoOT improves data alignment by maximizing mutual information.
problem Optimal transport's limitations in handling clusters, outliers, and new data.
method InfoOT extends optimal transport by maximizing mutual information while minimizing distances.
result InfoOT outperforms optimal transport in domain adaptation, cross-domain retrieval, and single-cell alignment.
This paper proposes a method to learn graph representations without supervision.
problem Learning high-quality graph representations without external supervision.
method Graphical Mutual Information (GMI) to measure graph and hidden representation correlation.
result The proposed method outperforms state-of-the-art unsupervised counterparts and sometimes supervised ones.
MIRO learns robust latent spaces by maximizing mutual information with future information.
problem Robust perception in complex, unstructured environments with low sample complexity.
method MIRO maximizes mutual information in a latent space for model-based reinforcement learning.
result MIRO outperforms reconstruction objectives in cluttered scenes.
InfoMax Autoencoder maximizes mutual information for robust data representation.
problem Learning robust data representations from raw data.
method Explicitly maximizes mutual information between input and hidden representation.
result IMAE outperforms state-of-the-art models in clustering performance.
InfoMax-VAE enhances VAEs for better representation learning.
problem Lack of meaningful learned representations in VAEs.
method Combines VAEs with mutual information maximization.
result Significantly boosts quality of learned high-level representations.
IGNN improves GNNs by maximizing edge-state transform mutual information.
problem Optimizing GNNs for better relational information.
method Variational information maximization to learn optimal transform parameters.
result IGNN achieves state-of-the-art performance on molecular graph tasks.
Feature selection is one of the most fundamental problems in machine learning. An extensive body of work on information-theoretic feature selection exists which is based on maximizing mutual information between subsets of features and class labels. Practical methods are forced to rely on approximations due to the diffi…
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, an…
Recent unsupervised representation learning methods maximize mutual information, but their success depends on architecture and estimator inductive biases.
problem Estimating mutual information is hard, and MI maximization can lead to entangled representations.
method The paper argues that the success of mutual information maximization methods depends on the choice of feature extractor architectures and the parametrization of MI estimators.
result The paper provides empirical evidence that the success of mutual information maximization methods is not solely due to MI properties.
TIM maximizes mutual information for few-shot learning, outperforming state-of-the-art methods.
problem Few-shot learning with limited labeled data.
method Transductive Information Maximization (TIM) with alternating-direction solver.
result Significant improvement in accuracy across various datasets and networks.
We present a two-stage approach for learning dictionaries for object classification tasks based on the principle of information maximization. The proposed method seeks a dictionary that is compact, discriminative, and generative. In the first stage, dictionary atoms are selected from an initial dictionary by maximizing…
A privacy-constrained information extraction problem is considered where for a pair of correlated discrete random variables (X,Y) governed by a given joint distribution, an agent observes Y and wants to convey to a potentially public user as much information about Y as possible without compromising the amount of …
InfoPrompt improves soft prompt tuning by maximizing mutual information, leading to better performance.
problem High sensitivity of prompt tuning to initial conditions and insufficient task-relevant information.
method Develops an information-theoretic framework to maximize mutual information between prompts and model parameters, using novel loss functions.
result InfoPrompt accelerates convergence and outperforms traditional methods.
A new regularizer boosts long-range dependency in sequence data.
problem Improving long-range dependency in sequence data models.
method Developed a mutual information regularizer to enhance sequence learning.
result The approach increases mutual information and likelihood on holdout data.
A new method selects models for ensemble learning to maximize mutual information, outperforming existing approaches.
problem Selecting models for ensemble learning to improve performance and reduce correlation issues.
method Formulate budgeted ensemble selection as maximizing mutual information, use Gaussian-copula to model correlated errors, propose a greedy mutual-information selection algorithm.
result Our method consistently outperforms strong baselines across multiple datasets.
InfoGraph learns graph-level representations via mutual information maximization.
problem Learning graph-level representations for unsupervised and semi-supervised scenarios.
method Maximizes mutual information between graph-level representation and substructure representations.
result InfoGraph outperforms state-of-the-art methods on graph classification and molecular property prediction.
The paper proposes a model to learn disentangled representations using mutual information.
problem Learning disentangled representations from shared and exclusive attributes.
method Mutual information maximization for shared attributes and minimization for disentanglement.
result The proposed model outperforms state-of-the-art models in representation disentanglement.
New method learns diverse solutions in reinforcement learning without gradient bias.
problem Lack of diverse solutions in reinforcement learning tasks.
method Maximizes state-action-based mutual information directly, using variational lower bound.
result Successfully learns an infinite set of diverse solutions.
Autoencoders learn compressed representations via mutual information maximization.
problem Learning efficient compressed representations of high-dimensional data.
method Proposes Uncertainty Autoencoders that treat latent representations as noisy projections and optimize mutual information.
result 32% improvement in statistical compressed sensing of high-dimensional datasets.
Empowerment-driven exploration improves performance in sparse reward environments.
problem Challenges in reinforcement learning, especially in sparse reward environments.
method Formulate empowerment as mutual information between states and actions, estimate using Mutual Information Neural Estimator and forward dynamics model.
result Empowerment-driven agents significantly improve performance on Montezuma's Revenge.
The paper analyzes generalization of noisy, iterative algorithms using maximal leakage.
problem Analyzing the generalization behavior of noisy, iterative learning algorithms.
method Information-theoretic framework with maximal leakage metric.
result Explicit upper bounds on maximal leakage for various scenarios.
Study on information evolution in interactive decision making using multi-armed bandits.
problem Understanding information dynamics in interactive decision making.
method Stochastic multi-armed bandit problem, focusing on optimal arm with a fixed margin.
result Distinct growth phases in mutual information, showing decoupling between success probability and information gain.
Paper proposes InfoAE for disentangled representation learning.
problem Learning disentangled representations from unlabeled data.
method InfoAE learns disentangled representation by maximizing mutual information.
result Achieved 98.9% test accuracy on MNIST with unsupervised training.
The paper introduces submodular information measures for machine learning applications.
problem Generalizing information-theoretic measures to non-random variables.
method Developing combinatorial information measures based on submodular functions.
result Submodular mutual information is submodular in one argument for certain submodular functions.
A new HRL method learns hierarchical policies using mutual information maximization.
problem Learning hierarchical policies in reinforcement learning for structured tasks.
method Mutual information maximization for latent variable learning, advantage-weighted importance sampling for option policies, deterministic policy gradient for optimization.
result Enhanced performance in continuous control tasks through learned hierarchical policies.
This paper improves multi-class calibration methods using mutual information maximization-based binning.
problem Calibration of deep neural network predictions, especially for small prior classes.
method I-Max concept for binning, shared class-wise calibration strategy.
result Improves multi-class ranking and calibration performance using a small calibration set.
Wasserstein dependency measure improves unsupervised representation learning.
problem Incomplete representations from mutual information maximization.
method Wasserstein dependency measure using Wasserstein distance instead of KL divergence.
result Improved results on tasks with high mutual information.
Estimates utility functions and information costs from YouTube comments.
problem Estimating rational inattention in Bayesian agents.
method Deep learning for clustering framing information, inverse reinforcement learning.
result Constructive estimates of utility and information costs.
A scalable algorithm compresses large categorical feature vocabularies while maximizing mutual information.
problem Learning from categorical features with very large vocabularies.
method Submodular optimization to maximize mutual information between compressed feature and target labels.
result The algorithm achieves a 63% approximation of the global optimal solution for mutual information.
New mutual information framework improves contrastive learning for vision tasks.
problem Maximizing mutual information for better unsupervised learning representations.
method Reformulated mutual information as a lower bound, introducing new negative sampling strategies.
result Improved representations outperform previous methods in various vision tasks.
Cross-entropy loss linked to metric learning, outperforming complex pairwise losses.
problem Improving metric learning performance without complex optimization schemes.
method Theoretical analysis linking cross-entropy to pairwise losses, showing cross-entropy as an upper bound and equivalent to mutual information maximization.
result Minimizing cross-entropy is equivalent to maximizing mutual information, leading to state-of-the-art performance.
MSRL learns a representation maximizing mutual info with response variables.
problem Learning sufficient representations for complex, multi-dimensional data.
method Variational mutual information, deep neural networks, generalized Dudley's inequality.
result MSRL achieves consistent and accurate representation learning.