The paper proposes methods to extract and analyze individual variable information from complex dependencies.
problem Analyzing and understanding complex dependencies between multiple variables.
method Reversible normalization and iterative dependency reduction to extract individual information, and use it for direct mutual information and multi-feature Granger causality analysis.
result Decoupling of variables to analyze their individual information and direct mutual information transfers.
New method estimates spin system mutual information using neural networks.
problem Estimating mutual information in spin systems.
method Monte Carlo sampling enhanced by autoregressive neural networks.
result Area law satisfied for temperatures away from critical temperature.
A new method estimates mutual information using neural classifiers.
problem Estimating mutual information for high-dimensional data is challenging.
method Trains a classifier to estimate joint distribution probability.
result Demonstrates high accuracy and reduces variance compared to variational methods.
Two novel methods estimate multiple FDR directions for binary categorical responses.
problem Estimating multiple FDR directions for categorical responses.
method Information maximization and square loss mutual information.
result Statistical consistency of the proposed methods established.
Proposes a framework to improve VAE latent codes using mutual information.
problem Lack of explicit measure for VAE latent variable quality.
method Variational Mutual Information Maximization Framework.
result Improves relationships between latent codes and observations.
Improved Monte-Carlo models by constraining mutual information between latent and observable variables.
problem Training density models leads to latent variables being useless.
method Weave tighter Monte-Carlo bounds with mutual information constraints.
result Improved training of models with continuous and discrete latent variables.
This paper reviews information theory in open-world machine learning.
problem Lack of a unified theoretical foundation for open-world machine learning.
method Synthesis of information theoretic approaches.
result Established a pathway toward provable and trustworthy open world intelligence.
MINDE estimates Mutual Information using neural diffusion models.
problem Estimating Mutual Information between random variables.
method Score-based diffusion models to estimate Kullback Leibler divergence.
result MINDE is more accurate than existing methods, especially for challenging distributions.
Unified bounds linking compressibility, fractal dimensions, and mutual information.
problem Understanding generalization in stochastic learning algorithms.
method Rate-distortion theory applied to machine learning generalization.
result Unified bounds linking compressibility, fractal dimensions, and mutual information.
Develops a method to explain deep learning models for complex systems.
problem Rapid simulation-based prototyping of complex systems with high-dimensional CVs and QoIs.
method Moment-independent global sensitivity analysis using differential mutual information.
result Surrogate model driven by mutual information provides useful rankings and optimizations.
Feature selection, which searches for the most representative features in observed data, is critical for health data analysis. Unlike feature extraction, such as PCA and autoencoder based methods, feature selection preserves interpretability, meaning that the selected features provide direct information about certain h…
A new framework uses information theory to detect anomalies in images without labeled data.
problem Detect anomalies in images without labeled data.
method A direct objective function using information theory to maximize the distance between normal and anomalous data.
result The proposed framework significantly outperforms state-of-the-arts on multiple benchmark datasets.
Paper revisits Deep Variational Information Bottleneck and proposes a new optimization approach.
problem Limitations of Deep Variational Information Bottleneck in optimizing mutual information.
method Proposes a new optimization approach by circumventing the limitation of requiring both Markov chains during optimisation.
result Shows how to optimise a lower bound for mutual information, circumventing the limitation of requiring both Markov chains.
The article introduces inferential moments for analyzing uncertain multivariable systems.
problem Handling inference tasks in uncertain multivariable systems.
method Bayesian inference and quantification of inferential moments.
result Quantification of inferential moments and their connection to mutual information.
Proposes a method to improve financial time series forecasting using compact representations and contrastive loss.
problem Financial time series forecasting with small datasets and overfitting issues.
method Class-conditioned latent variable model, mutual information maximization, contrastive loss, deep autoregressive models.
result Empirical experiments show improved performance compared to state-of-the-art methods.
TIM maximizes mutual information for few-shot learning, outperforming state-of-the-art methods.
problem Few-shot learning with limited labeled data.
method Transductive Information Maximization (TIM) with alternating-direction solver.
result Significant improvement in accuracy across various datasets and networks.
New neural network approach using mutual information.
problem Training neural networks for imbalanced datasets.
method Converts neural network classifiers to mutual information evaluators.
result New form of softmax leads to better classification accuracy, especially for imbalanced datasets.
Paper describes profiles of multivariate normal distributions and novel estimators for mutual information.
problem Estimating mutual information for complex distributions.
method Analytical description of profiles, introduction of Bend and Mix Models, Monte Carlo estimation.
result Bend and Mix Models accurately estimate mutual information profiles and provide Bayesian estimates.
A new framework uses directed information to efficiently select context chunks.
problem Efficiently selecting relevant context chunks for query understanding.
method Directed Information γ-covering framework, formulated as a γ-cover problem, with a greedy algorithm for context selection. result The γ-covering algorithm provides clear advantages in hard-decision regimes like context compression and single-slot prompt selection. Improved bounds on learning algorithms' performance using conditional mutual information.
problem Bounding the generalization error of learning algorithms.
method Introducing conditional mutual information and disintegrated mutual information to tighten bounds.
result New bounds are tighter than previous ones, especially for noisy, iterative algorithms.
Paper benchmarks mutual info estimators on diverse distributions.
problem Evaluating mutual information estimators on complex, real-world distributions.
method Constructs a diverse family of known-ground truth distributions, proposes a benchmark platform.
result Highlights differences in classical and neural estimators' performance across various conditions.
MIM learns useful representations with high mutual information.
problem Learning useful representations for downstream tasks.
method Symmetric Jensen-Shannon divergence and mutual information regularizer in an encoder/decoder framework.
result MIM learns high mutual information representations without posterior collapse.
The paper argues that normalized mutual information is biased in clustering and community detection.
problem Bias in normalized mutual information for clustering and community detection.
method Introducing a modified version of mutual information to correct for information content and spurious dependence.
result The modified mutual information leads to different conclusions about which algorithms are best for community detection.
Neural estimator improves mutual information estimation in high dimensions.
problem Estimating mutual information in high dimensions is challenging.
method Parametrizing conditional densities with normalizing flows and using block autoregressive structure.
result Improved mutual information estimation on benchmark tasks.
Paper proposes a new unsupervised method for cross-modality data translation without requiring direct mappings.
problem Tackles the challenge of zero-shot cross-modality data translation with fidelity.
method Mutual Information guided Diffusion cross-modality data translation Model (MIDiffusion) using score-matching and stochastic diffusion.
result Empirically shows advanced performance compared to other generative models.
We find the maximum mutual information for neural networks and its key determinants.
problem Understanding the maximum mutual information in neural architectures.
method Derived closed-form expression for maximum mutual information across neural network families.
result Maximum mutual information stems from a generalized formula and is influenced by network width and statistical invariances.
Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confide…
Improved method for encoding contingency tables reduces mutual information bias.
problem Mutual information bias in measuring label similarity.
method Improved method for encoding contingency tables to reduce information cost.
result Better bound on reduced mutual information in typical use cases.
MIM learns joint distributions with mutual information and low divergence.
problem Learning joint distributions over observations and latent variables.
method Probabilistic auto-encoder with three design principles: low divergence, high mutual information, and low marginal entropy.
result MIM learns representations with high mutual information, consistent encoding and decoding distributions, effective latent clustering, and comparable data log likelihood to VAE.
Softmax cross-entropy optimizes mutual information in neural networks.
problem Understanding the relationship between mutual information and classification neural networks.
method Demonstrated that optimizing softmax cross-entropy maximizes mutual information between inputs and labels.
result Softmax cross-entropy can approximate mutual information and highlight relevant image regions.
Proposes a new bound on generalization error using conditional mutual information.
problem Improving the generalization error bound in machine learning.
method Combines error decomposition and conditional mutual information techniques.
result New bound is order-wise better than previous ones in a simple Gaussian setting.
The paper introduces submodular information measures for machine learning applications.
problem Generalizing information-theoretic measures to non-random variables.
method Developing combinatorial information measures based on submodular functions.
result Submodular mutual information is submodular in one argument for certain submodular functions.
Proposes a framework to maximize mutual information in VAE models for better latent code representation.
problem Lack of explicit measurement of the quality of learned representations in VAE models.
method Variational Mutual Information Maximization Framework for VAE.
result Maximizes mutual information between latent codes and observations, improving latent code representation.
Unified framework simplifies DPO algorithms for LLM alignment.
problem Vast number of DPO variants complicates model alignment.
method Mutual information inspired unifying framework with flexible priors.
result Many DPO variants can be derived from the new framework.
New estimator improves mutual information estimation.
problem Estimating mutual information in data science and machine learning.
method Proposes a new estimator that uses a preliminary estimate of the data distribution.
result A preliminary estimate helps in estimating mutual information more accurately.
New insights link RLHF and contrastive learning for better model alignment.
problem Aligning large language models with human values.
method Interpreting RLHF and DPO as contrastive learning methods based on mutual information.
result Proposed Mutual Information Optimization (MIO) improves model performance.
The paper proposes a method to learn structured representations from unlabeled data using mutual information maximization.
problem Learning structured representations from unlabeled data.
method Adversarial maximization of mutual information between a structured latent variable and a target variable.
result The proposed method outperforms current baselines in document hashing and yields highly compressed interpretable representations.
Paper proposes mutual information learning for deep learning classifiers.
problem Overfitting in deep learning models.
method Mutual information learning framework to train classifiers.
result MILCs achieve better generalization than conditional entropy classifiers.
The paper proposes a model to learn disentangled representations using mutual information.
problem Learning disentangled representations from shared and exclusive attributes.
method Mutual information maximization for shared attributes and minimization for disentanglement.
result The proposed model outperforms state-of-the-art models in representation disentanglement.
TCMI assesses mutual dependence of continuous variables without parametric assumptions.
problem Estimating mutual information from continuous distributions.
method TCMI extends mutual information to continuous variables using cumulative distributions.
result TCMI facilitates feature selection and ranking of variable sets.
New model quantifies how much machine learning models can reveal about individual data usage.
problem Measuring and reducing the leakage of membership information from machine learning models.
method Using information theory, conditional mutual information leakage, and Kullback-Leibler divergence to quantify and bound the leakage.
result The amount of membership information leakage is reduced by adding Gaussian (ε,δ)-differentially-private additive noises. We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, an…
Study mutual info for community detection with covariate and correlated networks.
problem Community detection with covariate and correlated networks.
method Asymptotic upper bound and MMSE matrix heuristic analysis.
result Explicit characterization of combined information effects.
Reshef et al. recently proposed a new statistical measure, the "maximal information coefficient" (MIC), for quantifying arbitrary dependencies between pairs of stochastic quantities. MIC is based on mutual information, a fundamental quantity in information theory that is widely understood to serve this need. MIC, howev…
Sequence models assign probabilities to variable-length sequences such as natural language texts. The ability of sequence models to capture temporal dependence can be characterized by the temporal scaling of correlation and mutual information. In this paper, we study the mutual information of recurrent neural networks …
Active feature selection uses mutual information to choose fewer labels for better feature selection.
problem Selecting features with limited labeled data.
method Uses active feature selection with mutual information criterion, optimizing label selection for better feature quality.
result Algorithm selects features with higher mutual information using fewer labels than the data set size.
In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex lo…
In this paper, we present a new approach to interpret deep learning models. By coupling mutual information with network science, we explore how information flows through feedforward networks. We show that efficiently approximating mutual information allows us to create an information measure that quantifies how much in…