Softmax emerges naturally in neural networks as a measure of conditional mutual information.
problem The artificial nature of softmax in neural networks.
method Information-theoretic perspective to derive log-softmax and evaluate conditional mutual information.
result Training deterministic neural networks through log-softmax maximises conditional mutual information.
A new algorithm PD improves stock-correlation network clustering and robustness.
problem Improving clustering and robustness of stock-correlation networks.
method Proposes a new proportional degree algorithm to filter information on a complete graph of normalised mutual information.
result The PD algorithm produces a network with better homogeneity and robustness compared to PMFG.
Forecastability measures predictive information across horizons.
problem How much predictive information is available at each prediction horizon?
method Develops the consequences of mutual information between future observations and information set.
result Forecastability is a profile reflecting process dependence structure, with properties like compression and truncation error.
Dynamic time warping (DTW) can be used to compute the similarity between two sequences of generally differing length. We propose a modification to DTW that performs individual and independent pairwise alignment of feature trajectories. The modified technique, termed feature trajectory dynamic time warping (FTDTW), is a…
A new measure normalizes clustering accuracy to evaluate algorithms better.
problem Evaluation of clustering algorithms is challenging due to limitations of existing measures.
method Proposes a new, normalised clustering accuracy measure.
result The new measure identifies worst-case scenarios and is more interpretable.
Study models Indian stock market using hyperbolic geometry for market stability and volatility analysis.
problem Identifying market stability and volatility in the Indian stock market.
method Modelled as a heterogeneous scale-free network, embedded in a 2D hyperbolic space, applied coalescent embedding, hyperbolic kmeans, and Bollinger Band analysis.
result Clusters in the embedded network better represent market communities than Euclidean clusters, allowing for early detection of market changes.
Squared families are a new model class derived from linear transformations, offering convenient properties and universal approximation.
problem Developing a new class of probability models that are easier to handle and have useful properties.
method Introducing squared families as families of probability densities obtained by squaring a linear transformation of a statistic, and showing their properties and applications.
result Squared families have convenient properties and can approximate target densities well.
New neural network approach using mutual information.
problem Training neural networks for imbalanced datasets.
method Converts neural network classifiers to mutual information evaluators.
result New form of softmax leads to better classification accuracy, especially for imbalanced datasets.
Paper describes profiles of multivariate normal distributions and novel estimators for mutual information.
problem Estimating mutual information for complex distributions.
method Analytical description of profiles, introduction of Bend and Mix Models, Monte Carlo estimation.
result Bend and Mix Models accurately estimate mutual information profiles and provide Bayesian estimates.
Neural network estimates mutual information for ICA.
problem Learning statistically independent features from noisy data.
method Minimizing mutual information estimated by a MINE network in a differentiable encoder network.
result Qualitatively equal solution to FastICA for blind-source-separation.
Improved bounds on learning algorithms' performance using conditional mutual information.
problem Bounding the generalization error of learning algorithms.
method Introducing conditional mutual information and disintegrated mutual information to tighten bounds.
result New bounds are tighter than previous ones, especially for noisy, iterative algorithms.
Paper benchmarks mutual info estimators on diverse distributions.
problem Evaluating mutual information estimators on complex, real-world distributions.
method Constructs a diverse family of known-ground truth distributions, proposes a benchmark platform.
result Highlights differences in classical and neural estimators' performance across various conditions.
MIM learns useful representations with high mutual information.
problem Learning useful representations for downstream tasks.
method Symmetric Jensen-Shannon divergence and mutual information regularizer in an encoder/decoder framework.
result MIM learns high mutual information representations without posterior collapse.
The paper argues that normalized mutual information is biased in clustering and community detection.
problem Bias in normalized mutual information for clustering and community detection.
method Introducing a modified version of mutual information to correct for information content and spurious dependence.
result The modified mutual information leads to different conclusions about which algorithms are best for community detection.
Enhances exploration in hierarchical networks using mutual information.
problem Limited exploration in hierarchical Deep Q-Networks.
method Adversarial Soft Actor-Critic with mutual information optimization.
result Improves hierarchical network exploration through mutual information maximization.
Neural estimator improves mutual information estimation in high dimensions.
problem Estimating mutual information in high dimensions is challenging.
method Parametrizing conditional densities with normalizing flows and using block autoregressive structure.
result Improved mutual information estimation on benchmark tasks.
We find the maximum mutual information for neural networks and its key determinants.
problem Understanding the maximum mutual information in neural architectures.
method Derived closed-form expression for maximum mutual information across neural network families.
result Maximum mutual information stems from a generalized formula and is influenced by network width and statistical invariances.
New mutual information measure improves classification and community detection accuracy.
problem Standard mutual information measure can be inaccurate under real-world conditions.
method Corrected mutual information measure that accounts for all cases.
result Improved mutual information measure reduces errors in classification and community detection.
Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confide…
Improved method for encoding contingency tables reduces mutual information bias.
problem Mutual information bias in measuring label similarity.
method Improved method for encoding contingency tables to reduce information cost.
result Better bound on reduced mutual information in typical use cases.
MIM learns joint distributions with mutual information and low divergence.
problem Learning joint distributions over observations and latent variables.
method Probabilistic auto-encoder with three design principles: low divergence, high mutual information, and low marginal entropy.
result MIM learns representations with high mutual information, consistent encoding and decoding distributions, effective latent clustering, and comparable data log likelihood to VAE.
Softmax cross-entropy optimizes mutual information in neural networks.
problem Understanding the relationship between mutual information and classification neural networks.
method Demonstrated that optimizing softmax cross-entropy maximizes mutual information between inputs and labels.
result Softmax cross-entropy can approximate mutual information and highlight relevant image regions.
Proposes a new bound on generalization error using conditional mutual information.
problem Improving the generalization error bound in machine learning.
method Combines error decomposition and conditional mutual information techniques.
result New bound is order-wise better than previous ones in a simple Gaussian setting.
This paper studies mutual information in sequence models and finds Transformers excel in capturing long-range dependencies.
problem Understanding the expressive power of sequence models in capturing temporal dependencies.
method Theoretical and empirical analysis of linear and nonlinear RNNs, including Transformers.
result Transformers can capture long-range mutual information more efficiently than RNNs.
This paper investigates methods for quantifying similarity between audio signals, specifically for the task of of cover song detection. We consider an information-theoretic approach, where we compute pairwise measures of predictability between time series. We compare discrete-valued approaches operating on quantised au…
The paper introduces submodular information measures for machine learning applications.
problem Generalizing information-theoretic measures to non-random variables.
method Developing combinatorial information measures based on submodular functions.
result Submodular mutual information is submodular in one argument for certain submodular functions.
Proposes a framework to maximize mutual information in VAE models for better latent code representation.
problem Lack of explicit measurement of the quality of learned representations in VAE models.
method Variational Mutual Information Maximization Framework for VAE.
result Maximizes mutual information between latent codes and observations, improving latent code representation.
Quantile normalisation is a popular normalisation method for data subject to unwanted variations such as images, speech, or genomic data. It applies a monotonic transformation to the feature values of each sample to ensure that after normalisation, they follow the same target distribution for each sample. Choosing a "g…
A new method estimates mutual information using neural classifiers.
problem Estimating mutual information for high-dimensional data is challenging.
method Trains a classifier to estimate joint distribution probability.
result Demonstrates high accuracy and reduces variance compared to variational methods.
New estimator improves mutual information estimation.
problem Estimating mutual information in data science and machine learning.
method Proposes a new estimator that uses a preliminary estimate of the data distribution.
result A preliminary estimate helps in estimating mutual information more accurately.
The paper proposes a method to learn structured representations from unlabeled data using mutual information maximization.
problem Learning structured representations from unlabeled data.
method Adversarial maximization of mutual information between a structured latent variable and a target variable.
result The proposed method outperforms current baselines in document hashing and yields highly compressed interpretable representations.
Paper proposes mutual information learning for deep learning classifiers.
problem Overfitting in deep learning models.
method Mutual information learning framework to train classifiers.
result MILCs achieve better generalization than conditional entropy classifiers.
A new regularizer boosts long-range dependency in sequence data.
problem Improving long-range dependency in sequence data models.
method Developed a mutual information regularizer to enhance sequence learning.
result The approach increases mutual information and likelihood on holdout data.
The paper proposes a model to learn disentangled representations using mutual information.
problem Learning disentangled representations from shared and exclusive attributes.
method Mutual information maximization for shared attributes and minimization for disentanglement.
result The proposed model outperforms state-of-the-art models in representation disentanglement.
TCMI assesses mutual dependence of continuous variables without parametric assumptions.
problem Estimating mutual information from continuous distributions.
method TCMI extends mutual information to continuous variables using cumulative distributions.
result TCMI facilitates feature selection and ranking of variable sets.
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, an…
Study mutual info for community detection with covariate and correlated networks.
problem Community detection with covariate and correlated networks.
method Asymptotic upper bound and MMSE matrix heuristic analysis.
result Explicit characterization of combined information effects.
Reshef et al. recently proposed a new statistical measure, the "maximal information coefficient" (MIC), for quantifying arbitrary dependencies between pairs of stochastic quantities. MIC is based on mutual information, a fundamental quantity in information theory that is widely understood to serve this need. MIC, howev…
Active feature selection uses mutual information to choose fewer labels for better feature selection.
problem Selecting features with limited labeled data.
method Uses active feature selection with mutual information criterion, optimizing label selection for better feature quality.
result Algorithm selects features with higher mutual information using fewer labels than the data set size.
In this paper, we present a new approach to interpret deep learning models. By coupling mutual information with network science, we explore how information flows through feedforward networks. We show that efficiently approximating mutual information allows us to create an information measure that quantifies how much in…
This paper improves DNN generalization by accurately estimating mutual information.
problem Intractability of estimating mutual information in DNNs.
method Introduces a probabilistic representation of DNNs to accurately estimate mutual information.
result Derives a tighter generalization bound than previous relaxations.
Novel mutual information bound improves statistical inference rates.
problem Improving statistical inference rates in Bayesian nonparametrics.
method Introduces a novel mutual information bound.
result Improved contraction rates for fractional posteriors.
Study reveals mutual information is crucial for understanding algorithm performance in stochastic convex optimization.
problem Uncertainty in capturing the exceptional performance of learning algorithms using existing information-theoretic generalization bounds.
method Examined the relationship between mutual information and generalization in stochastic convex optimization.
result Mutual information is necessary for true risk minimization in stochastic convex optimization, indicating existing bounds fall short.
InfoBridge uses diffusion bridges to estimate mutual information accurately.
problem Estimating mutual information between random variables.
method Formulated mutual information estimation as a domain transfer problem using diffusion bridge models.
result Demonstrated unbiased estimator for various data types.
Paper proposes a novel method to reduce mutual information for missing data imputation.
problem Missing data imputation in datasets with missingness patterns.
method Iterative minimization of KL divergence between imputed data and missingness mask, using rectified flow training objective.
result The proposed method achieves superior imputation performance on synthetic and real-world datasets.
New method estimates mutual information using normalizing flows.
problem Mutual information estimation in high-dimensional data.
method Normalizing flows to map data to target distributions with known MI.
result Theoretical guarantees and practical advantages demonstrated.
Wasserstein dependency measure improves unsupervised representation learning.
problem Incomplete representations from mutual information maximization.
method Wasserstein dependency measure using Wasserstein distance instead of KL divergence.
result Improved results on tasks with high mutual information.
Hybrid method improves mutual information estimation from samples.
problem Estimating mutual information from joint distributions is challenging.
method Proposes a hybrid method combining discriminative and generative approaches.
result Hybrid methods yield tighter variational bounds on mutual information.