Proposes MIGE for accurate MI gradient estimation in high-dimensional settings.
problem Intractability of MI in continuous and high-dimensional settings.
method Score estimation of implicit distributions for gradient estimation of MI.
result MIGE provides tight and smooth gradient estimation of MI in high-dimensional settings.
New method estimates mutual information using normalizing flows.
problem Mutual information estimation in high-dimensional data.
method Normalizing flows to map data to target distributions with known MI.
result Theoretical guarantees and practical advantages demonstrated.
Study introduces a benchmark suite for evaluating neural MI estimators on real-world unstructured datasets.
problem Lack of comprehensive evaluation methods for neural MI estimators on real-world unstructured datasets.
method Developed a benchmark suite using same-class sampling and a binary symmetric channel trick.
result Showed accurate manipulation of true MI values of real-world datasets.
InfoBridge uses diffusion bridges to estimate mutual information accurately.
problem Estimating mutual information between random variables.
method Formulated mutual information estimation as a domain transfer problem using diffusion bridge models.
result Demonstrated unbiased estimator for various data types.
New methods estimate point-wise dependency from neural MI models.
problem Estimating point-wise dependency between different events.
method Developed two methods: Probabilistic Classifier and Density-Ratio Fitting.
result Demonstrated effectiveness in MI estimation, self-supervised representation learning, and cross-modal retrieval.
New MI bounds improve estimation in deep generative models.
problem Estimating mutual information without density information is intractable.
method Importance sampling, Annealed Importance Sampling, Generalized IWAE, MINE-AIS.
result Improved bounds for estimating mutual information in deep models.
A new contrastive MI estimator improves efficiency and tightness.
problem Efficient and tight mutual information estimation.
method Contrastive Fenchel-Legendre optimization.
result The FLO estimator is tight and converges under stochastic gradient descent.
Paper refines InfoNCE for accurate mutual information estimation.
problem Indirect connection of InfoNCE to mutual information estimation.
method Introduces InfoNCE-anchor with an auxiliary anchor class for consistent density ratio estimation.
result InfoNCE-anchor yields a plug-in MI estimator with significantly reduced bias.
This paper improves DNN generalization by accurately estimating mutual information.
problem Intractability of estimating mutual information in DNNs.
method Introduces a probabilistic representation of DNNs to accurately estimate mutual information.
result Derives a tighter generalization bound than previous relaxations.
Bayesian nonparametric framework improves robustness of MI estimation.
problem Challenges in exact MI computation in high dimensions with intractable likelihoods.
method Uses a Dirichlet process posterior to regularize MI loss, reducing sensitivity to fluctuations and outliers.
result Significant improvements in convergence over EDF-based methods, enhancing robustness and accuracy.
LMI approximates mutual information in high dimensions using learned low-dimensional representations.
problem Estimating mutual information between high-dimensional variables is challenging due to sample size limitations.
method Developed a method called latent MI (LMI) approximation that applies a nonparametric MI estimator to low-dimensional representations learned by a simple model architecture.
result LMI can approximate MI well for variables with >10^3 dimensions if their dependence structure has low intrinsic dimensionality.
Framework generates multimodal datasets with known MI for benchmarking.
problem Benchmarking mutual information estimators and SSL techniques.
method Flow-based generative model and structured causal framework.
result Regression performance improves with increasing MI between modalities.
Proposes CLUB for reliable MI minimization in high dimensions.
problem Estimating and minimizing mutual information in high-dimensional spaces.
method Contrastive Log-ratio Upper Bound (CLUB) for MI minimization.
result CLUB provides reliable estimation of mutual information.
InfoAtlas speeds up MI estimation for real-time data analysis.
problem Efficiently measuring statistical dependency between high-dimensional datasets.
method Directly infers mutual information in a single forward pass using a pretrained model.
result Matches state-of-the-art accuracy with 100x speedup.
Measuring Mutual Information (MI) between high-dimensional, continuous, random variables from observed samples has wide theoretical and practical applications. Recent work, MINE (Belghazi et al. 2018), focused on estimating tight variational lower bounds of MI using neural networks, but assumed unlimited supply of samp…
The Mutual Information (MI) is an often used measure of dependency between two random variables utilized in information theory, statistics and machine learning. Recently several MI estimators have been proposed that can achieve parametric MSE convergence rate. However, most of the previously proposed estimators have th…
We demonstrate that a popular class of nonparametric mutual information (MI) estimators based on k-nearest-neighbor graphs requires number of samples that scales exponentially with the true MI. Consequently, accurate estimation of MI between two strongly dependent variables is possible only for prohibitively large samp…
Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE ex…
The market impact (MI) of Volume Weighted Average Price (VWAP) orders is a convex function of a trading rate, but most empirical estimates of transaction cost are concave functions. How is this possible? We show that isochronic (constant trading time) MI is slightly convex, and isochoric (constant trading volume) MI is…
Paper connects contrastive learning to MI maximization and establishes robust methods for nonlinear ICA and subspace estimation.
problem Understanding and improving unsupervised representation learning and density ratio estimation.
method The paper connects contrastive learning to MI maximization, establishes new recovery conditions for nonlinear ICA, and proposes a practical outlier-robust method for nonlinear subspace estimation.
result The proposed methods can be seen as maximizing MI, performing nonlinear ICA, or estimating nonlinear subspaces, and are robust to outliers.
Improves BED scalability for implicit models.
problem Designing experiments for implicit models with intractable data distributions.
method Hybrid gradient approach combining variational MI estimator, ES, and SGA.
result Significantly improves scalability of BED for implicit models.
TMLE improves causal effect estimation in missing data scenarios with various positivity violations.
problem Estimating causal effects in studies with missing data and positivity violations.
method Targeted Maximum Likelihood Estimation (TMLE) with various missing data methods.
result Complete cases with TMLE incorporating an outcome-missingness model exhibit lower bias and greater robustness against positivity violations.
Many recent methods for unsupervised or self-supervised representation learning train feature extractors by maximizing an estimate of the mutual information (MI) between different views of the data. This comes with several immediate problems: For example, MI is notoriously hard to estimate, and using it as an objective…
New protocol makes neural MI estimators reliable in high-dimensional data.
problem Accurate estimation of mutual information in high-dimensional, undersampled data.
method Developed a practical protocol for neural MI estimators, incorporating statistical consistency checks, bias correction, and confidence intervals.
result Neural MI estimators can be made reliable when dependencies admit a low-dimensional latent representation.
Greedy MI maximization method outperforms existing approaches in nonlinear models.
problem Maximizing mutual information in nonlinear models with non-Gaussian noise.
method Greedy approaches based on log-Sobolev inequalities for computationally inexpensive MI lower bounds.
result Proposed method outperforms random selection and Gaussian approximations.
We stabilize MI-based losses by adding a regularization term, improving their performance and stability.
problem Instability of MI-based losses in machine learning.
method Added a novel regularization term to stabilize MI-based losses.
result Regularization stabilizes training and improves the performance of MI-based losses.
Study quantized models' privacy against membership inference attacks.
problem Privacy risk in quantized machine learning models.
method Proposed a new MIS indicator for post-training quantization procedures, minimizing empirical loss.
result Demonstrated effectiveness of new MIS indicator in assessing and ranking privacy risk.
Portable, Wearable and Wireless electrocardiogram (ECG) Systems have the potential to be used as point-of-care for cardiovascular disease diagnostic systems. Such wearable and wireless ECG systems require automatic detection of cardiovascular disease. Even in the primary care, automation of ECG diagnostic systems will …
Conventional mutual information (MI) based feature selection (FS) methods are unable to handle heterogeneous feature subset selection properly because of data format differences or estimation methods of MI between feature subset and class label. A way to solve this problem is feature transformation (FT). In this study,…
Study non-parametric value function estimation from a single path.
problem Estimating value function from a single trajectory in Markov reward processes.
method Kernel-based multi-step temporal difference (TD) estimates, including K-step look-ahead TD and TD(λ). result Non-asymptotic guarantees for TD estimates, capturing interactions between mixing time and model mis-specification.
New method detects information leakage using approximate Bayes predictor.
problem Unintentional exposure of sensitive information via observable data.
method Statistical learning theory and information theory framework, approximating Bayes predictor's log-loss and accuracy.
result MI can be accurately estimated to detect ILs, outperforming state-of-the-art baselines.
Mutual Information (MI) is often used for feature selection when developing classifier models. Estimating the MI for a subset of features is often intractable. We demonstrate, that under the assumptions of conditional independence, MI between a subset of features can be expressed as the Conditional Mutual Information (…
New insights link RLHF and contrastive learning for better model alignment.
problem Aligning large language models with human values.
method Interpreting RLHF and DPO as contrastive learning methods based on mutual information.
result Proposed Mutual Information Optimization (MIO) improves model performance.
Estimating and optimizing Mutual Information (MI) is core to many problems in machine learning; however, bounding MI in high dimensions is challenging. To establish tractable and scalable objectives, recent work has turned to variational bounds parameterized by neural networks, but the relationships and tradeoffs betwe…
Implicit stochastic models, where the data-generation distribution is intractable but sampling is possible, are ubiquitous in the natural sciences. The models typically have free parameters that need to be inferred from data collected in scientific experiments. A fundamental question is how to design the experiments so…
MINDE estimates Mutual Information using neural diffusion models.
problem Estimating Mutual Information between random variables.
method Score-based diffusion models to estimate Kullback Leibler divergence.
result MINDE is more accurate than existing methods, especially for challenging distributions.
Maximizes robustness in Bayesian experimental design under model uncertainty.
problem Brittleness of Bayesian experimental design under model misspecification.
method Formulates as a max--min game, uses Sibson's α-MI, and adopts PAC-Bayes framework.
result Establishes robust belief update and conditional information gain measure.
The Information Plane theory predicts autoencoders do not compress input information.
problem Understanding the training dynamics of hidden layers in autoencoders.
method Derive a theoretical convergence for the Information Plane of autoencoders using a Gram-matrix based mutual information estimator.
result Ideal autoencoders with a large bottleneck layer size do not compress input information, while a small size causes compression only in the encoder layers.
For all n, we define the n-dimensional critical catenoid Mn to be the unique rotationally symmetric, free boundary minimal hypersurface of non-trivial topology embedded in the closed unit ball in Rn+1. We show that the Morse index MI(n) of Mn satisfies the following asymptotic estimate as …
A scalable method for BED with implicit models using approximate gradients.
problem Efficiently estimating posterior distribution and maximizing MI for implicit models.
method Stochastic approximate gradient ascent with smoothed variational MI estimator.
result Significantly improves scalability of BED in high-dimensional problems.
We consider the problem of off-policy evaluation for reinforcement learning, where the goal is to estimate the expected reward of a target policy π using offline data collected by running a logging policy μ. Standard importance-sampling based approaches for this problem suffer from a variance that scales exponentia…
A new differential entropy estimator for neural networks training.
problem Lack of effective differential entropy estimators for neural network training.
method KNIFE: a fully parameterized, differentiable kernel-based estimator of differential entropy.
result KNIFE effectively estimates differential entropy and improves neural network training.
A new multi-label CPC method improves mutual information estimation and representation learning.
problem Underestimation of mutual information in contrastive predictive coding.
method Introducing a multi-label classification problem to overcome the logm bound in mutual information estimation. result The new method exceeds the logm bound and leads to better mutual information estimation and improved unsupervised representation learning. A method for finding most influential sets reduces a complex problem to a sequence of simpler top-k problems.
problem Identifying most influential subsets in complex models.
method Reduces the problem to a sequence of top-k problems using Dinkelbach's method. result The method returns a globally optimal set for the univariate ratio objective, including partial linear models.
This work provides guarantees for off-policy function estimation under realizability assumptions.
problem Estimating the value function of a policy under user-specified error-measuring distributions.
method The approach involves imposing a flexible regularization on the MIS objectives to account for an arbitrary user-specified distribution.
result Exact characterization of the optimal dual solution that determines the data-coverage assumption in the case of value-function learning.
Sensors which use electromagnetic induction (EMI) to excite a response in conducting bodies have long been investigated for subsurface explosive hazard detection. In particular, EMI sensors have been used to discriminate between different types of objects, and to detect objects with low metal content. One successful, p…
Causal discovery is a fundamental problem in statistics and has wide applications in different fields. Transfer Entropy (TE) is a important notion defined for measuring causality, which is essentially conditional Mutual Information (MI). Copula Entropy (CE) is a theory on measurement of statistical independence and is …
A typical goal of supervised dimension reduction is to find a low-dimensional subspace of the input space such that the projected input variables preserve maximal information about the output variables. The dependence maximization approach solves the supervised dimension reduction problem through maximizing a statistic…