InfoBridge uses diffusion bridges to estimate mutual information accurately.
problem Estimating mutual information between random variables.
method Formulated mutual information estimation as a domain transfer problem using diffusion bridge models.
result Demonstrated unbiased estimator for various data types.
Neural estimator improves mutual information estimation in high dimensions.
problem Estimating mutual information in high dimensions is challenging.
method Parametrizing conditional densities with normalizing flows and using block autoregressive structure.
result Improved mutual information estimation on benchmark tasks.
Paper describes profiles of multivariate normal distributions and novel estimators for mutual information.
problem Estimating mutual information for complex distributions.
method Analytical description of profiles, introduction of Bend and Mix Models, Monte Carlo estimation.
result Bend and Mix Models accurately estimate mutual information profiles and provide Bayesian estimates.
New estimator improves mutual information estimation.
problem Estimating mutual information in data science and machine learning.
method Proposes a new estimator that uses a preliminary estimate of the data distribution.
result A preliminary estimate helps in estimating mutual information more accurately.
Identifying features that leak information about sensitive attributes is a key challenge in the design of information obfuscation mechanisms. In this paper, we propose a framework to identify information-leaking features via information density estimation. Here, features whose information densities exceed a pre-defined…
Estimates conditional mutual information using a minmax formulation.
problem Estimating conditional mutual information in high dimensions.
method Uses a minmax optimization problem to train a neural network.
result Improves estimation accuracy compared to existing methods.
Paper benchmarks mutual info estimators on diverse distributions.
problem Evaluating mutual information estimators on complex, real-world distributions.
method Constructs a diverse family of known-ground truth distributions, proposes a benchmark platform.
result Highlights differences in classical and neural estimators' performance across various conditions.
New method estimates mutual information using normalizing flows.
problem Mutual information estimation in high-dimensional data.
method Normalizing flows to map data to target distributions with known MI.
result Theoretical guarantees and practical advantages demonstrated.
Optimizes information acquisition to reduce estimation risk and maximize utility.
problem Estimation risk in investor decision-making.
method Derives closed-form value functions using CARA and CRRA utility functions, employs variational methods to explore optimal acquisition.
result Acquiring information earlier is more valuable in reducing estimation risk and achieving higher utility.
Paper improves Fisher information estimation methods.
problem Estimating Fisher information for location parameters.
method Revisits and improves Bhattacharya estimator, introduces clipped estimator.
result Clipped estimator shows superior convergence rates in Gaussian noise.
Efficiently estimates distributed mean with side information, near-optimal and universal.
problem Distributed mean estimation with side information in communication constrained settings.
method Wyner-Ziv estimators for communication and computation efficiency.
result Near-optimal and universal recovery guarantees for distributed optimization and compression.
A new method estimates mutual information using neural classifiers.
problem Estimating mutual information for high-dimensional data is challenging.
method Trains a classifier to estimate joint distribution probability.
result Demonstrates high accuracy and reduces variance compared to variational methods.
Hybrid method improves mutual information estimation from samples.
problem Estimating mutual information from joint distributions is challenging.
method Proposes a hybrid method combining discriminative and generative approaches.
result Hybrid methods yield tighter variational bounds on mutual information.
Deep learning model estimates mutual information with low bias and variance.
problem Estimating mutual information between continuous variables.
method Supervised deep learning approach using Linfoot informational correlation as labels.
result Lower bias and variance compared to other methods.
Proposes MIGE for accurate MI gradient estimation in high-dimensional settings.
problem Intractability of MI in continuous and high-dimensional settings.
method Score estimation of implicit distributions for gradient estimation of MI.
result MIGE provides tight and smooth gradient estimation of MI in high-dimensional settings.
This paper improves DNN generalization by accurately estimating mutual information.
problem Intractability of estimating mutual information in DNNs.
method Introduces a probabilistic representation of DNNs to accurately estimate mutual information.
result Derives a tighter generalization bound than previous relaxations.
LogDet estimator improves entropy estimation in neural networks.
problem Inconsistent observations and diversified interpretation in neural networks.
method Proposes LogDet estimator for reliable entropy approximation.
result LogDet estimator overcomes distributional diversity issues.
Estimates interactions between modalities for multimodal data.
problem Accurately quantifying interactions between different data types.
method Developed Lightweight Sample-wise Multimodal Interaction (LSMI) estimator using pointwise information theory.
result LSMI reveals fine-grained dynamics in multimodal data.
MIC consistently estimates dependence in large datasets.
problem Estimating dependence between variable pairs in large datasets.
method Proving consistency of MIC as an estimator.
result MIC is a consistent estimator of population statistic MIC*.
Two Fisher information matrix estimators are analyzed for neural networks, focusing on their variances and trade-offs.
problem Estimating the Fisher information matrix in neural networks due to its high computational cost.
method Examined two popular diagonal Fisher information matrix estimators and their variances in neural networks for regression and classification.
result The variances of the estimators depend on the non-linearity with respect to different parameter groups and should not be neglected.
Review of information plane analyses in neural networks, highlighting mixed results and methodological challenges.
problem Understanding the relationship between information-theoretic compression and neural network performance.
method Literature review and detailed analysis of information quantity estimation methods.
result Information plane compression is not necessarily information-theoretic but compatible with geometric compression.
Paper presents estimators for entropy and information in probabilistic models.
problem Estimating entropy and mutual information in high dimensions is challenging.
method EEVI uses importance sampling with proposal distributions like amortized variational inference and sequential Monte Carlo.
result EEVI delivers accurate upper and lower bounds on information quantities.
Study robust mean estimation under coordinate-level corruptions using Hamming distance.
problem Robust mean estimation under realistic coordinate-level corruptions.
method Introduce a novel Hamming distance-based measure and present information-theoretic analysis.
result Data cleaning-inspired approaches can match information theoretic bounds for robust mean estimation.
Estimates expected information gain using density approximations and dimension reduction.
problem Estimating expected information gain in nonlinear and non-Gaussian settings.
method Flexible transport-based schemes for EIG estimation, optimal sample allocation, and gradient-based upper bounds on mutual information.
result Optimal sample allocation and dimension reduction schemes improve EIG estimation accuracy and convergence rate.
Estimates latent dimensionality for prediction tasks using mutual information.
problem Estimating the latent dimensionality needed for accurate prediction.
method Formulates the problem as an Information Bottleneck question and uses neural mutual information estimators with a hybrid critic to preserve latent geometry.
result The hybrid critic method provides a more accurate estimation of task-relevant dimensionality.
REMEDI improves neural entropy estimation across various tasks.
problem Challenges in estimating information theoretic quantities in high-dimensional data.
method Combines minimization of cross-entropy with estimation of deviation from data density.
result Improves accuracy in entropy estimation on synthetic and natural data.
Novel mutual information bound improves statistical inference rates.
problem Improving statistical inference rates in Bayesian nonparametrics.
method Introduces a novel mutual information bound.
result Improved contraction rates for fractional posteriors.
Framework for robust matrix estimation with side information.
problem High-dimensional matrix estimation with restrictive structure.
method Flexible framework decomposing matrix into four components: interaction, row, column, and residual.
result Improved imputation accuracy and treatment-effect estimation with side information.
KSG mutual information estimator, which is based on the distances of each sample to its k-th nearest neighbor, is widely used to estimate mutual information between two continuous random variables. Existing work has analyzed the convergence rate of this estimator for random variables whose densities are bounded away fr…
We consider the question of efficient estimation in the tails of Gaussian copulas. Our special focus is estimating expectations over multi-dimensional constrained sets that have a small implied measure under the Gaussian copula. We propose three estimators, all of which rely on a simple idea: identify certain \emph{dom…
In the information-based paradigm of inference, model selection is performed by selecting the candidate model with the best estimated predictive performance. The success of this approach depends on the accuracy of the estimate of the predictive complexity. In the large-sample-size limit of a regular model, the predicti…
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, an…
Paper extends information theory for efficient probabilistic modeling.
problem Efficient and data-efficient non-parametric density estimation.
method Structured generative model (SGM) using Rényi's information.
result SGM improves mutual information estimation and generative adversarial networks.
This paper improves error estimation in covariate shift by incorporating target information.
problem Error estimation is inaccurate in covariate shift scenarios.
method Proposes a redefinition of importance using target information for better error estimation.
result Incorporating target information leads to more accurate error estimation, especially with KLIEP.
Study shows the corrected Akaike criterion is inadmissible for estimating Kullback-Leibler discrepancy.
problem Inadmissibility of the corrected Akaike information criterion for estimating Kullback-Leibler discrepancy.
method Loss estimation framework to demonstrate inadmissibility and provide improved estimators.
result Improved estimators of Kullback-Leibler discrepancy are provided and perform well in reduced-rank situations.
Method estimates group structure in panel data using variance information.
problem Estimating group structure in panel data with unknown groups.
method Proposes a method to estimate unobserved groupings for panel data models using variance information.
result Superior performance compared to existing methods in simulations and empirical applications.
We consider a framework involving behavioral economics and machine learning. Rationally inattentive Bayesian agents make decisions based on their posterior distribution, utility function and information acquisition cost Renyi divergence which generalizes Shannon mutual information). By observing these decisions, how ca…
Information about intrinsic dimension is crucial to perform dimensionality reduction, compress information, design efficient algorithms, and do statistical adaptation. In this paper we propose an estimator for the intrinsic dimension of a data set. The estimator is based on binary neighbourhood information about the ob…
Several methods of estimating the mutual information of random variables have been developed in recent years. They can prove valuable for novel approaches to learning statistically independent features. In this paper, we use one of these methods, a mutual information neural estimation (MINE) network, to present a proof…
The conditional mutual information I(X;Y|Z) measures the average information that X and Y contain about each other given Z. This is an important primitive in many learning problems including conditional independence testing, graphical model inference, causal strength estimation and time-series problems. In several appl…
Information-theoretic quantities, such as conditional entropy and mutual information, are critical data summaries for quantifying uncertainty. Current widely used approaches for computing such quantities rely on nearest neighbor methods and exhibit both strong performance and theoretical guarantees in certain simple sc…
Paper improves ML estimation from incomplete data with robust M-estimator.
problem Estimating parameters from incomplete data with improved accuracy.
method Developed a robust M-estimator and a sandwich estimator for standard errors.
result Improved estimation accuracy with smaller standard errors than ML estimates.
New method detects information leakage using approximate Bayes predictor.
problem Unintentional exposure of sensitive information via observable data.
method Statistical learning theory and information theory framework, approximating Bayes predictor's log-loss and accuracy.
result MI can be accurately estimated to detect ILs, outperforming state-of-the-art baselines.
Optimizes kernel density ratios for better predictions and information measures.
problem Improving accuracy of kernel density estimates for density ratios.
method Derives an optimal weight function using calculus of variations.
result Reduces bias in kernel density estimates, leading to improved prediction posteriors and information-theoretic measures.
We study the problem of using i.i.d. samples from an unknown multivariate probability distribution p to estimate the mutual information of p. This problem has recently received attention in two settings: (1) where p is assumed to be Gaussian and (2) where p is assumed only to lie in a large nonparametric smooth…
Ensembles of classification and regression trees remain popular machine learning methods because they define flexible non-parametric models that predict well and are computationally efficient both during training and testing. During induction of decision trees one aims to find predicates that are maximally informative …
New bounds estimate learning algorithm performance using prediction information.
problem Estimating the performance of black-box learning algorithms.
method Information-theoretic bounds based on prediction information.
result Improved bounds applicable to deterministic algorithms and easier to estimate.
New framework converts offline to online estimation using black-box offline estimators.
problem Convert offline estimation algorithms to online estimation algorithms.
method Oracle-Efficient Online Estimation (OEOE) framework.
result Achieves near-optimal online estimation error via black-box offline estimators.