A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
New estimator improves mutual information estimation.
We introduce the Mutual Information Machine (MIM), a novel formulation of representation learning, using a joint distribution over the observations and latent state in an encoder/decoder framework. Our key principles are symmetry and mutual information, where symmetry encourages the encoder and decoder to learn differe…
We introduce the Mutual Information Machine (MIM), a probabilistic auto-encoder for learning joint distributions over observations and latent variables. MIM reflects three design principles: 1) low divergence, to encourage the encoder and decoder to learn consistent factorizations of the same underlying distribution; 2…
The conditional mutual information I(X;Y|Z) measures the average information that X and Y contain about each other given Z. This is an important primitive in many learning problems including conditional independence testing, graphical model inference, causal strength estimation and time-series problems. In several appl…
A privacy-constrained information extraction problem is considered where for a pair of correlated discrete random variables governed by a given joint distribution, an agent observes and wants to convey to a potentially public user as much information about as possible without compromising the amount of …
New model learns from missing modalities and class labels.
Hybrid method improves mutual information estimation from samples.
Estimates conditional mutual information using a minmax formulation.
A new method estimates mutual information using neural classifiers.
Paper proposes a novel method to reduce mutual information for missing data imputation.
We consider the problem of universal joint clustering and registration of images and define algorithms using multivariate information functionals. We first study registering two images using maximum mutual information and prove its asymptotic optimality. We then show the shortcomings of pairwise registration in multi-i…
AMI framework improves text generation by optimizing mutual information between source and target.
Proposes a framework to improve VAE latent codes using mutual information.
New MI bounds improve estimation in deep generative models.
The identification of relevant features, i.e., the driving variables that determine a process or the properties of a system, is an essential part of the analysis of data sets with a large number of variables. A mathematical rigorous approach to quantifying the relevance of these features is mutual information. Mutual i…
Novel bounds for deep MDA algorithms improve performance and efficiency.
BatchGFN uses generative flow networks for efficient batch active learning.
A new multi-label CPC method improves mutual information estimation and representation learning.
Researchers develop methods to learn neuron dynamics from colored noise.
A broad range of cross--domain generation researches boil down to matching a joint distribution by deep generative models (DGMs). Hitherto algorithms excel in pairwise domains while as increases, remain struggling to scale themselves to fit a joint distribution. In this paper, we propose a domain-scalable DGM, i…
We propose a test of independence of two multivariate random vectors, given a sample from the underlying population. Our approach, which we call MINT, is based on the estimation of mutual information, whose decomposition into joint and marginal entropies facilitates the use of recently-developed efficient entropy estim…
MINDE estimates Mutual Information using neural diffusion models.
Proposes a method to select features for subgroup datasets with systematic missing data.
Estimates copula density for complex data distributions.
We demonstrate that a popular class of nonparametric mutual information (MI) estimators based on k-nearest-neighbor graphs requires number of samples that scales exponentially with the true MI. Consequently, accurate estimation of MI between two strongly dependent variables is possible only for prohibitively large samp…
We are working to develop automated intelligent agents, which can act and react as learning machines with minimal human intervention. To accomplish this, an intelligent agent is viewed as a question-asking machine, which is designed by coupling the processes of inference and inquiry to form a model-based learning unit.…
New scheme optimizes BMI through probabilistic and geometric shaping.
The article introduces inferential moments for analyzing uncertain multivariable systems.
In data science, it is often required to estimate dependencies between different data sources. These dependencies are typically calculated using Pearson's correlation, distance correlation, and/or mutual information. However, none of these measures satisfy all the Granger's axioms for an "ideal measure". One such ideal…
Sparse GEMINI selects relevant features for clustering without assumptions.
New bounds derived for machine learning algorithms using convex functions.
Study entropic regularization of Gaussian measures and processes on Hilbert space.
We propose a simple, tractable lower bound on the mutual information contained in the joint generative density of any latent variable generative model: the GILBO (Generative Information Lower BOund). It offers a data-independent measure of the complexity of the learned latent variable description, giving the log of the…
MIND estimates mutual information from ordinal data without full distributional knowledge.
Study of Bayes optimal learning in high-dimensional linear regression with network side information.
In this work, we investigate the use of three information-theoretic quantities -- entropy, mutual information with the class variable, and a class selectivity measure based on Kullback-Leibler divergence -- to understand and study the behavior of already trained fully-connected feed-forward neural networks. We analyze …
Combining the Information Bottleneck model with deep learning by replacing mutual information terms with deep neural nets has proved successful in areas ranging from generative modelling to interpreting deep neural networks. In this paper, we revisit the Deep Variational Information Bottleneck and the assumptions neede…
A new method is proposed to compute connectivity measures on multivariate time series with gaps. Rather than removing or filling the gaps, the rows of the joint data matrix containing empty entries are removed and the calculations are done on the remainder matrix. The method, called measure adapted gap removal (MAGR), …
Information-theoretic quantities, such as entropy, are used to quantify the amount of information a given variable provides. Entropies can be used together to compute the mutual information, which quantifies the amount of information two variables share. However, accurately estimating these quantities from data is extr…
Flaky performance found in GNN SSL on RDBs, leading to worse linear evaluation.
The matrix-based Renyi's α-order entropy functional was recently introduced using the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the current theory in the matrix-based Renyi's α-order entropy functional only defines the entropy of a single…
The Partial Information Decomposition (PID) [arXiv:1004.2515] provides a theoretical framework to characterize and quantify the structure of multivariate information sharing. A new method (Idep) has recently been proposed for computing a two-predictor PID over discrete spaces. [arXiv:1709.06653] A lattice of maximum en…
A new algorithm improves efficiency in selecting examples for deep learning.
New neural network approach using mutual information.
Complex systems often exhibit multiple levels of organization covering a wide range of physical scales, so the study of the hierarchical decomposition of their structure and function is frequently convenient. To better understand this phenomenon, we introduce a generalization of information theory that works with hiera…
Paper describes profiles of multivariate normal distributions and novel estimators for mutual information.
Transformation Equivariant Representations (TERs) aim to capture the intrinsic visual structures that equivary to various transformations by expanding the notion of {\em translation} equivariance underlying the success of Convolutional Neural Networks (CNNs). For this purpose, we present both deterministic AutoEncoding…
Improved bounds on learning algorithms' performance using conditional mutual information.