arXiv research

Mutual information minimum spanning trees are used to explore nonlinear dependencies on Brazilian equity network in the periods from June/01/2015 to January/26/2016, in which Brazil was under the government of President Dilma Rousseff, and from January/27/2016 to September/08/2016 which includes the government transiti…

Exponential models of distributions are widely used in machine learning for classiffication and modelling. It is well known that they can be interpreted as maximum entropy models under empirical expectation constraints. In this work, we argue that for classiffication tasks, mutual information is a more suitable informa…

The paper tackles adversarial robustness by maximizing worst-case mutual information.

Defines MER for Bayesian learning, a gap between achievable and optimal performance.

The Partial Information Decomposition (PID) [arXiv:1004.2515] provides a theoretical framework to characterize and quantify the structure of multivariate information sharing. A new method (Idep) has recently been proposed for computing a two-predictor PID over discrete spaces. [arXiv:1709.06653] A lattice of maximum en…

Lumbermark clusters data robustly, slicing limbs of mutual reachability trees.

Paper finds formulas for mutual information and MMSE in matrix tensor product problems.

This article describes the R package varrank. It has a flexible implementation of heuristic approaches which perform variable ranking based on mutual information. The package is particularly suitable for exploring multivariate datasets requiring a holistic analysis. The core functionality is a general implementation of…

In this work, three lattice-free (LF) discriminative training criteria for purely sequence-trained neural network acoustic models are compared on LVCSR tasks, namely maximum mutual information (MMI), boosted maximum mutual information (bMMI) and state-level minimum Bayes risk (sMBR). We demonstrate that, analogous to L…

Study entropic regularization of Gaussian measures and processes on Hilbert space.

Feature selection aims to select the smallest feature subset that yields the minimum generalization error. In the rich literature in feature selection, information theory-based approaches seek a subset of features such that the mutual information between the selected features and the class labels is maximized. Despite …

The paper introduces V(I) to guide algorithm choice and parameter tuning in financial forecasting.

A new method improves ICA performance by approximating MDI.

Study finds exact limits for sparse regression with fewer observations than usual.

In this paper, we explore the detection of clusters of stocks that are in synergy in the Indian Stock Market and understand their behaviour in different circumstances. We have based our study on high frequency data for the year 2014. This was a year when general elections were held in India, keeping this in mind our da…

Conventional mutual information (MI) based feature selection (FS) methods are unable to handle heterogeneous feature subset selection properly because of data format differences or estimation methods of MI between feature subset and class label. A way to solve this problem is feature transformation (FT). In this study,…

New neural network approach using mutual information.

Paper describes profiles of multivariate normal distributions and novel estimators for mutual information.

Improved bounds on learning algorithms' performance using conditional mutual information.

Paper benchmarks mutual info estimators on diverse distributions.

The paper argues that normalized mutual information is biased in clustering and community detection.

Neural estimator improves mutual information estimation in high dimensions.

We find the maximum mutual information for neural networks and its key determinants.

DC-SIS selects features faster than mRMR for Parkinson's vocal diagnosis.

Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confide…

Improved method for encoding contingency tables reduces mutual information bias.

Proposes a new bound on generalization error using conditional mutual information.

The paper introduces submodular information measures for machine learning applications.

Proposes a framework to maximize mutual information in VAE models for better latent code representation.

We introduce the Mutual Information Machine (MIM), a novel formulation of representation learning, using a joint distribution over the observations and latent state in an encoder/decoder framework. Our key principles are symmetry and mutual information, where symmetry encourages the encoder and decoder to learn differe…

A new method estimates mutual information using neural classifiers.

New estimator improves mutual information estimation.

The paper proposes a method to learn structured representations from unlabeled data using mutual information maximization.

Paper proposes mutual information learning for deep learning classifiers.

Paper establishes generalization bounds for representation learning using Minimum Description Length.

We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, an…

Reshef et al. recently proposed a new statistical measure, the "maximal information coefficient" (MIC), for quantifying arbitrary dependencies between pairs of stochastic quantities. MIC is based on mutual information, a fundamental quantity in information theory that is widely understood to serve this need. MIC, howev…

Sequence models assign probabilities to variable-length sequences such as natural language texts. The ability of sequence models to capture temporal dependence can be characterized by the temporal scaling of correlation and mutual information. In this paper, we study the mutual information of recurrent neural networks …

Active feature selection uses mutual information to choose fewer labels for better feature selection.

In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex lo…

In this paper, we present a new approach to interpret deep learning models. By coupling mutual information with network science, we explore how information flows through feedforward networks. We show that efficiently approximating mutual information allows us to create an information measure that quantifies how much in…

This paper improves DNN generalization by accurately estimating mutual information.

The use of variable selection methods is particularly appealing in statistical problems with functional data. The obvious general criterion for variable selection is to choose the `most representative' or `most relevant' variables. However, it is also clear that a purely relevance-oriented criterion could lead to selec…

Novel mutual information bound improves statistical inference rates.

Study reveals mutual information is crucial for understanding algorithm performance in stochastic convex optimization.

InfoBridge uses diffusion bridges to estimate mutual information accurately.

Mutual information is widely applied to learn latent representations of observations, whilst its implication in classification neural networks remain to be better explained. We show that optimising the parameters of classification neural networks with softmax cross-entropy is equivalent to maximising the mutual informa…

New method estimates mutual information using normalizing flows.