The paper proposes methods to extract and analyze individual variable information from complex dependencies.
problem Analyzing and understanding complex dependencies between multiple variables.
method Reversible normalization and iterative dependency reduction to extract individual information, and use it for direct mutual information and multi-feature Granger causality analysis.
result Decoupling of variables to analyze their individual information and direct mutual information transfers.
CIB compresses variables causally, preserving key causal interactions.
problem Constructing causal variable abstractions in complex systems.
method Causal Information Bottleneck (CIB) method, extending IB to include causal structures.
result CIB produces causally interpretable abstractions that accurately capture causal relations.
Statistical inference is considered for variables of interest, called primary variables, when auxiliary variables are observed along with the primary variables. We consider the setting of incomplete data analysis, where some primary variables are not observed. Utilizing a parametric model of joint distribution of prima…
Identification of informative variables in an information system is often performed using simple one-dimensional filtering procedures that discard information about interactions between variables. Such approach may result in removing some relevant variables from consideration. Here we present an R package MDFS (MultiDi…
User releases data to service provider while balancing privacy and utility.
problem Balancing user privacy and service utility in data release.
method Formulated as a Markov decision process (MDP) and solved using deep reinforcement learning (RL).
result Achieved a trade-off between revealing useful information and protecting sensitive data.
The muti-layer information bottleneck (IB) problem, where information is propagated (or successively refined) from layer to layer, is considered. Based on information forwarded by the preceding layer, each stage of the network is required to preserve a certain level of relevance with regards to a specific hidden variab…
New causal measures improve feature selection in AI models.
problem Lack of causal interpretability in AI models.
method Introduces causal entropy and causal information gain to assess feature control.
result Demonstrates superiority of causal information gain in feature selection.
This work presents entropic constraints from DAGs with hidden variables.
problem Characterizing causal relations in systems with hidden variables.
method Entropic inequality constraints derived from e-separation relations. result These constraints can learn about true causal models from observed data.
This article describes the R package varrank. It has a flexible implementation of heuristic approaches which perform variable ranking based on mutual information. The package is particularly suitable for exploring multivariate datasets requiring a holistic analysis. The core functionality is a general implementation of…
Measuring the relationship between any pair of variables is a rich and active area of research that is central to scientific practice. In contrast, characterizing the common information among any group of variables is typically a theoretical exercise with few practical methods for high-dimensional data. A promising sol…
One of the most fundamental questions one can ask about a pair of random variables X and Y is the value of their mutual information. Unfortunately, this task is often stymied by the extremely large dimension of the variables. We might hope to replace each variable by a lower-dimensional representation that preserves th…
Sharp concentration bounds for i.i.d. variables.
problem Controlling the tail probabilities of independent variables.
method Extension of Sanov's theorem using large deviations and information theory.
result Matching concentration and anti-concentration bounds for i.i.d. samples of any size.
Williams and Beer (2010) proposed a nonnegative mutual information decomposition, based on the construction of redundancy lattices, which allows separating the information that a set of variables contains about a target variable into nonnegative components interpretable as the unique information of some variables not p…
Information-theoretic quantities, such as entropy, are used to quantify the amount of information a given variable provides. Entropies can be used together to compute the mutual information, which quantifies the amount of information two variables share. However, accurately estimating these quantities from data is extr…
Tensor-based method simplifies causal skeleton discovery.
problem Discover causal relationships between variables.
method Express associations as tensors to reduce dimensionality.
result Causal skeleton can be determined using pair-wise tensors.
The matrix-based Renyi's α-order entropy functional was recently introduced using the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the current theory in the matrix-based Renyi's α-order entropy functional only defines the entropy of a single…
New method uses sufficient statistics to infer causal relationships from observational data.
problem Inferring causal relationships from observational data with hidden variables.
method Information Bottleneck method applied to find functional sufficient statistics.
result New causal rules not obtainable from standard methods, validated on simulated and real data.
Paper shows noisy labels can improve PLR variable selection.
problem Variable selection in PLR is challenging due to noisy labels.
method Proposes a novel ADMM-based algorithm to fuse noisy labels.
result Fused noisy labels improve PLR performance in estimation and classification.
New algorithm selects relevant variables in high-dimensional graphical models.
problem Automatic selection of relevant variables in high-dimensional graphical models.
method Extends Chow and Liu's algorithm using mutual information and entropy coefficient of determination.
result Outperforms existing methods in selecting variables with explanatory power.
Solar algorithm selects variables faster and more accurately in high-dimensional data.
problem Variable selection in high-dimensional data with high accuracy and stability.
method Subsample-ordered least-angle regression (solar) and its coordinate descent generalization (solar-cd) using L0 norm solution path averaging. result Solar selects variables with high accuracy and stability, reducing redundant variable selection.
High-dimensional, large-sample astrophysical databases of galaxy clusters, such as the Chandra Deep Field South COMBO-17 database, provide measurements on many variables for thousands of galaxies and a range of redshifts. Current understanding of galaxy formation and evolution rests sensitively on relationships between…
Research builds an index measuring analysts' perception of informational asymmetry.
problem Measuring the level of informational asymmetry among companies.
method Developed an algorithm based on Elo rating to capture analysts' perception.
result The model shows good fit with significant variables: coverage, volatility, Tobin q, and size.
In many cases, feature selection is often more complicated than identifying a single subset of input variables that would together explain the output. There may be interactions that depend on contextual information, i.e., variables that reveal to be relevant only in some specific circumstances. In this setting, the con…
New model learns symmetry transformations from complex data.
problem Learning symmetry transformations in complex domains like chemical space.
method Two latent subspaces, deep information bottleneck, continuous mutual information regularizer.
result Model outperforms state-of-the-art methods on artificial and molecular datasets.
New framework improves multivariate time series forecasting by minimizing redundant information.
problem Improving multivariate time series forecasting with deep learning techniques.
method Cross-variable Decorrelation Aware feature Modeling (CDAM) and Temporal correlation Aware Modeling (TAM) to refine Channel-mixing and exploit temporal correlations.
result Significantly surpasses existing models in comprehensive tests.
TCMI assesses mutual dependence of continuous variables without parametric assumptions.
problem Estimating mutual information from continuous distributions.
method TCMI extends mutual information to continuous variables using cumulative distributions.
result TCMI facilitates feature selection and ranking of variable sets.
Improved Monte-Carlo models by constraining mutual information between latent and observable variables.
problem Training density models leads to latent variables being useless.
method Weave tighter Monte-Carlo bounds with mutual information constraints.
result Improved training of models with continuous and discrete latent variables.
A new method clusters mixed-type data efficiently.
problem Clustering mixed-type data with continuous and categorical variables.
method Extends Information Bottleneck principle to heterogeneous data using generalised product kernels.
result DIBmix outperforms four established methods in various scenarios.
Study uses non-parametric method to analyze EU ETS price determinants.
problem Understanding price determinants of EU ETS to inform policy.
method Non-parametric measure (Information Imbalance) to study variables.
result Commodity variables are most informative in Phase 3, while financial variables become more important in Phase 4.
SentenceMIM learns rich latent representations for variable-length language data.
problem Challenges in learning VAEs for variable-length language data, especially posterior collapse.
method Probabilistic auto-encoder trained with Mutual Information Machine (MIM) learning.
result SentenceMIM learns informative latent representations with high mutual information.
Learning with hidden variables is a central challenge in probabilistic graphical models that has important implications for many real-life problems. The classical approach is using the Expectation Maximization (EM) algorithm. This algorithm, however, can get trapped in local maxima. In this paper we explore a new appro…
Information bottleneck (IB) is a technique for extracting information in one random variable X that is relevant for predicting another random variable Y. IB works by encoding X in a compressed "bottleneck" random variable M from which Y can be accurately decoded. However, finding the optimal bottleneck variab…
Self-Distilled Disentanglement improves counterfactual predictions by separating variables.
problem Improving counterfactual predictions in the presence of confounders and unobserved variables.
method Self-Distilled Disentanglement framework based on information theory.
result Effective counterfactual inference in synthetic and real-world datasets.
The paper explores how agents can generalize to new environments with unseen variables.
problem Generalizing to new environments with unseen variables.
method Investigates and proposes a method for efficient re-use of past marginal information to achieve out-of-variable generalization.
result The residual distribution after fitting a classifier reveals partial derivatives of the true generating function with respect to unobserved causal parents.
Proposes a framework to improve VAE latent codes using mutual information.
problem Lack of explicit measure for VAE latent variable quality.
method Variational Mutual Information Maximization Framework.
result Improves relationships between latent codes and observations.
Meta-learning performance is affected by how task diversity is allocated, not just overall variability.
problem Meta-learning performance degrades when task diversity is unevenly distributed.
method Decomposed task-specific regression effects into structurally informative and orthogonal components.
result Meta-learning prediction degrades when a larger fraction of task variability is orthogonal and non-informative.
Proposes PEID for analyzing synergistic causation in complex systems.
problem Challenges in identifying and analyzing synergistic causation in complex systems.
method Partial Effective Information Decomposition (PEID) framework.
result Unified and computable characterization of synergistic causal relations.
We formulate and solve a tensor model using a latent-variable approach.
problem Parameter inference for Poisson canonical polyadic tensor models.
method Latent-variable formulation, Expectation-Maximization algorithms, Fisher information matrices.
result Derivation of Fisher information for PCP models, insights into model well-posedness.
fastHDMI improves neuroimaging variable selection in high-dimensional data.
problem Efficient variable screening in high-dimensional neuroimaging datasets.
method Three mutual information estimation methods implemented in fastHDMI.
result FFTKDE-based method superior for continuous nonlinear outcomes.
A new model captures variability in time series data.
problem Capturing high variability in time series data.
method Temporal latent variables and dynamic weight modifications.
result Demonstrated efficacy on various sequential data.
This paper extends transfer learning for linear regression to uncertain domain information.
problem Transfer learning for linear regression with uncertain domain information.
method A Dirichlet process is used to infer latent domain information from regression coefficients. A novel framework considers the joint distribution of variables.
result The proposed method controls bias better than previous pseudo-labelling approaches.
New bound on machine learning model performance using Jensen-Shannon information.
problem Understanding the performance of machine learning models.
method Proposes a new information-theoretic bound on generalization error.
result Shows that the new bound can be tighter than mutual information-based bounds under certain conditions.
DIET tests conditional independence using marginal dependence measures of residual information.
problem Computational intractability of conditional randomization tests (CRTs).
method DIET avoids fitting large models by leveraging marginal independence statistics of information residuals.
result DIET achieves higher power than other tractable CRTs on synthetic and real benchmarks.
MSRL learns a representation maximizing mutual info with response variables.
problem Learning sufficient representations for complex, multi-dimensional data.
method Variational mutual information, deep neural networks, generalized Dudley's inequality.
result MSRL achieves consistent and accurate representation learning.
Approximate inference via information projection has been recently introduced as a general-purpose approach for efficient probabilistic inference given sparse variables. This manuscript goes beyond classical sparsity by proposing efficient algorithms for approximate inference via information projection that are applica…
InfoBridge uses diffusion bridges to estimate mutual information accurately.
problem Estimating mutual information between random variables.
method Formulated mutual information estimation as a domain transfer problem using diffusion bridge models.
result Demonstrated unbiased estimator for various data types.
Recent developments have linked causal inference with Algorithmic Information Theory, and methods have been developed that utilize Conditional Kolmogorov Complexity to determine causation between two random variables. We present a method for inferring causal direction between continuous variables by using an MDL Binnin…
New method separates latent variables from noise in generative processes.
problem Reconstructing true latent variables from complex generative processes.
method General Incompressible-flow Network (GIN) for nonlinear ICA.
result Informative latent variables are automatically separated from noise.