Derives a new variational approach to information bottleneck.
problem Information theoretic inference and predictive modeling.
method Variational lower bound of predictive information bottleneck.
result Generalizes modern inference procedures and suggests new ones.
New method quantifies redundant information using information bottleneck.
problem Quantifying redundant information among multiple sources.
method Formulated as an information bottleneck problem, termed redundancy bottleneck.
result Extracts information that best predicts the target without revealing source identity.
Proposes Decodable Information Bottleneck for optimal representation learning.
problem Finding optimal representations for supervised learning.
method Integrates information retention and compression with the desired predictive family.
result Optimal representations lead to better expected test performance and can be estimated with guarantees.
Extends machine learning model training to include validation data.
problem Machine learning validation data usage in training.
method Extends predictive information bottleneck to include validation data.
result Validation data improves model training efficiency.
Estimates latent dimensionality for prediction tasks using mutual information.
problem Estimating the latent dimensionality needed for accurate prediction.
method Formulates the problem as an Information Bottleneck question and uses neural mutual information estimators with a hybrid critic to preserve latent geometry.
result The hybrid critic method provides a more accurate estimation of task-relevant dimensionality.
Estimating individual level treatment effects (ITE) from observational data is a challenging and important area in causal machine learning and is commonly considered in diverse mission-critical applications. In this paper, we propose an information theoretic approach in order to find more reliable representations for e…
The paper studies phase transitions in Information Bottleneck for representation learning.
problem Understanding the behavior of compression and prediction terms in IB objective.
method Studied phase transitions in IB objective using second-order calculus of variations and Fisher information matrix.
result IB phase transitions correspond to learning new classes and are related to maximum correlation between input and target orthogonal to the learned representation.
CIB compresses variables causally, preserving key causal interactions.
problem Constructing causal variable abstractions in complex systems.
method Causal Information Bottleneck (CIB) method, extending IB to include causal structures.
result CIB produces causally interpretable abstractions that accurately capture causal relations.
PI-SAC agents learn predictive information to improve RL efficiency.
problem Improving sample efficiency in reinforcement learning.
method PI-SAC agents use a contrastive version of Conditional Entropy Bottleneck to learn predictive information from past and future states.
result PI-SAC agents significantly improve sample efficiency on challenging continuous control tasks.
Extends information bottleneck to multi-view unsupervised learning.
problem Identifying superfluous information in unlabeled multi-view data.
method Multi-view information bottleneck model, leveraging data augmentation.
result State-of-the-art results on Sketchy and MIR-Flickr datasets.
AIB method improves robustness against adversarial perturbations.
problem Optimizing the IB principle for better robustness and understanding compression effects.
method Proposes adversarial information bottleneck (AIB) method to optimize IB principle without explicit distribution assumptions.
result Demonstrates effectiveness in learning more invariant representations and mitigating adversarial perturbations.
VIB balances empirical and Bayesian approaches in predictive models.
problem Balancing empirical and Bayesian methods in predictive models.
method VIB as a compromise between empirical and Bayesian objectives.
result VIB minimizes risks due to finite sampling of targets.
Proposes a multi-task learning model using variational information bottleneck.
problem Balancing performance and robustness across different tasks in multi-task learning.
method Variational Information Bottleneck (VIB) architecture for multi-task learning.
result The proposed model achieves competitive prediction accuracy under adversarial attacks.
Proposes a new method to selectively access privileged information in reinforcement learning.
problem Selective compression of privileged information in reinforcement learning.
method Formulates a variational bandwidth bottleneck to decide stochastically whether to access privileged information.
result Improves generalization and reduces access to costly information in reinforcement learning experiments.
Information bottleneck (IB) is a technique for extracting information in one random variable X that is relevant for predicting another random variable Y. IB works by encoding X in a compressed "bottleneck" random variable M from which Y can be accurately decoded. However, finding the optimal bottleneck variab…
New method improves deep neural network performance in regression tasks.
problem Improving generalization, robustness, and explainability of deep neural networks in regression.
method Developed a new Information Bottleneck approach using Cauchy-Schwarz divergence.
result Demonstrated superior performance on six real-world regression tasks.
Information bottleneck (IB) is a method for extracting information from one random variable X that is relevant for predicting another random variable Y. To do so, IB identifies an intermediate "bottleneck" variable T that has low mutual information I(X;T) and high mutual information I(Y;T). The "IB curve" cha…
Improves robustness of information bottleneck framework with sparsity-inducing prior.
problem Fixed-dimensional priors restrict flexibility and restrict robustness.
method Sparsity-inducing spike-slab categorical prior that learns dimension distribution per data point.
result Improves accuracy and robustness compared to traditional priors and other methods.
The IB theory explains how ML systems reduce data dimensions while preserving predictive power.
problem Understanding how machine learning systems reduce data dimensions effectively.
method Information-theoretic approach using mutual information.
result The best representation T maximally informs Y while minimizing mutual information with X. New framework predicts time series with missing values without imputation.
problem Predicting time series with missing values, especially when there's no ground truth for missing data.
method CRIB framework, combining attention mechanism and consistency regularization.
result CRIB framework predicts accurately even under high missing rates.
Graph neural networks struggle to propagate long-range information, causing over-squashing.
problem Graph neural networks struggle to propagate long-range information.
method Identified over-squashing as the bottleneck in GNNs, demonstrated on various models.
result Breaking the bottleneck improves GNNs' performance on long-range problems.
DAB enriches deep networks with uncertainty estimates using a codebook of training inputs.
problem Lack of uncertainty estimation in deep neural networks.
method Distance Aware Bottleneck (DAB) method that learns a codebook of training inputs.
result DAB achieves better OOD detection and misclassification prediction than prior methods.
CB-APM uses analyst consensus as a bottleneck to interpret stock returns.
problem Tackles the challenge of understanding and predicting stock returns using professional beliefs.
method Embeds analyst consensus as a structural bottleneck, treating it as a sufficient statistic for market information.
result CB-APM portfolios exhibit strong monotonic return gradients and robust across different economic conditions.
Paper revisits Deep Variational Information Bottleneck and proposes a new optimization approach.
problem Limitations of Deep Variational Information Bottleneck in optimizing mutual information.
method Proposes a new optimization approach by circumventing the limitation of requiring both Markov chains during optimisation.
result Shows how to optimise a lower bound for mutual information, circumventing the limitation of requiring both Markov chains.
HCBM improves deep learning explainability by non-linear concept aggregation.
problem Lack of explainable and accurate predictions in deep learning for high-stake decisions.
method Introduce Hoeffding Concept Bottleneck Models (HCBM) using Hoeffding functional decomposition of gradient-boosted trees for non-linear and sparse concept aggregation.
result HCBM outperforms standard linear CBM and is robust to interconcept leakage.
This paper introduces a new Lagrangian for the Information Bottleneck problem to simplify optimization.
problem Optimizing compressed representations for predicting Y while limiting information about X. method Introduces a general family of Lagrangians to explore the Information Bottleneck curve.
result Solves the original constrained optimization problem with a single optimization.
Meta learning with information theory and Gaussian processes.
problem Few-shot learning problems.
method Information bottleneck, mutual information, variational approximations, Gaussian processes.
result Competitive accuracy on few-shot classification problems.
New IBOs resolve contradictions in mutual information optimization.
problem Contradictions in mutual information optimization between IBP, PIBP, and recent results.
method Formulated and optimized variational bounds on new IBOs.
result New IBOs account for parameter selection in trained models.
The Information Plane theory predicts autoencoders do not compress input information.
problem Understanding the training dynamics of hidden layers in autoencoders.
method Derive a theoretical convergence for the Information Plane of autoencoders using a Gram-matrix based mutual information estimator.
result Ideal autoencoders with a large bottleneck layer size do not compress input information, while a small size causes compression only in the encoder layers.
Recent works investigated the generalization properties in deep neural networks (DNNs) by studying the Information Bottleneck in DNNs. However, the mea- surement of the mutual information (MI) is often inaccurate due to the density estimation. To address this issue, we propose to measure the dependency instead of MI be…
Tutorial on information bottleneck problems with connections to coding and learning.
problem Information bottleneck problems and their connections to coding and learning.
method Information theoretic perspective, practical methods, connections to various problems.
result Optimal trade-offs between relevance and complexity in discrete and vector Gaussian frameworks.
GWIB improves counterfactual regression by balancing latent distributions and reducing selection bias.
problem Selection bias between control and treatment groups negatively impacts counterfactual regression performance.
method GWIB uses Gromov-Wasserstein information bottleneck to maximize mutual information between covariates and outcomes while penalizing kernelized mutual information between latent representations and covariates.
result GWIB consistently outperforms state-of-the-art CFR methods in ITE estimation tasks.
The Information bottleneck method is an unsupervised non-parametric data organization technique. Given a joint distribution P(A,B), this method constructs a new variable T that extracts partitions, or clusters, over the values of A that are informative about B. The information bottleneck has already been applied to doc…
In this paper, we provide an information-theoretic interpretation of the Vector Quantized-Variational Autoencoder (VQ-VAE). We show that the loss function of the original VQ-VAE can be derived from the variational deterministic information bottleneck (VDIB) principle. On the other hand, the VQ-VAE trained by the Expect…
GeoIB uses information geometry to control compression in deep learning models.
problem The indirect and biased nature of traditional IB implementations in deep learning.
method GeoIB uses Fisher-Rao and Jacobian-Frobenius terms to control information compression directly.
result GeoIB achieves better trade-off between accuracy and compression than traditional IB methods.
SCBMs model causal effects using low-dimensional bottlenecks.
problem Causal effect estimation in high-dimensional systems.
method Structural causal models with low-dimensional summary statistics.
result SCBMs provide a flexible framework for task-specific dimension reduction.
New approach quantifies overfitting in high-dimensional regression.
problem Quantifying and avoiding overfitting in large neural networks.
method Information bottleneck theory to minimize residual information while maximizing relevant bits.
result Characterized the relative information efficiency of randomized regression compared to optimal algorithms.
Graph Information Bottleneck (GIB) optimizes graph representations for robustness against adversarial attacks.
problem Challenges in learning graph representations due to structure and feature information.
method GIB is an information-theoretic principle that balances expressiveness and robustness by maximizing mutual information between representation and target, while constraining mutual information with input data.
result GIB-based models are more robust to adversarial attacks, achieving up to 31% improvement.
A new method improves model generalization by recognizing representations.
problem Generalization issues in existing Information Bottlenecks.
method Recognizable Information Bottleneck (RIB) using f-CMI and density ratio matching.
result Improves model generalization through recognizability regularization.
Unified information-theoretic objectives for training deep neural networks.
problem Difficulty in computing information-theoretic quantities for large deep neural networks.
method Review and unify competing objectives, develop surrogate objectives.
result Surrogate objectives allow applying information bottleneck to modern neural network architectures.
Proposes a method to extract robust features that improve classifier robustness.
problem Improving classifier robustness to small perturbations in input space.
method Introduces an additional penalty term in the information bottleneck framework to minimize Fisher information, optimizing a variational bound using stochastic gradient descent.
result Optimally robust features are jointly Gaussian, and the method produces classifiers with increased robustness to perturbations.
New learning rules from information bottleneck improve deep learning without precise labels.
problem Training deep neural networks with backpropagation is biologically implausible.
method Kernelized information bottleneck principle with 3-factor Hebbian structure.
result The new learning rules perform nearly as well as backpropagation on image classification tasks.
Deep latent variable models are powerful tools for representation learning. In this paper, we adopt the deep information bottleneck model, identify its shortcomings and propose a model that circumvents them. To this end, we apply a copula transformation which, by restoring the invariance properties of the information b…
New method uses information bottleneck to quantify variable importance in neural networks.
problem Quantifying the importance of individual input variables in neural networks.
method Restricting the flow of information by adding noise to intermediate feature maps.
result Our method outperforms all baselines in five out of six settings.
New objective function improves model robustness.
problem Machine Learning robustness issues.
method Conditional Entropy Bottleneck (CEB) objective function.
result CEB models improve robust generalization.
A graph neural network detects beneficial feature interactions for recommender systems.
problem Feature interactions are crucial but not all are beneficial for recommendation accuracy.
method Graph neural network with L0 activation regularization for edge prediction.
result The model outperforms baselines and automatically identifies beneficial feature interactions.
Neural processes approximate Gaussian process inference, revealing three key costs.
problem Approximating Gaussian process inference with neural processes.
method Bounding KL divergence into three components: label contamination, information bottleneck, and amortization error.
result Characterization of three costs of amortizing Gaussian process inference with neural processes.
CBMs improve interpretability in RUL prediction for aircraft engines.
problem Lack of interpretability in deep learning models for asset prognostics.
method Concept Bottleneck Models (CBMs) for RUL prediction.
result CBMs achieve comparable or superior performance to black-box models while being more interpretable.