The paper interprets VQ-VAE loss as a form of information bottleneck.
problem Understanding the VQ-VAE loss function.
method Interpreted VQ-VAE loss as variational deterministic information bottleneck (VDIB) and variational information bottleneck (VIB).
result VQ-VAE loss can be derived from VDIB and approximated by VIB.
IB fails in deterministic scenarios, revealing three key issues.
problem Information bottleneck method's limitations in deterministic scenarios.
method Demonstrated three caveats and proposed a new functional.
result IB curve cannot be recovered by IB Lagrangian for deterministic Y. Derives a new variational approach to information bottleneck.
problem Information theoretic inference and predictive modeling.
method Variational lower bound of predictive information bottleneck.
result Generalizes modern inference procedures and suggests new ones.
New objective function improves model robustness.
problem Machine Learning robustness issues.
method Conditional Entropy Bottleneck (CEB) objective function.
result CEB models improve robust generalization.
Improves deep network generalization for image sequence reconstruction.
problem Improving generalization of deep networks for inverse image reconstruction.
method Proposes a network optimized by a variational approximation of the information bottleneck principle with stochastic latent space.
result Demonstrates improved generalization ability of inverse reconstruction networks through stochasticity and information bottleneck.
Develops a new unsupervised clustering method using Variational Information Bottleneck and Gaussian Mixture Model.
problem Unsupervised clustering of unlabeled data.
method Combines Variational Information Bottleneck and Gaussian Mixture Model in a deep neural network framework.
result Derives a new bound on the cost function and provides an algorithm for efficient computation.
Paper revisits Deep Variational Information Bottleneck and proposes a new optimization approach.
problem Limitations of Deep Variational Information Bottleneck in optimizing mutual information.
method Proposes a new optimization approach by circumventing the limitation of requiring both Markov chains during optimisation.
result Shows how to optimise a lower bound for mutual information, circumventing the limitation of requiring both Markov chains.
Lossy compression and clustering fundamentally involve a decision about what features are relevant and which are not. The information bottleneck method (IB) by Tishby, Pereira, and Bialek formalized this notion as an information-theoretic optimization problem and proposed an optimal tradeoff between throwing away as ma…
The Information bottleneck method is an unsupervised non-parametric data organization technique. Given a joint distribution P(A,B), this method constructs a new variable T that extracts partitions, or clusters, over the values of A that are informative about B. The information bottleneck has already been applied to doc…
Proposes a multi-task learning model using variational information bottleneck.
problem Balancing performance and robustness across different tasks in multi-task learning.
method Variational Information Bottleneck (VIB) architecture for multi-task learning.
result The proposed model achieves competitive prediction accuracy under adversarial attacks.
We present a simple case study, demonstrating that Variational Information Bottleneck (VIB) can improve a network's classification calibration as well as its ability to detect out-of-distribution data. Without explicitly being designed to do so, VIB gives two natural metrics for handling and quantifying uncertainty.
New Variational InfoMax objective improves neural network performance.
problem Optimizing neural networks using Bayesian Inference and Information Bottleneck.
method Derive Variational InfoMax (VIM) objective that maximizes InfoMax directly.
result VIM improves model performance in accuracy, robustness, and representation quality.
Proposes a new method to selectively access privileged information in reinforcement learning.
problem Selective compression of privileged information in reinforcement learning.
method Formulates a variational bandwidth bottleneck to decide stochastically whether to access privileged information.
result Improves generalization and reduces access to costly information in reinforcement learning experiments.
Draft proposes adapting neural networks to match naive Bayes classifiers.
problem Bridge between neural networks and naive Bayes classifiers.
method Class-conditional compression and disentanglement using variational bounds.
result Latent representations enable naive Bayes classifier performance.
Proposes a robust VIB approach using soft labels and mutual info estimation.
problem Improving robustness of VIB to adversarial perturbations.
method Refines categorical class information with soft labels from a reference network, relaxes Gaussian posterior assumption.
result Significantly outperforms benchmarked models on MNIST and CIFAR-10.
VIBI interprets black-box systems by selecting key features that are both brief and comprehensive.
problem Lack of concise and comprehensive explanations for black-box decision systems.
method VIBI uses the information bottleneck principle to select key features that are maximally compressed and informative.
result VIBI provides more concise and comprehensive explanations compared to existing methods.
New algorithm clusters sparse data effectively.
problem Challenges in clustering sparse data.
method Deterministic Information Bottleneck framework for joint feature weighting and clustering.
result Demonstrated effectiveness on real-world genomics data.
Tutorial on information bottleneck problems with connections to coding and learning.
problem Information bottleneck problems and their connections to coding and learning.
method Information theoretic perspective, practical methods, connections to various problems.
result Optimal trade-offs between relevance and complexity in discrete and vector Gaussian frameworks.
Improves unsupervised domain adaptation by enforcing feature extractor to focus on task-relevant information.
problem Leveraging label information from source domain for accurate target domain models without labels.
method Variational Information Bottleneck (VBDA) method that explicitly enforces feature extractor to ignore irrelevant task factors.
result Significantly outperforms state-of-the-art methods across three domain adaptation benchmark datasets.
In many applications, it is desirable to extract only the relevant aspects of data. A principled way to do this is the information bottleneck (IB) method, where one seeks a code that maximizes information about a 'relevance' variable, Y, while constraining the information encoded about the original data, X. Unfortunate…
VIB balances empirical and Bayesian approaches in predictive models.
problem Balancing empirical and Bayesian methods in predictive models.
method VIB as a compromise between empirical and Bayesian objectives.
result VIB minimizes risks due to finite sampling of targets.
CVIB uses information theory to learn counterfactuals from MNAR data without RCTs.
problem Debiasing learning from missing-not-at-random (MNAR) data in recommendation systems.
method CVIB, a variational information bottleneck, separates task-aware mutual information into factual and counterfactual parts.
result CVIB significantly enhances both shallow and deep models in recommendation systems.
Meta learning with information theory and Gaussian processes.
problem Few-shot learning problems.
method Information bottleneck, mutual information, variational approximations, Gaussian processes.
result Competitive accuracy on few-shot classification problems.
Extends machine learning model training to include validation data.
problem Machine learning validation data usage in training.
method Extends predictive information bottleneck to include validation data.
result Validation data improves model training efficiency.
New definition of disentanglement for non-independent factors of variation.
problem Current disentanglement definitions assume independent factors of variation, limiting their applicability.
method Definition based on information theory, related to Information Bottleneck Method, proposed measurement method.
result Proposed method correctly measures disentanglement with non-independent factors of variation.
Algorithm maps trade-off between clustering fidelity and representation size.
problem Optimizing trade-off between clustering fidelity and representation size.
method Introduces primal Deterministic Information Bottleneck (DIB) problem for discrete search spaces.
result Shows richer Pareto frontier over Lagrangian relaxation.
DAB enriches deep networks with uncertainty estimates using a codebook of training inputs.
problem Lack of uncertainty estimation in deep neural networks.
method Distance Aware Bottleneck (DAB) method that learns a codebook of training inputs.
result DAB achieves better OOD detection and misclassification prediction than prior methods.
Proposes a method to extract robust features that improve classifier robustness.
problem Improving classifier robustness to small perturbations in input space.
method Introduces an additional penalty term in the information bottleneck framework to minimize Fisher information, optimizing a variational bound using stochastic gradient descent.
result Optimally robust features are jointly Gaussian, and the method produces classifiers with increased robustness to perturbations.
Information bottleneck (IB) is a technique for extracting information in one random variable X that is relevant for predicting another random variable Y. IB works by encoding X in a compressed "bottleneck" random variable M from which Y can be accurately decoded. However, finding the optimal bottleneck variab…
VDB constrains discriminator to improve imitation learning and GANs.
problem Stability and performance issues in adversarial learning methods.
method Information bottleneck technique to modulate discriminator's accuracy.
result Significant improvements across imitation learning, inverse RL, and GANs.
New IBOs resolve contradictions in mutual information optimization.
problem Contradictions in mutual information optimization between IBP, PIBP, and recent results.
method Formulated and optimized variational bounds on new IBOs.
result New IBOs account for parameter selection in trained models.
This study analyzes VAEs using ID and II, revealing a transition in behaviour and distinct training phases.
problem Understanding the hidden representations and training phases of VAEs.
method Analysis using Intrinsic Dimension (ID) and Information Imbalance (II).
result VAEs exhibit a transition in behaviour and distinct training phases when the bottleneck size exceeds the Intrinsic Dimension of the data.
VUSFA improves transfer learning for target-driven navigation in AI2THOR.
problem Improving transfer reinforcement learning for complex visual navigation tasks.
method Introducing SFDP and Variational Information Bottlenecks to A3C agent.
result VUSFA achieves state-of-the-art performance and generalizability.
A new method clusters mixed-type data efficiently.
problem Clustering mixed-type data with continuous and categorical variables.
method Extends Information Bottleneck principle to heterogeneous data using generalised product kernels.
result DIBmix outperforms four established methods in various scenarios.
New method improves deep neural network performance in regression tasks.
problem Improving generalization, robustness, and explainability of deep neural networks in regression.
method Developed a new Information Bottleneck approach using Cauchy-Schwarz divergence.
result Demonstrated superior performance on six real-world regression tasks.
CB-APM uses analyst consensus as a bottleneck to interpret stock returns.
problem Tackles the challenge of understanding and predicting stock returns using professional beliefs.
method Embeds analyst consensus as a structural bottleneck, treating it as a sufficient statistic for market information.
result CB-APM portfolios exhibit strong monotonic return gradients and robust across different economic conditions.
FAVAE learns disentangled representations from sequential data.
problem Learning disentangled and interpretable representations from sequential data.
method FAVAE uses the information bottleneck principle without supervision.
result FAVAE can disentangle multiple dynamic factors.
New method approximates diffusion process posteriors using moment functions.
problem Approximating posteriors of stochastic differential equations.
method Constructs variational process as controlled prior, approximates posterior with moment functions, uses natural gradient descent.
result Richer variational approximations for state-dependent diffusion terms.
The information bottleneck (IB) approach to clustering takes a joint distribution P(X,Y) and maps the data X to cluster labels T which retain maximal information about Y (Tishby et al., 1999). This objective results in an algorithm that clusters data points based upon the similarity of their condit…
GIB improves neural network generalization by dynamically selecting task-relevant features across different sequential environments.
problem Poor generalization of deep neural networks to unseen environments.
method Proposes a gated information bottleneck (GIB) approach that dynamically drops spurious correlations and selects task-relevant features.
result GIB outperforms other IB approaches in adversarial robustness and OOD detection.
New method identifies shared topics in LLM inputs and outputs for better detection of hallucinations.
problem Detecting semantic drift in LLM responses from context.
method Transformed Deterministic Information Bottleneck (DIB) into UDIB for high-dimensional data.
result UDIB generates more informative topic representations for SDM, improving hallucination detection.
Survey on Information Bottleneck in Deep Learning.
problem Improving Deep Neural Networks (DNNs).
method Information Theory (IT) concepts applied to DNNs.
result IB is promising for analyzing and improving DNNs.
This paper introduces a new Lagrangian for the Information Bottleneck problem to simplify optimization.
problem Optimizing compressed representations for predicting Y while limiting information about X. method Introduces a general family of Lagrangians to explore the Information Bottleneck curve.
result Solves the original constrained optimization problem with a single optimization.
We study optimal solutions to an abstract optimization problem for measures, which is a generalization of classical variational problems in information theory and statistical physics. In the classical problems, information and relative entropy are defined using the Kullback-Leibler divergence, and for this reason optim…
BF-VAE estimates uncertainty from LF and HF QoI samples.
problem Balancing computational efficiency and numerical accuracy in uncertainty quantification.
method Bi-fidelity formulation of VAEs in latent space.
result BF-VAE improves accuracy with limited HF data.
This work clarifies the compression phenomenon in deep neural networks.
problem Understanding the compression aspect of information bottleneck theory in deep neural networks.
method Introduced an auxiliary noisy DNN framework to measure meaningful mutual information, developed an estimator for mutual information, and monitored clustering of hidden representations.
result Compression is driven by the progressive clustering of hidden representations of inputs from the same class.
The total variation distance is a core statistical distance between probability measures that satisfies the metric axioms, with value always falling in [0,1]. This distance plays a fundamental role in machine learning and signal processing: It is a member of the broader class of f-divergences, and it is related to …
New method quantifies redundant information using information bottleneck.
problem Quantifying redundant information among multiple sources.
method Formulated as an information bottleneck problem, termed redundancy bottleneck.
result Extracts information that best predicts the target without revealing source identity.