IB fails in deterministic scenarios, revealing three key issues.
problem Information bottleneck method's limitations in deterministic scenarios.
method Demonstrated three caveats and proposed a new functional.
result IB curve cannot be recovered by IB Lagrangian for deterministic Y. This paper introduces a new Lagrangian for the Information Bottleneck problem to simplify optimization.
problem Optimizing compressed representations for predicting Y while limiting information about X. method Introduces a general family of Lagrangians to explore the Information Bottleneck curve.
result Solves the original constrained optimization problem with a single optimization.
Reduces IB problem to a simpler, lower-dimensional problem.
problem Information bottleneck problem in high-dimensional spaces.
method Identifies sufficient statistic that factors conditional distribution, reducing IB to a lower-dimensional problem.
result Preserves full IB curve and optimal representations, making IB tractable.
AIB method improves robustness against adversarial perturbations.
problem Optimizing the IB principle for better robustness and understanding compression effects.
method Proposes adversarial information bottleneck (AIB) method to optimize IB principle without explicit distribution assumptions.
result Demonstrates effectiveness in learning more invariant representations and mitigating adversarial perturbations.
The Information Bottleneck (IB) is a conceptual method for extracting the most compact, yet informative, representation of a set of variables, with respect to the target. It generalizes the notion of minimal sufficient statistics from classical parametric statistics to a broader information-theoretic sense. The IB curv…
Lossy compression and clustering fundamentally involve a decision about what features are relevant and which are not. The information bottleneck method (IB) by Tishby, Pereira, and Bialek formalized this notion as an information-theoretic optimization problem and proposed an optimal tradeoff between throwing away as ma…
AgrLearn improves deep learning by optimizing multi-sample quantization.
problem Improving deep learning performance with fewer training samples.
method Develops AgrLearn, a vector IB quantization-based framework.
result Significant improvements in image recognition and text classification.
Study finds IBS useful for predicting ETF price movements.
problem Predicting short-term price movements in country ETFs.
method Quantitative analysis of historical price data using Mean Reversion.
result IBS can be a useful technical indicator for ETFs.
New method uses IB to train normalizing flows for better generative classification.
problem Training generative models with information theory for improved classification.
method Developed IB-INNs, a class of conditional normalizing flows trained with IB objective.
result IB-INNs offer improved uncertainty quantification and out-of-distribution detection compared to traditional generative classifiers.
The paper shows how to avoid the trivial representation in IB by choosing the right β.
problem Lack of theoretical guidance for choosing β in the IB method.
method Identifying a phase transition between learnable and unlearnable representations based on β.
result IB-Learnability is defined by the largest confident, typical, and imbalanced subset of examples.
A new framework for neural network classification using vector quantization.
problem Learning a neural network classifier under the IB principle.
method Aggregated Learning framework, combining vector quantization and variational techniques.
result The effectiveness of Aggregated Learning verified through experiments.
New method improves deep neural network performance in regression tasks.
problem Improving generalization, robustness, and explainability of deep neural networks in regression.
method Developed a new Information Bottleneck approach using Cauchy-Schwarz divergence.
result Demonstrated superior performance on six real-world regression tasks.
The paper studies phase transitions in Information Bottleneck for representation learning.
problem Understanding the behavior of compression and prediction terms in IB objective.
method Studied phase transitions in IB objective using second-order calculus of variations and Fisher information matrix.
result IB phase transitions correspond to learning new classes and are related to maximum correlation between input and target orthogonal to the learned representation.
In many applications, it is desirable to extract only the relevant aspects of data. A principled way to do this is the information bottleneck (IB) method, where one seeks a code that maximizes information about a 'relevance' variable, Y, while constraining the information encoded about the original data, X. Unfortunate…
IB-GAN improves multivariate time series classification under imbalance.
problem Class imbalance in multivariate time series classification.
method Unified approach combining data augmentation and classification via GANs.
result Significant performance gains for under-observed classes.
Information bottleneck (IB) is a technique for extracting information in one random variable X that is relevant for predicting another random variable Y. IB works by encoding X in a compressed "bottleneck" random variable M from which Y can be accurately decoded. However, finding the optimal bottleneck variab…
The Immersed Boundary (IB) method is a widely-used numerical methodology for the simulation of fluid-structure interaction problems. The IB method utilizes an Eulerian discretization for the fluid equations of motion while maintaining a Lagrangian representation of structural objects. Operators are defined for transmit…
Survey on Information Bottleneck in Deep Learning.
problem Improving Deep Neural Networks (DNNs).
method Information Theory (IT) concepts applied to DNNs.
result IB is promising for analyzing and improving DNNs.
IBS efficiently estimates log-likelihood without bias.
problem Estimating log-likelihood for complex models.
method Inverse Binomial Sampling (IBS) for efficient, unbiased estimation.
result IBS produces lower error in estimated parameters and maximum log-likelihood values.
GIB improves neural network generalization by dynamically selecting task-relevant features across different sequential environments.
problem Poor generalization of deep neural networks to unseen environments.
method Proposes a gated information bottleneck (GIB) approach that dynamically drops spurious correlations and selects task-relevant features.
result GIB outperforms other IB approaches in adversarial robustness and OOD detection.
GeoIB uses information geometry to control compression in deep learning models.
problem The indirect and biased nature of traditional IB implementations in deep learning.
method GeoIB uses Fisher-Rao and Jacobian-Frobenius terms to control information compression directly.
result GeoIB achieves better trade-off between accuracy and compression than traditional IB methods.
New method quantifies redundant information using information bottleneck.
problem Quantifying redundant information among multiple sources.
method Formulated as an information bottleneck problem, termed redundancy bottleneck.
result Extracts information that best predicts the target without revealing source identity.
A new method improves model generalization by recognizing representations.
problem Generalization issues in existing Information Bottlenecks.
method Recognizable Information Bottleneck (RIB) using f-CMI and density ratio matching.
result Improves model generalization through recognizability regularization.
In this paper, we propose a semi-supervised clustering method, CEC-IB, that models data with a set of Gaussian distributions and that retrieves clusters based on a partial labeling provided by the user (partition-level side information). By combining the ideas from cross-entropy clustering (CEC) with those from the inf…
Proposes GIB for recognizing informative subgraphs in graphs.
problem Recognizing a subgraph that is maximally informative yet compressive.
method Graph Information Bottleneck (GIB) framework, mutual information estimator, bi-level optimization, connectivity loss.
result IB-subgraph improves graph classification, interpretation, and denoising.
The information bottleneck (IB) approach to clustering takes a joint distribution P(X,Y) and maps the data X to cluster labels T which retain maximal information about Y (Tishby et al., 1999). This objective results in an algorithm that clusters data points based upon the similarity of their condit…
CIB compresses variables causally, preserving key causal interactions.
problem Constructing causal variable abstractions in complex systems.
method Causal Information Bottleneck (CIB) method, extending IB to include causal structures.
result CIB produces causally interpretable abstractions that accurately capture causal relations.
The theoretical basis for a candidate variational principle for the information bottleneck (IB) method is formulated within the ambit of the generalized nonadditive statistics of Tsallis. Given a nonadditivity parameter q, the role of the \textit{additive duality} of nonadditive statistics (q∗=2−q) in relating…
The IB theory explains how ML systems reduce data dimensions while preserving predictive power.
problem Understanding how machine learning systems reduce data dimensions effectively.
method Information-theoretic approach using mutual information.
result The best representation T maximally informs Y while minimizing mutual information with X. We model the behavior of three agent classes acting dynamically in a limit order book of a financial asset. Namely, we consider market makers (MM), high-frequency trading (HFT) firms, and institutional brokers (IB). Given a prior dynamic of the order book, similar to the one considered in the Queue-Reactive models [14,…
The paper applies Information Bottleneck theory to CNNs and finds compression phase not always present.
problem Understanding the behavior of convolutional neural networks.
method Employed Information Bottleneck theory to analyze CNNs.
result Compression phase not observed in all CNN cases.
SurvSurf predicts first hitting times for intermittent events without monotonic violations.
problem Predicting first hitting times for intermittent events with monotonicity guarantees.
method Partially monotonic neural network for sequential events, incorporating unobserved events.
result SurvSurf outperforms existing models in MSE and IBS metrics.
Extends likelihood ratio exponential families to analyze various optimization methods.
problem Analyzing optimization methods like rate-distortion and information bottleneck.
method Linking geometric mixture paths to exponential families and using hypothesis testing.
result Provides a common mathematical framework for understanding these methods.
Adapts RL regularization techniques to prevent overfitting.
problem Preventing reinforcement learning agents from overfitting to limited training environments.
method Selective Noise Injection (SNI) and Information Bottleneck (IB) techniques.
result Significantly improved generalization performance on Coinrun benchmark.
Graph Information Bottleneck (GIB) optimizes graph representations for robustness against adversarial attacks.
problem Challenges in learning graph representations due to structure and feature information.
method GIB is an information-theoretic principle that balances expressiveness and robustness by maximizing mutual information between representation and target, while constraining mutual information with input data.
result GIB-based models are more robust to adversarial attacks, achieving up to 31% improvement.
The version of Marsden-Ratiu reduction theorem for Nambu-Poisson manifolds by a regular distribution has been studied by Ibaˊn~ez et al. In this paper we show that the reduction is always ensured unless the distribution is zero. Next we extend the more general Falceto-Zambon Poisson reduct…
The paper examines FDL and SF algorithms using information bottleneck theory.
problem Understanding the behavior of FDL algorithms using information-theoretic terms.
method Formalization of FDL in information-theoretic terms, using information planes to analyze SF algorithm.
result Empirical support for the conjecture about the dynamics of FDL.
This work proves that large models can be compressed significantly without losing performance.
problem Achieving comparable performance with smaller models and less data.
method Developed a universal compression theory for neural networks and datasets.
result Proved that a generic permutation-invariant function can be compressed into a function of polylogarithmic size with vanishing error.
A method to learn distributed representations from multiple sources of information.
problem Learning representations from multiple, separate sources of information.
method Generalization of Tishby's IB method to the distributed setting, using variational bounds and iterative algorithms.
result Explicit characterizations of optimal tradeoffs between complexity and relevance for discrete and Gaussian models.
New findings show compressed representations are exponentially easier to learn.
problem Understanding the generalization power of neural networks.
method Studied the entropy of input variables as a simplicity assumption and proved a bound on sample complexity.
result Compressed representations are exponentially easier to learn, providing new insights into neural network generalization.
Paper establishes generalization bounds for representation learning using Minimum Description Length.
problem Designing efficient statistical supervised learning algorithms that generalize well to unseen data.
method Developed a compressibility framework using Minimum Description Length (MDL) to derive upper bounds on generalization error.
result Established the first theoretical generalization bounds for Information Bottleneck type encoders and representation learning.
The muti-layer information bottleneck (IB) problem, where information is propagated (or successively refined) from layer to layer, is considered. Based on information forwarded by the preceding layer, each stage of the network is required to preserve a certain level of relevance with regards to a specific hidden variab…
Learning with hidden variables is a central challenge in probabilistic graphical models that has important implications for many real-life problems. The classical approach is using the Expectation Maximization (EM) algorithm. This algorithm, however, can get trapped in local maxima. In this paper we explore a new appro…
New variational approach for privacy and fairness in data representations.
problem Learning private and fair representations while preserving relevant information.
method Variational formulation of privacy and fairness optimization problems using Lagrangians.
result Control over the trade-off between utility and privacy/fairness through a Lagrange multiplier parameter.
Paper proposes a method to estimate individual treatment effects reliably from observational data.
problem Estimating individual treatment effects from observational data is challenging and important.
method The approach uses the Information Bottleneck principle to find more reliable representations for ITE estimation.
result The proposed model achieves state-of-the-art results and provides more reliable prediction performances with uncertainty information.
New objective function improves model robustness.
problem Machine Learning robustness issues.
method Conditional Entropy Bottleneck (CEB) objective function.
result CEB models improve robust generalization.
Factor analysis is a statistical technique employed to evaluate how observed variables correlate through common factors and unique variables. While it is often used to analyze price movement in the unstable stock market, it does not always yield easily interpretable results. In this study, we develop improved factor mo…
Ribbon tangles are proper embeddings of tori and cylinders in the 4-ball~B4, "bounding" 3-manifolds with only ribbon disks as singularities. We construct an Alexander invariant A of ribbon tangles equipped with a representation of the fundamental group of their exterior in a free abelian group G. Th…