Proposes a method to extract robust features that improve classifier robustness.
problem Improving classifier robustness to small perturbations in input space.
method Introduces an additional penalty term in the information bottleneck framework to minimize Fisher information, optimizing a variational bound using stochastic gradient descent.
result Optimally robust features are jointly Gaussian, and the method produces classifiers with increased robustness to perturbations.
Graph Information Bottleneck (GIB) optimizes graph representations for robustness against adversarial attacks.
problem Challenges in learning graph representations due to structure and feature information.
method GIB is an information-theoretic principle that balances expressiveness and robustness by maximizing mutual information between representation and target, while constraining mutual information with input data.
result GIB-based models are more robust to adversarial attacks, achieving up to 31% improvement.
Efficient DSP features reduce speech recognition memory usage.
problem Limited memory on DSPs for speech recognition.
method Developed efficient bottleneck features (BNFs) for DSPs.
result Reduced speech recognition memory usage by 10x with minimal accuracy loss.
Improves unsupervised domain adaptation by enforcing feature extractor to focus on task-relevant information.
problem Leveraging label information from source domain for accurate target domain models without labels.
method Variational Information Bottleneck (VBDA) method that explicitly enforces feature extractor to ignore irrelevant task factors.
result Significantly outperforms state-of-the-art methods across three domain adaptation benchmark datasets.
GIB improves neural network generalization by dynamically selecting task-relevant features across different sequential environments.
problem Poor generalization of deep neural networks to unseen environments.
method Proposes a gated information bottleneck (GIB) approach that dynamically drops spurious correlations and selects task-relevant features.
result GIB outperforms other IB approaches in adversarial robustness and OOD detection.
Proposes a multi-task learning model using variational information bottleneck.
problem Balancing performance and robustness across different tasks in multi-task learning.
method Variational Information Bottleneck (VIB) architecture for multi-task learning.
result The proposed model achieves competitive prediction accuracy under adversarial attacks.
MPNNs struggle with class-bottlenecks and heterophily, leading to performance limitations.
problem Performance limitations of MPNNs under heterophily and structural bottlenecks.
method A statistical framework decomposing model performance into SNR components and proving bounds on sensitivity.
result Optimal graph structures for maximizing higher-order homophily are disjoint unions of single-class and two-class-bipartite clusters.
Study on feature learning in Leaky ResNets, explaining bottleneck structure.
problem Understanding feature learning in deep neural networks.
method Lagrangian and Hamiltonian reformulation of representation geodesics.
result Emergence of a bottleneck structure in large effective depth networks.
VIBI interprets black-box systems by selecting key features that are both brief and comprehensive.
problem Lack of concise and comprehensive explanations for black-box decision systems.
method VIBI uses the information bottleneck principle to select key features that are maximally compressed and informative.
result VIBI provides more concise and comprehensive explanations compared to existing methods.
CNNs use a bottleneck structure to focus on a few frequencies, affecting function representation.
problem Understanding how CNNs focus on specific frequencies in their feature learning.
method Defined Convolution Bottleneck (CBN) structure, measured CBN rank, and analyzed parameter norms.
result Parameter norm scales with depth and CBN rank, and networks with optimal parameters exhibit this structure.
New method uses information bottleneck to quantify variable importance in neural networks.
problem Quantifying the importance of individual input variables in neural networks.
method Restricting the flow of information by adding noise to intermediate feature maps.
result Our method outperforms all baselines in five out of six settings.
Deep latent variable models are powerful tools for representation learning. In this paper, we adopt the deep information bottleneck model, identify its shortcomings and propose a model that circumvents them. To this end, we apply a copula transformation which, by restoring the invariance properties of the information b…
End-to-end neural network improves QbE-STD in multilingual speech search.
problem Query by example spoken term detection in zero-resource scenarios.
method Use of multilingual bottleneck features and CNN for pattern matching, integrated into a fully neural network framework.
result CNN-based matching outperforms DTW-based matching using bottleneck features.
Large neural networks learn low-dimensional representations that balance complexity and regularity.
problem Understanding the tradeoff between low-dimensional representations and complexity in deep neural networks.
method Computed finite depth corrections to reveal a measure of regularity that bounds the pseudo-determinant of the Jacobian.
result Proved the conjectured bottleneck structure in learned features as network depth increases, showing almost all hidden representations are approximately low-dimensional and weight matrices have singular values close to 1.
A graph neural network detects beneficial feature interactions for recommender systems.
problem Feature interactions are crucial but not all are beneficial for recommendation accuracy.
method Graph neural network with L0 activation regularization for edge prediction.
result The model outperforms baselines and automatically identifies beneficial feature interactions.
Improved ASR-free keyword spotting in under-resourced languages.
problem Dynamic time warping for keyword spotting in languages with limited resources.
method Multilingual bottleneck extractor and correspondence autoencoder integration.
result More than 11% absolute improvement in ROC AUC over MFCCs.
VUSFA improves transfer learning for target-driven navigation in AI2THOR.
problem Improving transfer reinforcement learning for complex visual navigation tasks.
method Introducing SFDP and Variational Information Bottlenecks to A3C agent.
result VUSFA achieves state-of-the-art performance and generalizability.
Tutorial on information bottleneck problems with connections to coding and learning.
problem Information bottleneck problems and their connections to coding and learning.
method Information theoretic perspective, practical methods, connections to various problems.
result Optimal trade-offs between relevance and complexity in discrete and vector Gaussian frameworks.
New algorithm clusters sparse data effectively.
problem Challenges in clustering sparse data.
method Deterministic Information Bottleneck framework for joint feature weighting and clustering.
result Demonstrated effectiveness on real-world genomics data.
CIB compresses variables causally, preserving key causal interactions.
problem Constructing causal variable abstractions in complex systems.
method Causal Information Bottleneck (CIB) method, extending IB to include causal structures.
result CIB produces causally interpretable abstractions that accurately capture causal relations.
Graph rewiring method alleviates over-squashing in GNNs.
problem Over-squashing and bottlenecks in graph neural networks.
method Curvature-based graph rewiring method.
result Negatively curved edges cause over-squashing in GNNs.
New approach combines invariance and information bottleneck for OOD generalization.
problem OOD generalization failures in classification tasks.
method Revisit linear regression tasks, prove information bottleneck constraint necessary, propose combined approach.
result Combined invariance and information bottleneck approach improves OOD generalization.
The paper examines FDL and SF algorithms using information bottleneck theory.
problem Understanding the behavior of FDL algorithms using information-theoretic terms.
method Formalization of FDL in information-theoretic terms, using information planes to analyze SF algorithm.
result Empirical support for the conjecture about the dynamics of FDL.
Proposes a VAE with a discrete bottleneck for better text generation.
problem VAEs struggle with latent variable auto-regressive decoding in text generation.
method Introduces a discretized bottleneck to enforce latent feature matching in a compact space.
result Demonstrates improved text generation capabilities across various tasks.
The paper applies Information Bottleneck theory to CNNs and finds compression phase not always present.
problem Understanding the behavior of convolutional neural networks.
method Employed Information Bottleneck theory to analyze CNNs.
result Compression phase not observed in all CNN cases.
Paper introduces template functions for featurizing persistence diagrams.
problem Featurizing persistence diagrams for machine learning.
method Characterizes compactness, constructs dense subsets of continuous functions.
result Template functions enable supervised learning with persistence diagrams.
Proposes a new method to selectively access privileged information in reinforcement learning.
problem Selective compression of privileged information in reinforcement learning.
method Formulates a variational bandwidth bottleneck to decide stochastically whether to access privileged information.
result Improves generalization and reduces access to costly information in reinforcement learning experiments.
Novel framework improves graph learning for out-of-distribution generalization.
problem Graph out-of-distribution generalization challenges in neural networks.
method Invariant Graph Learning based on Information bottleneck theory (InfoIGL).
result Achieves state-of-the-art performance in graph classification tasks under OOD generalization.
LFD method improves text classification by making features clearer and less label-leaking.
problem Creating interpretable text representations that are both predictive and understandable.
method LFD method: proposes lexical and semantic features from contrastive text pairs, screens candidates using κ, and selects features by residual gain. result LFD features achieve higher human-human and human-LLM agreement than baseline concepts and are less label-leaking.
Proposes a new method to improve selective inference for Lasso models.
problem Over-conditioning due to conditioning on feature signs in selective inference for Lasso.
method Parametric programming approach to avoid conditioning on signs and identify feature selection events.
result Improves power and practicality of selective inference for Lasso models.
Self-supervised learning helps train deep features without needing lots of labeled data.
problem Annotation bottleneck in deep learning.
method Four main families of self-supervised approaches applied to various data modalities.
result Self-supervised methods can now rival fully supervised pre-training across multiple data types.
Paper introduces a technique to simplify RNN policies for better understanding and analysis.
problem Difficulty in explaining and analyzing RNN policies due to continuous-valued memory vectors and observation features.
method Quantized Bottleneck Insertion technique to learn finite representations of RNN vectors and features.
result Finite representations of RNN policies can be as small as 3 discrete memory states and 10 observations, improving interpretability.
We consider multilingual bottleneck features (BNFs) for nearly zero-resource keyword spotting. This forms part of a United Nations effort using keyword spotting to support humanitarian relief programmes in parts of Africa where languages are severely under-resourced. We use 1920 isolated keywords (40 types, 34 minutes)…
New research shows non-bottlenecked autoencoders can outperform bottlenecked ones for anomaly detection.
problem The necessity of a bottleneck in autoencoders for anomaly detection.
method Investigated two ways to remove bottlenecks: overparameterising the latent layer and introducing skip connections. Carried out extensive experiments on various AE types and datasets.
result Non-bottlenecked autoencoders can outperform bottlenecked ones, improving anomaly detection performance.
Proposes IIB for domain generalization, overcoming failure modes of IRM.
problem Domain generalization with nonlinear classifiers and pseudo-invariant features.
method Invariant Information Bottleneck (IIB) using mutual information and variational formulation.
result Significantly outperforms IRM on synthetic datasets and real-world benchmarks.
We simplify information measure computation using learned features.
problem Computing information measures from raw data is computationally expensive.
method Developed a separable design for computing information measures from learned feature representations.
result A variety of information measures can be computed efficiently through learned feature representations.
Enhanced financial forecasting using supervised autoencoders with noise augmentation and triple labeling.
problem Improving investment strategy performance on noisy financial data.
method Supervised autoencoders with noise augmentation and triple barrier labeling.
result Supervised autoencoders with balanced noise augmentation and bottleneck size significantly boost strategy effectiveness.
MLPerf benchmark suite evaluates diverse ML applications, highlighting system bottlenecks.
problem Understanding and optimizing ML applications across different models and systems.
method Analysis of MLPerf benchmark suite characteristics and comparison with previous benchmarks.
result Optimal distributed deep learning training requires dedicated low latency interconnects.
SCBMs model causal effects using low-dimensional bottlenecks.
problem Causal effect estimation in high-dimensional systems.
method Structural causal models with low-dimensional summary statistics.
result SCBMs provide a flexible framework for task-specific dimension reduction.
Wide neural networks with narrow bottlenecks behave like deep Gaussian processes.
problem Understanding the behavior of neural networks with narrow layers in the wide limit.
method Analyzing the wide limit of BNNs with narrow bottlenecks, showing they behave like a composition of GPs.
result Wide neural networks with narrow bottlenecks form a composition of GPs, termed a bottleneck NNGP.
In many applications, it is desirable to extract only the relevant aspects of data. A principled way to do this is the information bottleneck (IB) method, where one seeks a code that maximizes information about a 'relevance' variable, Y, while constraining the information encoded about the original data, X. Unfortunate…
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an…
Model agnostic feature importance for DNNs in NLP.
problem Characterizing DNNs as black boxes and explaining their decision-making process.
method Phrase-wise feature importance calculation for model agnostic DNNs.
result Robust and generalizable approach to feature importance calculation.
Theory explains deep nonlinear networks' plateaus and transitions.
problem Understanding long plateaus and feature acquisition transitions in deep nonlinear networks.
method Derived an exact identity for Frobenius norms, classified activation functions, and reduced matrix flow to a scalar ODE.
result Escape time law τ⋆=Θ(ε−(r−2)) for deep nonlinear networks, where r is the number of bottleneck layers. New learning rules from information bottleneck improve deep learning without precise labels.
problem Training deep neural networks with backpropagation is biologically implausible.
method Kernelized information bottleneck principle with 3-factor Hebbian structure.
result The new learning rules perform nearly as well as backpropagation on image classification tasks.
Sparse neural networks visualize paired transcriptomic and electrophysiological data.
problem Efficiently analyzing and visualizing paired multivariate neuroscientific data.
method Sparse deep neural networks with a two-dimensional bottleneck and group lasso penalty.
result Biologically interpretable two-dimensional visualizations of paired data.
Study shows bottlenecks improve image segmentation quality.
problem Robust object discovery in real-world images remains challenging.
method Empirical investigation of reconstruction bottlenecks in GENESIS model.
result Reconstruction bottlenecks determine reconstruction and segmentation quality.
Deep ResNets favor low bottleneck rank with proper hyperparameters.
problem Understanding the inductive bias of deep neural networks.
method Computed minimum-norm weights of a deep linear ResNet.
result Deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.