This paper explores collaborative information bottleneck for multi-source scenarios.
problem Fundamental limits of collaborative information extraction in multi-source scenarios.
method Two-way Collaborative Information Bottleneck (TW-CIB) and Collaborative Distributed Information Bottleneck (CDIB) problems.
result Derivation of inner and outer bounds to the complexity-relevance region for TW-CIB and CDIB.
VAEs improve collaborative filtering for implicit feedback.
problem Limited modeling capacity of linear factor models in collaborative filtering.
method Introduced a generative model with multinomial likelihood and used Bayesian inference for parameter estimation.
result Significantly outperforms state-of-the-art baselines on real-world datasets.
The paper interprets VQ-VAE loss as a form of information bottleneck.
problem Understanding the VQ-VAE loss function.
method Interpreted VQ-VAE loss as variational deterministic information bottleneck (VDIB) and variational information bottleneck (VIB).
result VQ-VAE loss can be derived from VDIB and approximated by VIB.
Derives a new variational approach to information bottleneck.
problem Information theoretic inference and predictive modeling.
method Variational lower bound of predictive information bottleneck.
result Generalizes modern inference procedures and suggests new ones.
A framework for collaborative learning reduces communication rounds.
problem Collaborative learning with distributed features and privacy concerns.
method Federated Stochastic Block Coordinate Descent (FedBCD) algorithm.
result The algorithm achieves O ( T ) O(\sqrt{T}) O ( T ) communication rounds and O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) accuracy. Paper revisits Deep Variational Information Bottleneck and proposes a new optimization approach.
problem Limitations of Deep Variational Information Bottleneck in optimizing mutual information.
method Proposes a new optimization approach by circumventing the limitation of requiring both Markov chains during optimisation.
result Shows how to optimise a lower bound for mutual information, circumventing the limitation of requiring both Markov chains.
New method quantifies redundant information using information bottleneck.
problem Quantifying redundant information among multiple sources.
method Formulated as an information bottleneck problem, termed redundancy bottleneck.
result Extracts information that best predicts the target without revealing source identity.
The paper identifies universal features for high-dimensional data inference.
problem Identifying universal low-dimensional features from high-dimensional data for inference tasks.
method Introduces natural notions of universality and shows a local equivalence among them, using information geometry.
result Reveals the complementary roles of various data analysis techniques.
Tutorial on information bottleneck problems with connections to coding and learning.
problem Information bottleneck problems and their connections to coding and learning.
method Information theoretic perspective, practical methods, connections to various problems.
result Optimal trade-offs between relevance and complexity in discrete and vector Gaussian frameworks.
The Information bottleneck method is an unsupervised non-parametric data organization technique. Given a joint distribution P(A,B), this method constructs a new variable T that extracts partitions, or clusters, over the values of A that are informative about B. The information bottleneck has already been applied to doc…
Proposes Decodable Information Bottleneck for optimal representation learning.
problem Finding optimal representations for supervised learning.
method Integrates information retention and compression with the desired predictive family.
result Optimal representations lead to better expected test performance and can be estimated with guarantees.
Develops a new unsupervised clustering method using Variational Information Bottleneck and Gaussian Mixture Model.
problem Unsupervised clustering of unlabeled data.
method Combines Variational Information Bottleneck and Gaussian Mixture Model in a deep neural network framework.
result Derives a new bound on the cost function and provides an algorithm for efficient computation.
VIB improves classification and uncertainty quantification.
problem Improving classification calibration and uncertainty quantification.
method Presented a simple case study of VIB.
result VIB improves classification calibration and uncertainty quantification naturally.
SCBMs model causal effects using low-dimensional bottlenecks.
problem Causal effect estimation in high-dimensional systems.
method Structural causal models with low-dimensional summary statistics.
result SCBMs provide a flexible framework for task-specific dimension reduction.
Paper proposes a new model for disentangled latent representations using copula transformations.
problem Disentanglement of latent features in deep latent variable models.
method Adopted deep information bottleneck model, applied copula transformation to restore invariance and sparsity.
result The new model achieves disentanglement and sparsity of latent features.
New method extracts relevant information from X X X for predicting Y Y Y .
problem Finding optimal bottleneck variable in IB optimization problem.
method Non-parametric upper bound for mutual information, neural network implementation.
result Method achieves better performance than variational IB on real-world datasets.
Novel algorithm reduces delays and communication in decentralized learning.
problem Decentralized learning with straggling nodes and high communication costs.
method QuanTimed-DSGD: deadline-imposed gradient computation and quantized model exchange.
result Converges to global optimal for convex functions, finds first-order stationary points for non-convex.
Proposes a new method to selectively access privileged information in reinforcement learning.
problem Selective compression of privileged information in reinforcement learning.
method Formulates a variational bandwidth bottleneck to decide stochastically whether to access privileged information.
result Improves generalization and reduces access to costly information in reinforcement learning experiments.
JPS improves joint policies for multi-agent collaboration in imperfect information games.
problem Learning good joint policies for multi-agent collaboration with imperfect information.
method Decomposes global changes to localized policy changes, iteratively improving joint policies without re-evaluating the entire game.
result JPS improves solutions provided by unilateral approaches and outperforms algorithms designed for collaborative policy learning.
SHIFT method optimally estimates heterogeneous discrete distributions with limited communication.
problem Collaborative learning of discrete distributions under heterogeneity and communication constraints.
method Two-stage method: First, users learn a central distribution; then, fine-tune this to estimate individual distributions.
result SHIFT is minimax optimal in the model of heterogeneity and under communication constraints.
Graph Information Bottleneck (GIB) optimizes graph representations for robustness against adversarial attacks.
problem Challenges in learning graph representations due to structure and feature information.
method GIB is an information-theoretic principle that balances expressiveness and robustness by maximizing mutual information between representation and target, while constraining mutual information with input data.
result GIB-based models are more robust to adversarial attacks, achieving up to 31% improvement.
A new method improves model generalization by recognizing representations.
problem Generalization issues in existing Information Bottlenecks.
method Recognizable Information Bottleneck (RIB) using f-CMI and density ratio matching.
result Improves model generalization through recognizability regularization.
Unified information-theoretic objectives for training deep neural networks.
problem Difficulty in computing information-theoretic quantities for large deep neural networks.
method Review and unify competing objectives, develop surrogate objectives.
result Surrogate objectives allow applying information bottleneck to modern neural network architectures.
CIB compresses variables causally, preserving key causal interactions.
problem Constructing causal variable abstractions in complex systems.
method Causal Information Bottleneck (CIB) method, extending IB to include causal structures.
result CIB produces causally interpretable abstractions that accurately capture causal relations.
Extends machine learning model training to include validation data.
problem Machine learning validation data usage in training.
method Extends predictive information bottleneck to include validation data.
result Validation data improves model training efficiency.
Proposes a method to extract robust features that improve classifier robustness.
problem Improving classifier robustness to small perturbations in input space.
method Introduces an additional penalty term in the information bottleneck framework to minimize Fisher information, optimizing a variational bound using stochastic gradient descent.
result Optimally robust features are jointly Gaussian, and the method produces classifiers with increased robustness to perturbations.
New method uses information bottleneck to estimate causal effects from missing data.
problem Estimating causal effects from high-dimensional observational data with missing covariates.
method Uses information bottleneck to compress covariates and transfer relevant information.
result Achieves state-of-the-art performance in causal inference benchmarks and real applications.
A novel federated learning framework resolves structural misalignment in model fusion.
problem Structural misalignment in model fusion due to chaotic information distribution.
method Feature-oriented regulation method ( Ψ Ψ Ψ -Net) to ensure feature information allocation and dedicated collaboration schemes. result Effective enhancement of federated learning applicability to heterogeneous settings with improved convergence speed, accuracy, and efficiency.
A variational method simplifies information bottleneck for sparse, relevant features.
problem Extracting relevant aspects of high-dimensional, non-gaussian data.
method Variational information bottleneck (IB) method with sparse coding.
result Recover features that are both relevant and sparse.
Graph DNA uses Bloom filters to efficiently encode deep graph neighborhoods for better collaborative filtering.
problem Collaborative filtering struggles with exploiting deeper graph neighborhoods due to high time and space complexity.
method Graph DNA employs Bloom filters to compute approximate deep neighborhood information in linear time, enabling efficient encoding and utilization in collaborative filtering.
result Graph DNA significantly improves collaborative filtering performance with minimal computational and memory overhead.
Extends information bottleneck to multi-view unsupervised learning.
problem Identifying superfluous information in unlabeled multi-view data.
method Multi-view information bottleneck model, leveraging data augmentation.
result State-of-the-art results on Sketchy and MIR-Flickr datasets.
Estimates latent dimensionality for prediction tasks using mutual information.
problem Estimating the latent dimensionality needed for accurate prediction.
method Formulates the problem as an Information Bottleneck question and uses neural mutual information estimators with a hybrid critic to preserve latent geometry.
result The hybrid critic method provides a more accurate estimation of task-relevant dimensionality.
VIBI interprets black-box systems by selecting key features that are both brief and comprehensive.
problem Lack of concise and comprehensive explanations for black-box decision systems.
method VIBI uses the information bottleneck principle to select key features that are maximally compressed and informative.
result VIBI provides more concise and comprehensive explanations compared to existing methods.
TGP enhances collaborative filtering with side information using Gaussian Processes.
problem Improving collaborative filtering with side information.
method Formulated a Tucker Gaussian Process (TGP) that incorporates low-rank matrix factorisation and side information.
result Enhanced predictive performance for collaborative filtering problems.
New method uses information bottleneck to quantify variable importance in neural networks.
problem Quantifying the importance of individual input variables in neural networks.
method Restricting the flow of information by adding noise to intermediate feature maps.
result Our method outperforms all baselines in five out of six settings.
New objective function improves model robustness.
problem Machine Learning robustness issues.
method Conditional Entropy Bottleneck (CEB) objective function.
result CEB models improve robust generalization.
Introduces deterministic information bottleneck (DIB) replacing mutual information with entropy.
problem Compression and feature selection in lossy compression and clustering.
method Formulates deterministic information bottleneck (DIB) using entropy instead of mutual information, resulting in deterministic encoder (hard clustering).
result DIB outperforms IB in terms of DIB cost function and offers computational efficiency gains.
Collaborative filtering (CF) is a successful approach commonly used by many recommender systems. Conventional CF-based methods use the ratings given to items by users as the sole source of information for learning to make recommendation. However, the ratings are often very sparse in many applications, causing CF-based …
Paper proposes a method to estimate individual treatment effects reliably from observational data.
problem Estimating individual treatment effects from observational data is challenging and important.
method The approach uses the Information Bottleneck principle to find more reliable representations for ITE estimation.
result The proposed model achieves state-of-the-art results and provides more reliable prediction performances with uncertainty information.
Proposes a multi-task learning model using variational information bottleneck.
problem Balancing performance and robustness across different tasks in multi-task learning.
method Variational Information Bottleneck (VIB) architecture for multi-task learning.
result The proposed model achieves competitive prediction accuracy under adversarial attacks.
Draft proposes adapting neural networks to match naive Bayes classifiers.
problem Bridge between neural networks and naive Bayes classifiers.
method Class-conditional compression and disentanglement using variational bounds.
result Latent representations enable naive Bayes classifier performance.
Advances in collaborative filtering and ranking methods.
problem Improving recommendation systems efficiency and accuracy.
method Graph information encoding, pairwise and listwise approaches, regularization techniques, personalization.
result New methods significantly improve recommendation system performance.
Proposes a robust VIB approach using soft labels and mutual info estimation.
problem Improving robustness of VIB to adversarial perturbations.
method Refines categorical class information with soft labels from a reference network, relaxes Gaussian posterior assumption.
result Significantly outperforms benchmarked models on MNIST and CIFAR-10.
Graph neural networks struggle to propagate long-range information, causing over-squashing.
problem Graph neural networks struggle to propagate long-range information.
method Identified over-squashing as the bottleneck in GNNs, demonstrated on various models.
result Breaking the bottleneck improves GNNs' performance on long-range problems.
Improves robustness of information bottleneck framework with sparsity-inducing prior.
problem Fixed-dimensional priors restrict flexibility and restrict robustness.
method Sparsity-inducing spike-slab categorical prior that learns dimension distribution per data point.
result Improves accuracy and robustness compared to traditional priors and other methods.
The paper examines FDL and SF algorithms using information bottleneck theory.
problem Understanding the behavior of FDL algorithms using information-theoretic terms.
method Formalization of FDL in information-theoretic terms, using information planes to analyze SF algorithm.
result Empirical support for the conjecture about the dynamics of FDL.
ResNet learns to compress information during training.
problem Applying information bottleneck principle to deep learning models.
method Trained PixelCNN++ models to measure mutual information between ResNet hidden layers and input data.
result Compression occurs in both classification and autoencoding training regimes.
A new model VCM improves collaborative filtering by synchronously linking two VAEs.
problem Cold start and data sparsity issues in CF-based recommender systems.
method Proposes a variational collaborative model (VCM) that synchronously links two VAEs.
result VCM outperforms state-of-the-art methods on real-life datasets.