MSD removes dequantization bottleneck in LLM inference by approximating high-precision activations.
problem Dequantization bottleneck in LLM inference on modern AI accelerators.
method MSD decomposes high-precision activations into multiple low-precision components for direct multiplication with quantized weights.
result MSD avoids INT8-to-BF16 weight conversion, reducing dequantization cycles and HBM traffic.
Media is generally stored digitally and is therefore discrete. Many successful deep distribution models in deep learning learn a density, i.e., the distribution of a continuous random variable. Naïve optimization on discrete data leads to arbitrarily high likelihoods, and instead, it has become standard practice to add…
This work explores the relation between trainability and dequantization in variational QML models.
problem Understanding the interplay between trainability and dequantization in variational QML models.
method Provide precise definitions of trainability and dequantization, study their relation, and introduce recipes for building PQC-based QML models.
result Identify conditions under which trainability and non-dequantization are not mutually exclusive.
Method estimates densities on manifolds using dequantization.
problem Estimating densities on non-Euclidean manifolds.
method Inspired by dequantization, coordinate transformation, and normalizing flows.
result Successfully models densities on spheres, tori, and orthogonal groups.
Flow models have recently made great progress at modeling ordinal discrete data such as images and audio. Due to the continuous nature of flow models, dequantization is typically applied when using them for such discrete data, resulting in lower bound estimates of the likelihood. In this paper, we introduce subset flow…
Flow-based generative models are powerful exact likelihood models with efficient sampling and inference. Despite their computational efficiency, flow-based models generally have much worse density modeling performance compared to state-of-the-art autoregressive models. In this paper, we investigate and improve upon thr…
Quantum circuits reveal pathways to dequantization in machine learning models.
problem Navigating the complex landscape of quantum machine learning models and algorithms.
method Introducing a framework connecting quantum circuit structure to function representability.
result Fundamental properties of quantum circuits determine classical simulability of models.
In this work, we propose a deep neural network architecture motivated by primal-dual splitting methods from convex optimization. We show theoretically that there exists a close relation between the derived architecture and residual networks, and further investigate this connection in numerical experiments. Moreover, we…
New research shows non-bottlenecked autoencoders can outperform bottlenecked ones for anomaly detection.
problem The necessity of a bottleneck in autoencoders for anomaly detection.
method Investigated two ways to remove bottlenecks: overparameterising the latent layer and introducing skip connections. Carried out extensive experiments on various AE types and datasets.
result Non-bottlenecked autoencoders can outperform bottlenecked ones, improving anomaly detection performance.
SCBMs model causal effects using low-dimensional bottlenecks.
problem Causal effect estimation in high-dimensional systems.
method Structural causal models with low-dimensional summary statistics.
result SCBMs provide a flexible framework for task-specific dimension reduction.
We consider the reconstruction problem in compressed sensing in which the observations are recorded in a finite number of bits. They may thus contain quantization errors (from being rounded to the nearest representable value) and saturation errors (from being outside the range of representable values). Our formulation …
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an…
There has recently been much work on the "wide limit" of neural networks, where Bayesian neural networks (BNNs) are shown to converge to a Gaussian process (GP) as all hidden layers are sent to infinite width. However, these results do not apply to architectures that require one or more of the hidden layers to remain n…
Study shows bottlenecks improve image segmentation quality.
problem Robust object discovery in real-world images remains challenging.
method Empirical investigation of reconstruction bottlenecks in GENESIS model.
result Reconstruction bottlenecks determine reconstruction and segmentation quality.
Deep ResNets favor low bottleneck rank with proper hyperparameters.
problem Understanding the inductive bias of deep neural networks.
method Computed minimum-norm weights of a deep linear ResNet.
result Deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.
CB-APM uses analyst consensus as a bottleneck to interpret stock returns.
problem Tackles the challenge of understanding and predicting stock returns using professional beliefs.
method Embeds analyst consensus as a structural bottleneck, treating it as a sufficient statistic for market information.
result CB-APM portfolios exhibit strong monotonic return gradients and robust across different economic conditions.
Wide neural networks become linear, but adding bottlenecks makes them bilinear or multilinear.
problem Understanding the transition of neural networks from linearity to higher-order functions.
method Analyzing the behavior of randomly initialized wide neural networks with and without bottleneck layers.
result Bottleneck layers transform the network's function from linear to bilinear or multilinear.
New method quantifies redundant information using information bottleneck.
problem Quantifying redundant information among multiple sources.
method Formulated as an information bottleneck problem, termed redundancy bottleneck.
result Extracts information that best predicts the target without revealing source identity.
Paper introduces new methods for modeling categorical data.
problem Training generative models on categorical data like text and segmentation.
method Argmax Flows and Multinomial Diffusion models.
result Models outperform existing methods in log-likelihood.
Paper proposes Sp-GD for sparse max-affine regression with theoretical guarantees.
problem Sparse max-affine regression model selection and estimation.
method Sparse Gradient Descent (Sp-GD) initialization using sparse PCA and covering search.
result Sp-GD provides ε-accurate estimates with optimal number of observations.
The Mahler volume of a centrally symmetric convex body K is defined as M(K)= (Vol K)(Vol K^dual). Mahler conjectured that this volume is minimized when K is a cube. We introduce the bottleneck conjecture, which stipulates that a certain convex body K^diamond subset K X K^dual has least volume when K is an ellipsoid. If…
The Information bottleneck method is an unsupervised non-parametric data organization technique. Given a joint distribution P(A,B), this method constructs a new variable T that extracts partitions, or clusters, over the values of A that are informative about B. The information bottleneck has already been applied to doc…
MO2 learns useful behaviours from past experience for new tasks.
problem Discovering useful behaviours from past experience and transferring them to new tasks.
method Model-Based Offline Options (MO2) framework supporting sample-efficient bottleneck option discovery over continuous state-action spaces.
result MO2 outperforms recent option learning methods on complex long-horizon continuous control tasks.
Graph neural networks struggle to propagate long-range information, causing over-squashing.
problem Graph neural networks struggle to propagate long-range information.
method Identified over-squashing as the bottleneck in GNNs, demonstrated on various models.
result Breaking the bottleneck improves GNNs' performance on long-range problems.
In this paper, we provide an information-theoretic interpretation of the Vector Quantized-Variational Autoencoder (VQ-VAE). We show that the loss function of the original VQ-VAE can be derived from the variational deterministic information bottleneck (VDIB) principle. On the other hand, the VQ-VAE trained by the Expect…
MPNNs struggle with class-bottlenecks and heterophily, leading to performance limitations.
problem Performance limitations of MPNNs under heterophily and structural bottlenecks.
method A statistical framework decomposing model performance into SNR components and proving bounds on sensitivity.
result Optimal graph structures for maximizing higher-order homophily are disjoint unions of single-class and two-class-bipartite clusters.
CEB enhances model resilience through simple entropy bottleneck.
problem Improving model robustness against adversarial attacks.
method Conditional Entropy Bottleneck (CEB) combined with data augmentation.
result CEB significantly boosts adversarial robustness on various benchmarks.
Deep reinforcement learning has recently shown many impressive successes. However, one major obstacle towards applying such methods to real-world problems is their lack of data-efficiency. To this end, we propose the Bottleneck Simulator: a model-based reinforcement learning method which combines a learned, factorized …
Combining the Information Bottleneck model with deep learning by replacing mutual information terms with deep neural nets has proved successful in areas ranging from generative modelling to interpreting deep neural networks. In this paper, we revisit the Deep Variational Information Bottleneck and the assumptions neede…
Proposes Decodable Information Bottleneck for optimal representation learning.
problem Finding optimal representations for supervised learning.
method Integrates information retention and compression with the desired predictive family.
result Optimal representations lead to better expected test performance and can be estimated with guarantees.
We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks. The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages. It mitigates exploding and vanishing gradients, resulting in the ability…
Novel flows generate molecules without post-processing.
problem Generating new molecules efficiently and without post-processing issues.
method Continuous normalizing E(3)-equivariant flows based on node ODEs coupled as a graph PDE.
result Generated samples achieve state-of-the-art performance on QM9 and ZINC250K benchmarks.
PCBMs turn any neural network into interpretable models without dense annotations.
problem Restrictive nature of CBMs and lack of dense concept annotations in training data.
method Introduce PCBMs that can turn any neural network into interpretable models without dense annotations.
result PCBMs can turn any neural network into interpretable models without dense annotations, improving interpretability and performance.
The paper identifies network bottlenecks using minimax paths in stochastic networks.
problem Identifying bottlenecks in networks with stochastic weights.
method Modeling as combinatorial semi-bandit problem, applying combinatorial Thompson Sampling, and approximating the original objective due to computational intractability.
result Established an upper bound on Bayesian regret and evaluated Thompson Sampling performance on real-world networks.
CIB compresses variables causally, preserving key causal interactions.
problem Constructing causal variable abstractions in complex systems.
method Causal Information Bottleneck (CIB) method, extending IB to include causal structures.
result CIB produces causally interpretable abstractions that accurately capture causal relations.
This work analyzes how bottleneck layers and skip connections affect linear denoising autoencoders' generalization.
problem Understanding the generalization of linear denoising autoencoders in overparameterized regimes.
method Analyzes two-layer linear denoising autoencoders with a bottleneck layer and skip connection, deriving test risk formulas.
result Bottleneck layers introduce an additional complexity measure, while skip connections can mitigate variance.
Extends ONNX for quantized neural networks with new formats and operators.
problem Handling arbitrary-precision quantization in neural networks.
method Introduces new formats and operators in ONNX to represent quantized neural networks.
result Enabled representation of uniform quantization in neural networks.
Deep latent variable models are powerful tools for representation learning. In this paper, we adopt the deep information bottleneck model, identify its shortcomings and propose a model that circumvents them. To this end, we apply a copula transformation which, by restoring the invariance properties of the information b…
Unified framework for robust, stable, and efficient density ratio estimation.
problem Density-chasm and support-chasm problems in density ratio estimation.
method Dequantified diffusion-Schrödinger bridge (D3RE) framework with DDBI and DSBI.
result Offers uniform approximation and bounded time scores in theory and empirical performance.
A new method learns quantization boundaries in continuous space using tessellation.
problem Mapping between discrete and continuous distributions is difficult.
method Constructs normalizing flows on convex polytopes with exact likelihood evaluations.
result Improves likelihood evaluation and quantization learning across various data modalities.
New objective function improves model robustness.
problem Machine Learning robustness issues.
method Conditional Entropy Bottleneck (CEB) objective function.
result CEB models improve robust generalization.
Proposes a new method to selectively access privileged information in reinforcement learning.
problem Selective compression of privileged information in reinforcement learning.
method Formulates a variational bandwidth bottleneck to decide stochastically whether to access privileged information.
result Improves generalization and reduces access to costly information in reinforcement learning experiments.
HCBM improves deep learning explainability by non-linear concept aggregation.
problem Lack of explainable and accurate predictions in deep learning for high-stake decisions.
method Introduce Hoeffding Concept Bottleneck Models (HCBM) using Hoeffding functional decomposition of gradient-boosted trees for non-linear and sparse concept aggregation.
result HCBM outperforms standard linear CBM and is robust to interconcept leakage.
Concept bottleneck models enable concept manipulation for model interpretation.
problem Training models to directly predict labels without intermediate concepts.
method Train models to predict intermediate concepts at training time, then use these concepts to predict labels.
result Concept bottleneck models achieve competitive accuracy with standard models while enabling concept manipulation.
Estimates latent dimensionality for prediction tasks using mutual information.
problem Estimating the latent dimensionality needed for accurate prediction.
method Formulates the problem as an Information Bottleneck question and uses neural mutual information estimators with a hybrid critic to preserve latent geometry.
result The hybrid critic method provides a more accurate estimation of task-relevant dimensionality.
We present a simple case study, demonstrating that Variational Information Bottleneck (VIB) can improve a network's classification calibration as well as its ability to detect out-of-distribution data. Without explicitly being designed to do so, VIB gives two natural metrics for handling and quantifying uncertainty.
Graph rewiring method alleviates over-squashing in GNNs.
problem Over-squashing and bottlenecks in graph neural networks.
method Curvature-based graph rewiring method.
result Negatively curved edges cause over-squashing in GNNs.
Unified information-theoretic objectives for training deep neural networks.
problem Difficulty in computing information-theoretic quantities for large deep neural networks.
method Review and unify competing objectives, develop surrogate objectives.
result Surrogate objectives allow applying information bottleneck to modern neural network architectures.