In this paper, we provide an information-theoretic interpretation of the Vector Quantized-Variational Autoencoder (VQ-VAE). We show that the loss function of the original VQ-VAE can be derived from the variational deterministic information bottleneck (VDIB) principle. On the other hand, the VQ-VAE trained by the Expect…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Variation of Information”
PRI-VAE learns disentangled representations by optimizing principle-of-relevant-information.
problem Learning disentangled representations under VAE framework remains unknown.
method Proposes PRI-VAE, a novel learning objective to optimize disentanglement.
result Demonstrates effectiveness of PRI-VAE on four benchmark datasets.
New Variational InfoMax objective improves neural network performance.
problem Optimizing neural networks using Bayesian Inference and Information Bottleneck.
method Derive Variational InfoMax (VIM) objective that maximizes InfoMax directly.
result VIM improves model performance in accuracy, robustness, and representation quality.
Proposes a framework to maximize mutual information in VAE models for better latent code representation.
problem Lack of explicit measurement of the quality of learned representations in VAE models.
method Variational Mutual Information Maximization Framework for VAE.
result Maximizes mutual information between latent codes and observations, improving latent code representation.
New definition of disentanglement for non-independent factors of variation.
problem Current disentanglement definitions assume independent factors of variation, limiting their applicability.
method Definition based on information theory, related to Information Bottleneck Method, proposed measurement method.
result Proposed method correctly measures disentanglement with non-independent factors of variation.
In classic papers, Zellner demonstrated that Bayesian inference could be derived as the solution to an information theoretic functional. Below we derive a generalized form of this functional as a variational lower bound of a predictive information bottleneck objective. This generalized functional encompasses most moder…
Mutual information bounds generalization error in variational classifiers.
problem Controlling overfitting in variational classifiers.
method Derive bounds on generalization error using mutual information.
result Mutual information bounds the generalization error in variational classifiers.
Enhances graph modeling with hyperbolic geometry and variational inference.
problem Challenges in modeling relational data with complex dependencies.
method Semi-implicit hierarchical variational Bayes with Poincaré embedding and mutual information regularization.
result Improves graph representation quality and flexibility in edge prediction and node classification.
Feature selection is one of the most fundamental problems in machine learning. An extensive body of work on information-theoretic feature selection exists which is based on maximizing mutual information between subsets of features and class labels. Practical methods are forced to rely on approximations due to the diffi…
Variational autoencoders learn unsupervised data representations, but these models frequently converge to minima that fail to preserve meaningful semantic information. For example, variational autoencoders with autoregressive decoders often collapse into autodecoders, where they learn to ignore the encoder input. In th…
Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embeddingcs.LG
In this paper, we develop an unsupervised generative clustering framework that combines the Variational Information Bottleneck and the Gaussian Mixture Model. Specifically, in our approach, we use the Variational Information Bottleneck method and model the latent space as a mixture of Gaussians. We derive a bound on th…
Proposes a robust VIB approach using soft labels and mutual info estimation.
problem Improving robustness of VIB to adversarial perturbations.
method Refines categorical class information with soft labels from a reference network, relaxes Gaussian posterior assumption.
result Significantly outperforms benchmarked models on MNIST and CIFAR-10.
Proposes a multi-task learning model using variational information bottleneck.
problem Balancing performance and robustness across different tasks in multi-task learning.
method Variational Information Bottleneck (VIB) architecture for multi-task learning.
result The proposed model achieves competitive prediction accuracy under adversarial attacks.
A new method estimates mutual information using neural classifiers.
problem Estimating mutual information for high-dimensional data is challenging.
method Trains a classifier to estimate joint distribution probability.
result Demonstrates high accuracy and reduces variance compared to variational methods.
We present a simple case study, demonstrating that Variational Information Bottleneck (VIB) can improve a network's classification calibration as well as its ability to detect out-of-distribution data. Without explicitly being designed to do so, VIB gives two natural metrics for handling and quantifying uncertainty.
The key idea of variational auto-encoders (VAEs) resembles that of traditional auto-encoder models in which spatial information is supposed to be explicitly encoded in the latent space. However, the latent variables in VAEs are vectors, which can be interpreted as multiple feature maps of size 1x1. Such representations…
Unified notation simplifies information-theoretic concepts in machine learning.
problem Opaque notation for information-theoretic quantities in machine learning.
method Proposed a practical and unified notation for information-theoretic quantities.
result Unified notation facilitates new intuitions and rederivations in machine learning.
The paper proposes a method to learn structured representations from unlabeled data using mutual information maximization.
problem Learning structured representations from unlabeled data.
method Adversarial maximization of mutual information between a structured latent variable and a target variable.
result The proposed method outperforms current baselines in document hashing and yields highly compressed interpretable representations.
Optimizes kernel density ratios for better predictions and information measures.
problem Improving accuracy of kernel density estimates for density ratios.
method Derives an optimal weight function using calculus of variations.
result Reduces bias in kernel density estimates, leading to improved prediction posteriors and information-theoretic measures.
New method approximates diffusion process posteriors using moment functions.
problem Approximating posteriors of stochastic differential equations.
method Constructs variational process as controlled prior, approximates posterior with moment functions, uses natural gradient descent.
result Richer variational approximations for state-dependent diffusion terms.
The mutual information is a core statistical quantity that has applications in all areas of machine learning, whether this is in training of density models over multiple data modalities, in maximising the efficiency of noisy transmission channels, or when learning behaviour policies for exploration by artificial agents…
We investigate the problem of learning representations that are invariant to certain nuisance or sensitive factors of variation in the data while retaining as much of the remaining information as possible. Our model is based on a variational autoencoding architecture with priors that encourage independence between sens…
We introduce TrustVI, a fast second-order algorithm for black-box variational inference based on trust-region optimization and the reparameterization trick. At each iteration, TrustVI proposes and assesses a step based on minibatches of draws from the variational distribution. The algorithm provably converges to a stat…
In many applications, it is desirable to extract only the relevant aspects of data. A principled way to do this is the information bottleneck (IB) method, where one seeks a code that maximizes information about a 'relevance' variable, Y, while constraining the information encoded about the original data, X. Unfortunate…
This paper introduces a new member of the family of Variational Autoencoders (VAE) that constrains the rate of information transferred by the latent layer. The latent layer is interpreted as a communication channel, the information rate of which is bound by imposing a pre-set signal-to-noise ratio. The new constraint s…
New method scales Gaussian processes with derivatives using variational inference.
problem Scaling Gaussian processes with derivative information for high-dimensional problems.
method Introducing inducing directional derivatives to sparsify derivative information using variational inference.
result Achieves fully scalable Gaussian process regression with derivatives.
Algorithm improves variational inference in Wasserstein distance.
problem Improving variational inference methods for complex models.
method Wasserstein contraction analysis of coordinate ascent.
result General and sharp convergence guarantees for various models.
New method extracts factors of variation from data without much supervision.
problem Disentangling complex sensory inputs into simple factors of variation without much supervision.
method Develops a new approach for disentanglement under structural assumptions, reducing the need for auxiliary information.
result Disentanglement is possible even when auxiliary information does not ensure conditional independence, with less auxiliary information required.
Geometric Variational Inference improves efficiency in complex probability distributions.
problem Efficiently accessing information in non-linear and high-dimensional probability distributions.
method Geometric Variational Inference (geoVI) uses Riemannian geometry and the Fisher information metric to construct a coordinate transformation.
result geoVI provides a more efficient variational approximation by a normal distribution, demonstrated on various problems.
VB uses natural gradients in information geometry.
problem Estimating or computing natural gradients in VB.
method Natural-gradient descent algorithm and Bayesian Learning Rule.
result Simplification of Bayes' rule and generalization of quadratic surrogates.
CO-BED optimizes experiments using Bayesian methods and information theory.
problem Optimizing experiments in a context-dependent manner.
method Formalizes contextual optimization with Bayesian experimental design, employing information-theoretic principles and black-box variational methods.
result CO-BED provides a general solution for contextual optimization problems.
VES-Gamma adapts EI using information-theoretic principles.
problem Optimizing black-box functions using Bayesian optimization.
method Variational Entropy Search (VES) and VES-Gamma algorithm.
result VES-Gamma improves EI by incorporating information-theoretic concepts.
New MIF architecture improves posterior approximations in Bayesian models.
problem Challenges in variational inference for complex hierarchical models.
method Combines VIP and autoregressive flow with prior information and hierarchical ordering.
result Empirically, MIF delivers tighter posterior approximations and state-of-the-art performance.
KFT improves tensor forecasting by incorporating side information.
problem Tensor factorization weaknesses in latent factors.
method Kernel Fried Tensor (KFT) with variational inference.
result Superior performance over LightGBM and FFM.
pVAE combines physics and machine learning for robust data representations.
problem Improving machine learning models' robustness to environmental factors of variation.
method Physics-informed variational autoencoder integrating physical knowledge with neural networks.
result pVAE outperforms competing models in extrapolation and interpretability.
Proposes a new method to selectively access privileged information in reinforcement learning.
problem Selective compression of privileged information in reinforcement learning.
method Formulates a variational bandwidth bottleneck to decide stochastically whether to access privileged information.
result Improves generalization and reduces access to costly information in reinforcement learning experiments.
Improved Monte-Carlo models by constraining mutual information between latent and observable variables.
problem Training density models leads to latent variables being useless.
method Weave tighter Monte-Carlo bounds with mutual information constraints.
result Improved training of models with continuous and discrete latent variables.
The performance of many network learning applications crucially hinges on the success of network embedding algorithms, which aim to encode rich network information into low-dimensional vertex-based vector representations. This paper considers a novel variational formulation of network embeddings, with special focus on …
Optimizes information acquisition to reduce estimation risk and maximize utility.
problem Estimation risk in investor decision-making.
method Derives closed-form value functions using CARA and CRRA utility functions, employs variational methods to explore optimal acquisition.
result Acquiring information earlier is more valuable in reducing estimation risk and achieving higher utility.
Meta learning with information theory and Gaussian processes.
problem Few-shot learning problems.
method Information bottleneck, mutual information, variational approximations, Gaussian processes.
result Competitive accuracy on few-shot classification problems.
A new method for Bayesian inference in high dimensions using projected Stein variational gradient descent.
problem Bayesian inference challenges in high-dimensional data.
method Adapting Stein variational gradient descent to exploit intrinsic low dimensionality of data.
result pSVGD is more accurate and efficient than SVGD, especially in high-dimensional settings.
Physics-informed neural networks (PINNs) [31] use automatic differentiation to solve partial differential equations (PDEs) by penalizing the PDE in the loss function at a random set of points in the domain of interest. Here, we develop a Petrov-Galerkin version of PINNs based on the nonlinear approximation of deep neur…
DIVI clusters noisy high-dimensional data with stable feature gating.
problem Challenging clustering in high-dimensional noisy data.
method Data-informed variational clustering framework combining global feature gating and adaptive structure growth.
result DIVI performs competitively under severe feature noise and remains computationally feasible.
Bayesian model selection via mean-field variational approximation improves efficiency and accuracy.
problem Bayesian model selection under model mis-specification and latent variables.
method Mean-field variational approximation with non-asymptotic properties and geometric convergence.
result ELBO tends to select models closer to the true model than BIC as sample size increases.
Proposes PI-VAE for solving SDEs with limited measurements.
problem Solving SDEs with limited measurements of system parameters.
method Physics-informed Variational Autoencoder (PI-VAE) integrating VAE and governing equations.
result Satisfactory accuracy and efficiency compared to PI-WGAN.
We use variational Gaussian approximations to analyze parametric models with unknown data-generating distributions.
problem Analyzing inference and learning in parametric models with unknown or intractable data-generating distributions.
method Replica method with variational Gaussian approximation in grand canonical formalism.
result Stationarity conditions adaptively determine parameters of the trial Hamiltonian for each dataset.
The Variational AutoEncoder (VAE) learns simultaneously an inference and a generative model, but only one of these models can be learned at optimum, this behaviour is associated to the ELBO learning objective, that is optimised by a non-informative generator. In order to solve such an issue, we provide a learning objec…
AMI framework improves text generation by optimizing mutual information between source and target.
problem Previous MI approaches ignored the backward network, leading to loose variational bounds.
method AMI is a saddle point optimization framework that iteratively promotes and demotes generated instances.
result AMI significantly outperforms baselines on various text generation tasks.