The recognition network in deep latent variable models such as variational autoencoders (VAEs) relies on amortized inference for efficient posterior approximation that can scale up to large datasets. However, this technique has also been demonstrated to select suboptimal variational parameters, often resulting in consi…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,982 papers · 148 categories
Trend · papers per month
48 results for “variational training”
This study examines how randomness affects machine learning model performance.
problem The impact of randomness on machine learning model performance.
method Empirical study comparing randomness in model training and dataset partitioning.
result Randomness in model training causes more variation in FFNNs than tree-based methods.
A new method reduces variance in training discrete latent variable models.
problem High variance in stochastic gradient estimators for discrete latent variable models.
method Double control variates for score function estimators using Taylor expansions.
result Our method can have lower variance compared to other estimators.
Amortized variational inference (AVI) replaces instance-specific local inference with a global inference network. While AVI has enabled efficient training of deep generative models such as variational autoencoders (VAE), recent empirical work suggests that inference networks can produce suboptimal variational parameter…
New method trains latent variable models using various f-divergences.
problem Training latent variable models with alternative f-divergences.
method Variational approach combined with Spread Divergence.
result General method for training models with any f-divergence.
Combines variational autoencoders with normalizing flows for faster training.
problem Training normalizing flow models like Glow is slow and requires deep architectures.
method Integrates Glow with a variational autoencoder to speed up training.
result The combined model achieves similar image quality and likelihood to Glow but trains faster.
DGPs with variational inference suffer from SNR issues that degrade gradient estimates, leading to unreliable training.
problem SNR issues in gradient estimates for DGPs with variational inference.
method Adapted doubly reparameterized gradient estimators for DGP training.
result Fix improves predictive performance of DGP models.
Stabilizing Training of Generative Adversarial Nets via Langevin Stein Variational Gradient Descentcs.LG
Paper proposes LSVGD to stabilize GAN training via Langevin Stein Variational Gradient Descent.
problem Mode collapse and performance deterioration in GAN training.
method Langevin Stein Variational Gradient Descent (LSVGD) incorporating noise to stabilize training.
result LSVGD improves performance and stability of various GAN models.
Unbalanced GANs stabilize GAN training by pre-training the generator with VAE.
problem Stable training of GANs to avoid mode collapses and improve image quality.
method Pre-train GAN generator with VAE, balance generator and discriminator training, prevent discriminator's early convergence.
result Unbalanced GANs reduce mode collapses and outperform ordinary GANs in stability, convergence, and image quality.
In this paper, we introduce a new form of amortized variational inference by using the forward KL divergence in a joint-contrastive variational loss. The resulting forward amortized variational inference is a likelihood-free method as its gradient can be sampled without bias and without requiring any evaluation of eith…
Variational inference improves training of generative flow networks.
problem Training generative flow networks efficiently and accurately.
method Define variational objectives in terms of KL divergences and optimize convex combinations.
result Variational inference methods can reduce the variance of gradients in training generative flow networks.
Semi-Implicit Variational Inference (SIVI) is improved with SIVI-SM using score matching.
problem Intractable densities in variational distributions hinder SIVI training.
method SIVI-SM uses score matching to handle intractable densities in a minimax formulation.
result SIVI-SM outperforms ELBO-based SIVI methods in Bayesian inference tasks.
Bayesian method improves SOM training for dynamic data.
problem Training Self Organizing Maps (SOM) for non-stationary data.
method Variational Bayesian approach with adaptive neighborhood radius.
result Method outperforms other adaptive methods in high dimensions.
Lower bound for VAE training objective for binary data.
problem Finding a lower bound for the ELBO of Bernoulli VAE.
method Interpretable lower bound, modified initialization, faster training architecture, PCA for latent space dimension.
result Theoretical result and improved performance of new architecture.
Improved uncertainty estimation in neural networks with VBLL.
problem Improving uncertainty estimation in neural networks.
method Deterministic variational formulation for training Bayesian last layer neural networks.
result Improves predictive accuracy, calibration, and out-of-distribution detection.
Simplified DGPs training by fixing inducing inputs to subset of data.
problem Challenging training of deep Gaussian processes.
method Fixed subset of data for inducing inputs, variational sampling.
result Significant reduction in trainable parameters and computation cost without performance degradation.
Proposes a new method for continual learning in neural networks.
problem Challenges in applying sequential Bayesian inference to neural networks.
method Sequential function-space variational inference.
result Neural networks trained with the proposed method achieve better predictive accuracy.
Stable training of deep normalizing flows for high-dimensional variational inference.
problem Training deep normalizing flows for high-dimensional posterior distributions is infeasible due to high stochastic gradient variance.
method Proposed a combination of soft-thresholding of scale and bijective soft log transformation to stabilize training.
result Stable training of Real NVPs for posterior distributions with thousands of dimensions is possible.
We reinterpreting the variational inference in a new perspective. Via this way, we can easily prove that EM algorithm, VAE, GAN, AAE, ALI(BiGAN) are all special cases of variational inference. The proof also reveals the loss of standard GAN is incomplete and it explains why we need to train GAN cautiously. From that, w…
New methods reduce communication in distributed training for variational inequalities.
problem Reducing communication in distributed training for high-dimensional models.
method Distributed methods with compressed communication for solving variational inequalities.
result Theoretical guarantees and practical algorithms for compressed communication.
Unified analysis of efficient local training methods for distributed variational inequalities.
problem Efficient distributed/federated learning for variational inequality problems.
method Unified convergence analysis of communication-efficient local training methods.
result First local gradient descent-accent algorithms with improved communication complexity.
Proposes a new framework for uncertainty-aware LLM post-training.
problem Heterogeneous, conflicting data in large language models.
method α-Rényi variational framework for learning distributions over post-training parameters.
result Enables training examples to be softly routed across ensemble members, promoting model specialisation and providing uncertainty estimates.
Proposes model-based robust deep learning to handle natural variation in data.
problem Deep learning's fragility to natural variation in data.
method Develops model-based robust training algorithms using deep generative models to learn natural variation.
result Deep neural networks trained with model-based algorithms outperform standard and norm-bounded robust algorithms.
Improved GAN training stability through clipping and reweighting.
problem Inconsistent GAN training leading to inferior performance.
method Proposes a variational GAN framework with probability ratio clipping and sample reweighting.
result Significantly improved performance across various GAN tasks.
Alternative neural network training using monotone variational inequality.
problem Training neural networks efficiently and with guarantees.
method Using monotone variational inequality to solve non-convex problems efficiently.
result Our approach leads to fast convergence and competitive performance compared to traditional methods.
RTVAE uses β-divergence to detect anomalies in tabular data robustly.
problem Outliers in tabular data affect VAE training and anomaly detection.
method Robust VAE with β-divergence for mixed categorical and continuous features.
result Demonstrates effectiveness on network traffic anomaly detection.
New method reduces training cost by using approximate gradients.
problem Training neural networks is computationally expensive.
method Uses control variates to approximate gradients without full backward pass.
result Efficacy demonstrated on a vision transformer classification task.
This paper proposes a method to train energy-based models using variational auto-encoders for efficient sampling.
problem Training energy-based models by maximum likelihood is challenging due to intractable partition functions and difficult sampling from the model distribution.
method The authors propose using a variational auto-encoder to initialize finite-step MCMC sampling, specifically Langevin dynamics, to train the energy-based model.
result The proposed method enables training energy-based models using maximum likelihood, generating samples comparable to GANs and EBMs.
New methods minimize GFlowNet training divergences for better sampling.
problem Training GFlowNets with KL divergence leads to biased and high-variance estimators.
method Design and implement efficient estimators for four divergence measures.
result Properly minimizing these divergences yields a provably correct and effective training scheme.
A standard Variational Autoencoder, with a Euclidean latent space, is structurally incapable of capturing topological properties of certain datasets. To remove topological obstructions, we introduce Diffusion Variational Autoencoders with arbitrary manifolds as a latent space. A Diffusion Variational Autoencoder uses t…
In this paper we propose two novel bounds for the log-likelihood based on Kullback-Leibler and the Rényi divergences, which can be used for variational inference and in particular for the training of Variational AutoEncoders. Our proposal is motivated by the difficulties encountered in training VAEs on continuous datas…
This paper improves prediction uncertainty estimation by inferring variation from neuron activation strength.
problem Estimating prediction uncertainty from ensemble methods is expensive and inaccurate.
method Introduced randomness into model training and inferred prediction variation from neuron activation strength.
result Average R squared on MovieLens is 0.56 and on Criteo is 0.81, with strong performance in variation detection.
A key advance in learning generative models is the use of amortized inference distributions that are jointly trained with the models. We find that existing training objectives for variational autoencoders can lead to inaccurate amortized inference distributions and, in some cases, improving the objective provably degra…
Proposes a new VAE model to avoid posterior collapse by modeling latent variable dependencies.
problem Posterior collapse in variational autoencoders due to assumption of factorized variational posterior.
method Introduces Gaussian Copula Variational Autoencoder (GCVAE) to model latent variable dependencies explicitly.
result Empirical results show GCVAE can avoid posterior collapse while maintaining competitive performance.
Proposes variational autoencoder for efficient MMSE estimation.
problem Efficient parameterized MMSE estimation for noisy observations.
method Variational autoencoder models data distribution, approximates MMSE.
result Proposed estimator performs well compared to state-of-the-art.
New bounds enable training of probabilistic models for deep networks.
problem Training scalable latent variable models for deep networks.
method Introducing new variational bounds for specific output layers of neural networks.
result Analytical bounds for certain output layers allow training without re-parameterization or Monte Carlo approximations.
New method improves GP regression by relaxing variational assumption.
problem Improving variational Gaussian processes for better predictive performance.
method Relaxing the variational assumption to a more general distribution for optimization.
result New tighter evidence lower bound for GP regression.
Proposes a robust VIB approach using soft labels and mutual info estimation.
problem Improving robustness of VIB to adversarial perturbations.
method Refines categorical class information with soft labels from a reference network, relaxes Gaussian posterior assumption.
result Significantly outperforms benchmarked models on MNIST and CIFAR-10.
This paper characterizes VAE training pathologies and their effects on tasks.
problem Characterizing VAE training pathologies and their impact on downstream tasks.
method Concretely characterizing conditions for VAE training pathologies and their connection to specific downstream tasks.
result Connects VAE training pathologies to specific downstream tasks like learning compressed and disentangled representations, adversarial robustness, and semi-supervised learning.
Bayesian optimization improved with variational last layer training.
problem Bayesian optimization performance on complex input correlations.
method Connecting variational Bayesian last layer training to exact conditioning in Gaussian processes, developing an efficient online training algorithm.
result VBLL networks significantly outperform GPs and match well-tuned GPs on benchmark tasks.
Paper develops efficient variational inference for sparse deep learning with theoretical guarantees.
problem Sparse deep learning's challenge of huge storage consumption and sparse structure recovery.
method Bayesian treatment with spike-and-slab priors and continuous relaxation of Bernoulli distribution for computationally efficient variational inferences.
result Provides variational posterior contraction rate, justifying consistency of the proposed method.
Quantum neural tangent kernels help understand variational quantum circuits in machine learning.
problem Designing and predicting performance of variational quantum circuits.
method Using quantum neural tangent kernels and dynamical equations for loss functions.
result Analytical solutions for training dynamics in variational quantum circuits.
In this paper, we provide an information-theoretic interpretation of the Vector Quantized-Variational Autoencoder (VQ-VAE). We show that the loss function of the original VQ-VAE can be derived from the variational deterministic information bottleneck (VDIB) principle. On the other hand, the VQ-VAE trained by the Expect…
PVI combines federated learning and variational inference for probabilistic model training.
problem Federated learning's lack of probabilistic model uncertainty estimation.
method Partitioned variational inference (PVI) framework for federated probabilistic model training.
result PVI unifies fragmented literature and demonstrates effectiveness in various federated settings.
Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary lat…
Neural attention has become central to many state-of-the-art models in natural language processing and related domains. Attention networks are an easy-to-train and effective method for softly simulating alignment; however, the approach does not marginalize over latent alignments in a probabilistic sense. This property …
Develops VPINNs for solving PDEs with reduced training cost and improved accuracy.
problem Solving partial differential equations efficiently and accurately.
method Integrates variational forms of PDEs into neural network loss functions, using Legendre polynomials as test spaces.
result VPINNs outperform PINNs in terms of accuracy and speed for solving PDEs.
Transport Score Climbing: Variational Inference Using Forward KL and Adaptive Neural Transportstat.ML
TSC uses HMC and adaptive transport maps to optimize forward KL for variational inference.
problem Variational inference underestimates uncertainty when minimizing reverse KL.
method TSC uses Hamiltonian Monte Carlo and adaptive transport maps to optimize KL(p||q).
result TSC achieves competitive performance in training variational autoencoders on large-scale data.