Overpruned variational approximations can lead to worse predictions in neural networks.
problem Performance gap between expressive and less expressive variational approximations in neural networks.
method Identify and explain variational over-pruning as the cause of the performance gap.
result Theoretical explanation for the performance gap between expressive and less expressive variational approximations.
Variational inference improves neural network matrix factorization for stochastic blockmodels.
problem Improving predictive performance of neural network matrix factorization for stochastic blockmodels.
method Construct Bayesian neural networks and fit with variational inference.
result Variational inference can achieve equivalent performance to neural networks on Movielens data.
Efficient neural network ensembles detect outliers.
problem Outlier detection in neural networks.
method Variational Bayesian neural networks with gradient descent sampling.
result Outlier detection results comparable to other methods.
Variational Laplace improves Bayesian neural networks performance.
problem Improving Bayesian neural networks performance.
method Develops variational Laplace for BNNs, exploiting curvature of likelihood.
result Variational Laplace outperforms other inference methods.
Variational Laplace improves Bayesian neural network performance without sampling.
problem Improving Bayesian neural network performance and calibration.
method Develops a new variational Laplace method for BNNs, exploiting curvature of likelihood.
result Variational Laplace outperforms standard VI methods in test performance and calibration.
Improved uncertainty estimation in neural networks with VBLL.
problem Improving uncertainty estimation in neural networks.
method Deterministic variational formulation for training Bayesian last layer neural networks.
result Improves predictive accuracy, calibration, and out-of-distribution detection.
New framework explains deep neural networks using variational spline theory.
problem Understanding functions learned by deep neural networks.
method Developed a variational framework and function space.
result Deep ReLU networks are solutions to regularized data fitting problems over the proposed function space.
Neural networks solve variational inequalities for optimal stopping problems.
problem Solving variational inequalities for optimal stopping problems in finance.
method Proposed neural network approach using loss functions directly incorporating variational inequality on whole domain.
result Existence and convergence of neural networks whose losses converge to zero.
NVGD uses neural networks to infer distributions without kernel choices.
problem Challenges in choosing kernel functions for SVGD.
method NVGD parameterizes the witness function of the Stein discrepancy with a neural network.
result NVGD achieves good performance on various inference problems.
Study variation spaces for neural networks, linking them to approximation theory.
problem Understanding the variation spaces of shallow neural networks.
method Examined variation spaces defined by convex hulls and integral representations for a dictionary of functions.
result Found that Barron space, spectral Barron space, and Radon BV space are variation spaces for certain neural networks.
Variational Bayes simplifies Bayesian neural networks for uncertainty quantification.
problem Quantifying uncertainty in neural networks' outputs.
method Approximates intractable Bayesian integrals using variational methods.
result Comparison of various approximation methods in literature.
Optimizes MCMC chains with neural control variates.
problem Reducing variance in Markov Chain Monte Carlo (MCMC) simulations.
method Uses neural networks as control variates to minimize asymptotic variance.
result Derives optimal convergence rate under various ergodicity assumptions.
Quantum neural tangent kernels help understand variational quantum circuits in machine learning.
problem Designing and predicting performance of variational quantum circuits.
method Using quantum neural tangent kernels and dynamical equations for loss functions.
result Analytical solutions for training dynamics in variational quantum circuits.
Proposes a new method for continual learning in neural networks.
problem Challenges in applying sequential Bayesian inference to neural networks.
method Sequential function-space variational inference.
result Neural networks trained with the proposed method achieve better predictive accuracy.
Bayesian neural networks ignore data in infinite units limit.
problem Pathological behavior of posterior in over-parameterized networks.
method Mean-field variational inference in infinite hidden units limit.
result Posterior mean converges to zero, ignoring data.
Deep neural networks maximize variation when few nodes change activation.
problem Maximizing variation in deep neural networks.
method Theoretical analysis of ReLU activation function and layer node numbers.
result Maximal variation occurs when few nodes change activation.
New Bayesian neural network model improves performance and compresses networks.
problem Improving predictive performance and compressing neural networks.
method Proposes a new variational family with radial and directional components.
result Improves predictive performance and yields compressed networks.
Advances variational Bayesian neural networks using singular learning theory.
problem Discrepancies between predictive performance and variational objective in BNNs.
method Corrected asymptotic form of singular posterior distributions to inform variational family design.
result Improvements in variational free energy and generalization error with proposed normalizing flow.
Closed-form variational objectives for Bayesian neural networks with ReLU layers.
problem Efficient computation of Bayesian neural networks with closed-form variational objectives.
method Single-layer networks with piecewise polynomial activations (ReLU). Structured Normal variational distributions for Normal likelihoods. Approximate lower bounds for other likelihoods.
result Closed-form computation of variational lower bounds, predictive mean, and variance for Bayesian neural networks.
fBNNs use stochastic processes for variational inference in neural networks.
problem Difficulties in specifying priors and posteriors in high-dimensional weight spaces.
method Maximize Evidence Lower Bound (ELBO) on stochastic processes, using spectral Stein gradient estimator.
result fBNNs provide reliable uncertainty estimates and extrapolate well with structured priors.
New method makes variational inference robust for Bayesian neural networks.
problem Fragility of variational Bayes in neural networks.
method Deterministic approximation of moments and hierarchical prior selection.
result Good predictive performance in heteroscedastic regression.
Compact parameterization improves Bayesian neural network performance.
problem Improving performance of Bayesian neural networks using variational methods.
method Restricting variational distribution to a k-tied Normal distribution with low-rank factorization.
result Compact parameterization improves signal-to-noise ratio and convergence speed.
DSVNP uses global and local latent variables for improved neural process predictions.
problem Limited expressiveness of vanilla neural processes in capturing target-specific local variation.
method Introduces DSVNP combining global and local latent variables for prediction.
result Competitive prediction performance in multi-output regression and uncertainty estimation.
Develops VPINNs for solving PDEs with reduced training cost and improved accuracy.
problem Solving partial differential equations efficiently and accurately.
method Integrates variational forms of PDEs into neural network loss functions, using Legendre polynomials as test spaces.
result VPINNs outperform PINNs in terms of accuracy and speed for solving PDEs.
Elvet solves differential equations and variational problems with neural networks.
problem Solving complex differential and variational equations with arbitrary conditions.
method Machine learning, specifically neural networks, to represent and solve equations.
result Elvet can solve a wide range of differential and variational problems.
New method guarantees global convergence in variational inference.
problem Limited convergence to local optima in variational inference.
method Minimizes inclusive KL divergence using neural networks and neural tangent kernel.
result Gradient descent dynamics converge to a unique solution in function space.
New method estimates uncertainty in knowledge graph embeddings using neural variational inference.
problem Estimating uncertainty in knowledge graph embeddings.
method Constructs an inference network conditioned on symbolic representations of entities and relation types in a Knowledge Graph.
result Improved predictive uncertainty estimates during link prediction.
New variational formula for Rényi divergences improves neural network estimation in high dimensions.
problem Estimating Rényi divergences in high-dimensional systems.
method Derive and apply a variational formula for Rényi divergences over various function spaces.
result Neural network estimators of Rényi divergences are consistent under certain conditions.
SFSVI uses Gaussian mixtures to approximate neural network outputs for continual learning.
problem Learning new tasks without forgetting old ones in neural networks.
method Sequential function-space variational inference with Gaussian mixture approximation.
result Gaussian mixture SFSVI outperforms other methods in continual learning.
NCV uses neural networks to improve Monte Carlo integration.
problem Improving variance reduction in parametric Monte Carlo integration.
method NCV combines a normalizing flow and a neural network to approximate the integrand and solve the integral equation, with a neural importance sampler to estimate the difference.
result NCV achieves state-of-the-art performance in light transport simulation with reduced noise and negligible bias.
Variational neural networks optimize activation functions using gradient descent.
problem Lack of guiding principles for choosing activation functions in neural networks.
method Variational neural networks use a linear combination of candidate functions, optimizing via gradient descent.
result Optimal activation functions can be found using gradient descent.
Recent advances in neural variational inference have spawned a renaissance in deep latent variable models. In this paper we introduce a generic variational inference framework for generative and conditional models of text. While traditional variational methods derive an analytic approximation for the intractable distri…
New method uses variational inference for better reinforcement learning policies.
problem Improving policy gradient methods in reinforcement learning.
method Explicitly minimizing KL divergence to match target distribution for Bayesian parameterization.
result Enhanced performance in TRPO, PPO, and Bayesian Neural Network reinforcement learning.
Combines neural networks with variational inference for better uncertainty quantification.
problem Overconfident predictions from traditional neural networks and time-consuming Bayesian optimization.
method VIFO (Variational Inference on the Final-Layer Output) using neural networks to learn mean and variance.
result VIFO provides a good tradeoff in run time and uncertainty quantification, especially for out of distribution data.
We use neural networks as control variates with geometric integration techniques.
problem Analytic integration of neural network approximations for variance reduction.
method Integration domain subdivision using computational geometry for MLPs with continuous piecewise linear activation functions.
result Neural networks can be used as control variates with geometric integration methods.
BBNN improves neural network accuracy and uncertainty quantification.
problem Overfitting and lack of interpretability in probabilistic neural networks.
method Boosted Bayesian Neural Networks (BBNN) using Boosting Variational Inference (BVI).
result BBNN achieves ~5% higher accuracy and superior uncertainty quantification.
Proposes neural networks for variance reduction in Monte Carlo estimation.
problem High variance in Monte Carlo estimations for complex functions.
method Uses neural networks to learn control variates from auxiliary random variables.
result Significant variance reduction in thermodynamic integration and reinforcement learning.
New method uses neural networks to solve statistical mechanics problems.
problem Statistical mechanics of systems with finite size.
method Variational autoregressive neural networks with reinforcement learning.
result Directly computes free energy, entropy, magnetizations, and correlations.
Bayesian sparsification reduces memory usage in RNNs without significant loss of accuracy.
problem High memory usage in RNNs for text analysis tasks.
method Apply Sparse Variational Dropout and Binary Variational Dropout to sparsify RNNs.
result Achieve 99.5% sparsity on sentiment analysis with no quality drop and up to 87% on language modeling with slight loss of accuracy.
Develops unbiased variational inference method for better model performance.
problem Improving variational inference methods for better model performance.
method Defines an expressive variational family using a simple reparameterizable distribution and deep neural networks, directly optimizing the ELBO.
result Achieves tighter ELBO and better predictive performance than existing approaches at similar computational cost.
Noisy natural gradient improves variational inference for Bayesian neural nets.
problem Tradeoff between simple and complex variational families in Bayesian neural nets.
method Adaptive weight noise in natural gradient ascent to implicitly fit variational posteriors.
result Noisy natural gradient algorithms can train full-covariance variational posteriors efficiently.
Improved Bayesian neural networks using normalizing flows for better predictive accuracy and uncertainty.
problem Improving predictive accuracy and uncertainty in Bayesian neural networks.
method Interpreting multiplicative noise as auxiliary variables and using normalizing flows for better approximations.
result Significant improvement in predictive accuracy and uncertainty compared to classical mean field methods.
We develop a variational framework for SDEs driven by fractional noise.
problem Capturing long-term dependencies in SDEs driven by fractional noise.
method Markov approximation of fractional Brownian motion, variational inference, neural networks.
result Efficient variational inference of posterior path measures for neural-SDEs.
A new method infers neural trajectories in real-time, improving experimental design.
problem Real-time inference of neural trajectories for immediate feedback.
method Exponential family variational Kalman filter (eVKF) for online learning.
result eVKF achieves competitive performance on synthetic and real-world data.
UDN adapts depth to data complexity, outperforming standard neural networks.
problem Adapting neural network depth to data complexity.
method Variational inference for infinitely deep neural networks with a novel algorithm.
result UDN outperforms standard neural networks and other infinite-depth approaches.
TM-VI uses flexible transformation models to approximate complex posteriors in Bayesian models.
problem Approximating complex posteriors in Bayesian models with limited flexibility.
method Transformation models for variational inference (TM-VI).
result TM-VI allows accurate approximation of complex posteriors in models with one parameter and works in a mean-field fashion for multi-parameter models.
Proposes MOPED method for choosing priors in Bayesian DNNs.
problem Challenges in specifying meaningful priors for deep neural networks.
method Two-stage hierarchical modeling with empirical Bayes.
result MOPED enables scalable variational inference and reliable uncertainty quantification.
New Variational InfoMax objective improves neural network performance.
problem Optimizing neural networks using Bayesian Inference and Information Bottleneck.
method Derive Variational InfoMax (VIM) objective that maximizes InfoMax directly.
result VIM improves model performance in accuracy, robustness, and representation quality.