Variational inference improves neural network matrix factorization for stochastic blockmodels.
problem Improving predictive performance of neural network matrix factorization for stochastic blockmodels.
method Construct Bayesian neural networks and fit with variational inference.
result Variational inference can achieve equivalent performance to neural networks on Movielens data.
Variational Laplace improves Bayesian neural networks performance.
problem Improving Bayesian neural networks performance.
method Develops variational Laplace for BNNs, exploiting curvature of likelihood.
result Variational Laplace outperforms other inference methods.
Variational Laplace improves Bayesian neural network performance without sampling.
problem Improving Bayesian neural network performance and calibration.
method Develops a new variational Laplace method for BNNs, exploiting curvature of likelihood.
result Variational Laplace outperforms standard VI methods in test performance and calibration.
New framework explains deep neural networks using variational spline theory.
problem Understanding functions learned by deep neural networks.
method Developed a variational framework and function space.
result Deep ReLU networks are solutions to regularized data fitting problems over the proposed function space.
Improved uncertainty estimation in neural networks with VBLL.
problem Improving uncertainty estimation in neural networks.
method Deterministic variational formulation for training Bayesian last layer neural networks.
result Improves predictive accuracy, calibration, and out-of-distribution detection.
Variational Bayes simplifies Bayesian neural networks for uncertainty quantification.
problem Quantifying uncertainty in neural networks' outputs.
method Approximates intractable Bayesian integrals using variational methods.
result Comparison of various approximation methods in literature.
Neural networks solve variational inequalities for optimal stopping problems.
problem Solving variational inequalities for optimal stopping problems in finance.
method Proposed neural network approach using loss functions directly incorporating variational inequality on whole domain.
result Existence and convergence of neural networks whose losses converge to zero.
New Bayesian neural network model improves performance and compresses networks.
problem Improving predictive performance and compressing neural networks.
method Proposes a new variational family with radial and directional components.
result Improves predictive performance and yields compressed networks.
Study variation spaces for neural networks, linking them to approximation theory.
problem Understanding the variation spaces of shallow neural networks.
method Examined variation spaces defined by convex hulls and integral representations for a dictionary of functions.
result Found that Barron space, spectral Barron space, and Radon BV space are variation spaces for certain neural networks.
Closed-form variational objectives for Bayesian neural networks with ReLU layers.
problem Efficient computation of Bayesian neural networks with closed-form variational objectives.
method Single-layer networks with piecewise polynomial activations (ReLU). Structured Normal variational distributions for Normal likelihoods. Approximate lower bounds for other likelihoods.
result Closed-form computation of variational lower bounds, predictive mean, and variance for Bayesian neural networks.
Paper proposes a method to improve variational inference for sparse networks.
problem Variational inference struggles with sparse networks, leading to inaccurate community detection.
method The method involves hard thresholding the posterior of community assignment after each iteration.
result The proposed method accurately recovers true community labels in sparse networks.
Proposes a new method for continual learning in neural networks.
problem Challenges in applying sequential Bayesian inference to neural networks.
method Sequential function-space variational inference.
result Neural networks trained with the proposed method achieve better predictive accuracy.
Advances variational Bayesian neural networks using singular learning theory.
problem Discrepancies between predictive performance and variational objective in BNNs.
method Corrected asymptotic form of singular posterior distributions to inform variational family design.
result Improvements in variational free energy and generalization error with proposed normalizing flow.
Variational inference improves training of generative flow networks.
problem Training generative flow networks efficiently and accurately.
method Define variational objectives in terms of KL divergences and optimize convex combinations.
result Variational inference methods can reduce the variance of gradients in training generative flow networks.
IVON optimizes large neural networks, matching or outperforming Adam.
problem The inefficacy of variational learning in large neural networks.
method Improved Variational Online Newton (IVON) optimizer.
result IVON consistently matches or outperforms Adam for large networks.
Bayesian neural networks ignore data in infinite units limit.
problem Pathological behavior of posterior in over-parameterized networks.
method Mean-field variational inference in infinite hidden units limit.
result Posterior mean converges to zero, ignoring data.
The motivations for using variational inference (VI) in neural networks differ significantly from those in latent variable models. This has a counter-intuitive consequence; more expressive variational approximations can provide significantly worse predictions as compared to those with less expressive families. In this …
Develops VPINNs for solving PDEs with reduced training cost and improved accuracy.
problem Solving partial differential equations efficiently and accurately.
method Integrates variational forms of PDEs into neural network loss functions, using Legendre polynomials as test spaces.
result VPINNs outperform PINNs in terms of accuracy and speed for solving PDEs.
Deep neural networks maximize variation when few nodes change activation.
problem Maximizing variation in deep neural networks.
method Theoretical analysis of ReLU activation function and layer node numbers.
result Maximal variation occurs when few nodes change activation.
A new method learns text network embeddings by combining generative autoencoder and homophilic priors.
problem Improving performance of network learning applications, especially for textual networks.
method Variational Homophilic Embedding (VHE) - a fully generative model that optimizes a variational autoencoder for semantic information and a homophilic prior for structural information.
result VHE outperforms existing methods in various tasks on real-world textual networks.
Variational inference provides approximations to the computationally intractable posterior distribution in Bayesian networks. A prominent medical application of noisy-or Bayesian network is to infer potential diseases given observed symptoms. Previous studies focus on approximating a handful of complicated pathological…
New Variational InfoMax objective improves neural network performance.
problem Optimizing neural networks using Bayesian Inference and Information Bottleneck.
method Derive Variational InfoMax (VIM) objective that maximizes InfoMax directly.
result VIM improves model performance in accuracy, robustness, and representation quality.
Compact parameterization improves Bayesian neural network performance.
problem Improving performance of Bayesian neural networks using variational methods.
method Restricting variational distribution to a k-tied Normal distribution with low-rank factorization.
result Compact parameterization improves signal-to-noise ratio and convergence speed.
Amortized variational inference (AVI) replaces instance-specific local inference with a global inference network. While AVI has enabled efficient training of deep generative models such as variational autoencoders (VAE), recent empirical work suggests that inference networks can produce suboptimal variational parameter…
fBNNs use stochastic processes for variational inference in neural networks.
problem Difficulties in specifying priors and posteriors in high-dimensional weight spaces.
method Maximize Evidence Lower Bound (ELBO) on stochastic processes, using spectral Stein gradient estimator.
result fBNNs provide reliable uncertainty estimates and extrapolate well with structured priors.
NVGD uses neural networks to infer distributions without kernel choices.
problem Challenges in choosing kernel functions for SVGD.
method NVGD parameterizes the witness function of the Stein discrepancy with a neural network.
result NVGD achieves good performance on various inference problems.
In this work we perform outlier detection using ensembles of neural networks obtained by variational approximation of the posterior in a Bayesian neural network setting. The variational parameters are obtained by sampling from the true posterior by gradient descent. We show our outlier detection results are comparable …
Recent efforts on combining deep models with probabilistic graphical models are promising in providing flexible models that are also easy to interpret. We propose a variational message-passing algorithm for variational inference in such models. We make three contributions. First, we propose structured inference network…
New method makes variational inference robust for Bayesian neural networks.
problem Fragility of variational Bayes in neural networks.
method Deterministic approximation of moments and hierarchical prior selection.
result Good predictive performance in heteroscedastic regression.
Elvet solves differential equations and variational problems with neural networks.
problem Solving complex differential and variational equations with arbitrary conditions.
method Machine learning, specifically neural networks, to represent and solve equations.
result Elvet can solve a wide range of differential and variational problems.
SFSVI uses Gaussian mixtures to approximate neural network outputs for continual learning.
problem Learning new tasks without forgetting old ones in neural networks.
method Sequential function-space variational inference with Gaussian mixture approximation.
result Gaussian mixture SFSVI outperforms other methods in continual learning.
New variational formula for Rényi divergences improves neural network estimation in high dimensions.
problem Estimating Rényi divergences in high-dimensional systems.
method Derive and apply a variational formula for Rényi divergences over various function spaces.
result Neural network estimators of Rényi divergences are consistent under certain conditions.
Variational neural networks optimize activation functions using gradient descent.
problem Lack of guiding principles for choosing activation functions in neural networks.
method Variational neural networks use a linear combination of candidate functions, optimizing via gradient descent.
result Optimal activation functions can be found using gradient descent.
We propose a general framework for solving statistical mechanics of systems with finite size. The approach extends the celebrated variational mean-field approaches using autoregressive neural networks, which support direct sampling and exact calculation of normalized probability of configurations. It computes variation…
Combines neural networks with variational inference for better uncertainty quantification.
problem Overconfident predictions from traditional neural networks and time-consuming Bayesian optimization.
method VIFO (Variational Inference on the Final-Layer Output) using neural networks to learn mean and variance.
result VIFO provides a good tradeoff in run time and uncertainty quantification, especially for out of distribution data.
Develops a new variational estimator for node popularity in bipartite networks.
problem Estimating node popularity in bipartite networks with varying patterns.
method Variational Expectation-Maximization (VEM) framework for the Two-Way Node Popularity Model (TNPM).
result The proposed method achieves superior estimation accuracy across different types of networks.
BBNN improves neural network accuracy and uncertainty quantification.
problem Overfitting and lack of interpretability in probabilistic neural networks.
method Boosted Bayesian Neural Networks (BBNN) using Boosting Variational Inference (BVI).
result BBNN achieves ~5% higher accuracy and superior uncertainty quantification.
A new model integrates LSTM and copulas for high-dimensional financial data.
problem Modeling high-dimensional dependencies across financial markets.
method Variational LSTM with regular vine copulas.
result Outperforms benchmarks in cross-market portfolio forecasting.
We develop a method to learn neural network activations with controlled Lipschitz constant.
problem Increase neural network capacity while controlling Lipschitz constant.
method Variational framework to learn activation functions with piecewise-linear constraints.
result Proves existence of solutions with continuous and piecewise-linear activations.
Paper introduces vector-valued variation spaces for multi-output neural networks.
problem Understanding and optimizing multi-output neural networks.
method Development of vector-valued variation spaces and representer theorem.
result Novel bounds for layer widths in deep networks and a convex optimization method for compression.
TM-VI uses flexible transformation models to approximate complex posteriors in Bayesian models.
problem Approximating complex posteriors in Bayesian models with limited flexibility.
method Transformation models for variational inference (TM-VI).
result TM-VI allows accurate approximation of complex posteriors in models with one parameter and works in a mean-field fashion for multi-parameter models.
Improves Bayesian neural networks inference efficiency and accuracy.
problem Inflexibility of factorized structure in Dropout posterior.
method Introduces Variational Structured Dropout (VSD) with orthogonal transformation.
result VSD induces adaptive regularization and better generalization.
GraphVRNN generates graphs with latent variables and node attributes.
problem Generating diverse and complex graph structures.
method Probabilistic autoregressive model for graph generation.
result GraphVRNN can model complicated distributions and generate plausible structures and node attributes.
Improved neural network inference with eigenvalue correction.
problem Inference of flexible variational posteriors is computationally expensive.
method Eigenvalue correction to matrix-variate Gaussian posterior.
result Empirically, the method outperforms existing algorithms.
Variational dropout and ARD are shown to be equivalent, improving sparsity in neural networks.
problem Improper priors in variational dropout models and theoretical issues.
method Applying ARD to Bayesian DNNs with Gaussian approximations and exploring hierarchical priors.
result Variational dropout and ARD objectives are equivalent, with improved sparsity.
Solves the VAE learning issue by optimizing a more informative generator.
problem Learning issue in VAE where only one model can be optimal at a time.
method Introduces Variational InfoMax (VIM) learning objective to optimize both inference and generative models.
result Derives a learning objective that optimizes both inference and generative models.
We use neural networks as control variates with geometric integration techniques.
problem Analytic integration of neural network approximations for variance reduction.
method Integration domain subdivision using computational geometry for MLPs with continuous piecewise linear activation functions.
result Neural networks can be used as control variates with geometric integration methods.
We face network data from various sources, such as protein interactions and online social networks. A critical problem is to model network interactions and identify latent groups of network nodes. This problem is challenging due to many reasons. For example, the network nodes are interdependent instead of independent o…