Variational inference is a powerful concept that underlies many iterative approximation algorithms; expectation propagation, mean-field methods and belief propagations were all central themes at the school that can be perceived from this unifying framework. The lectures of Manfred Opper introduce the archetypal example…
Unified framework for Gaussian process approximations using Power EP.
problem Computational and analytical intractabilities in Gaussian process applications.
method Power Expectation Propagation for pseudo-point approximations of Gaussian processes.
result Unified framework outperforms existing methods on regression and classification tasks.
CEP improves inference efficiency and accuracy by conditional moment matching.
problem Intractable moment matching in EP.
method Conditional expectation propagation (CEP) performs conditional moment matching and expectation.
result CEP achieves better inference quality and efficiency.
Loss-calibrated EP improves Bayesian decision-making by focusing on utility-sensitive posterior approximations.
problem Bayesian decision-making under asymmetric utility functions.
method Loss-calibrated expectation propagation (Loss-EP) that tilts the posterior towards higher utility decisions.
result Loss-EP can capture useful information for decision-making under asymmetric penalties.
Bayesian learning is often hampered by large computational expense. As a powerful generalization of popular belief propagation, expectation propagation (EP) efficiently approximates the exact Bayesian computation. Nevertheless, EP can be sensitive to outliers and suffer from divergence for difficult cases. To address t…
Unified framework for efficient Gaussian process inference.
problem Efficient inference in non-conjugate Gaussian process models.
method Combines expectation propagation with linearization for improved efficiency.
result Unified view of various inference schemes, including classical smoothers and EP.
Graphical-GAN combines Bayesian networks and GANs for structured data modeling.
problem Modeling structured data with complex dependencies.
method Integrates Bayesian networks and GANs, introduces structured recognition model, and generalizes EP algorithm.
result Successfully learns discrete and temporal structures on visual datasets.
Study various numerical methods for expectation propagation in machine learning.
problem Improving the convergence and stability of the expectation propagation algorithm for large-scale learning tasks.
method Numerical approximation strategies including Laplace method, Gaussian quadrature, and variational sampling.
result Variational sampling yields the best convergence for the expectation propagation algorithm in training linear binary classifiers.
DP-SEP privatizes EP by refining a single factor per data point.
problem Private inference of complex models with limited memory.
method Stochastic Expectation Propagation with differential privacy.
result DP-SEP provides better posterior estimates with guaranteed privacy.
Efficiently trains Gaussian process classifiers on large datasets using expectation propagation.
problem Training Gaussian process classifiers on large datasets.
method Expectation propagation for distributed and stochastic training.
result The method can handle datasets with millions of data instances and is competitive with variational approaches.
Paper proposes a new MIMO detection algorithm using Gaussian Mixture Expectation Propagation.
problem Challenges in MIMO detection due to interference and noise in high-order high-dimensional systems.
method The approach uses a Gaussian Mixture Model (GMM) approximation for Belief Propagation (BP) and Expectation Propagation (EP) messages to improve detection accuracy.
result The proposed algorithm outperforms state-of-the-art detection algorithms while maintaining low computational complexity.
Binary classification models get more efficient predictive probabilities.
problem Computing predictive probabilities in Bayesian probit models is computationally challenging.
method Use of expectation propagation (EP) to find a closed-form expression for predictive probabilities.
result Closed-form predictive probabilities improve over existing methods.
A new particle-based LPB for continuous MRFs with improved accuracy and efficiency.
problem Efficient inference in pairwise MRFs with continuous states.
method Adaptive EP-based proposal distributions for particle belief propagation.
result Proposed algorithm provides more accurate results than PBP with lower computational cost.
Efficient EP algorithm improves smoothing distribution inference in financial models.
problem Computational intractability of smoothing distribution in high dimensions.
method Adapted expectation propagation (EP) algorithms for the unified skew-normal family.
result Accuracy gains in financial illustrations over existing approximate algorithms.
Paper generalizes EC inference for loopy models.
problem Probabilistic inference in loopy models.
method Generalized Expectation Consistency (GEC) method.
result GEC can be applied to MAP and MMSE estimation.
Bayesian method for feature selection with grouping info using expectation propagation.
problem Feature selection with grouping info and sparsity constraints.
method Sparse-group Bayesian feature selection using expectation propagation.
result Our method outperforms existing methods in terms of feature selection accuracy and computational efficiency.
Improved sparse Gaussian processes using structured scaling matrices and Power-EP framework.
problem Scaling Gaussian processes for large datasets.
method Structured diagonal scaling matrix and Power-EP framework.
result Structured approximations improve performance without increasing computational cost.
Proposes ETB for tighter bounds in robust training of deep networks.
problem Training robust deep neural networks to adversarial attacks.
method Proposes ETB, a provably tighter bound approach for interval bound propagation.
result Achieves orders of magnitudes tighter bounds compared to IBP.
Paper develops EP algorithm for t-exponential family using q-algebra.
problem Efficient learning algorithm for t-exponential family distributions.
method Borrowing q-algebra from statistical physics, develop EP algorithm.
result Demonstrates performance of EP algorithm on Bayes point machine and Student-t process classification.
Stochastic EP improves memory efficiency for large datasets in Gaussian process classification.
problem Memory limitations in EP for large datasets.
method Stochastic Expectation Propagation (EP) for large scale Gaussian process classification.
result Stochastic EP avoids memory scaling with dataset size, improving scalability.
This paper improves training of deep Gaussian processes using novel probabilistic backpropagation.
problem Training scalable deep Gaussian processes (DGPs) efficiently.
method Develops a novel extension of probabilistic backpropagation that leverages stochastic Expectation Propagation for scalable Bayesian learning of DGPs.
result Demonstrates improved performance over traditional Gaussian process regression on real-world datasets.
We propose a novel algorithm to solve the expectation propagation relaxation of Bayesian inference for continuous-variable graphical models. In contrast to most previous algorithms, our method is provably convergent. By marrying convergent EP ideas from (Opper&Winther 05) with covariance decoupling techniques (Wipf&Nag…
We consider the problem of joint modelling of metabolic signals and gene expression in systems biology applications. We propose an approach based on input-output factorial hidden Markov models and propose a structured variational inference approach to infer the structure and states of the model. We start from the class…
Unified framework for training SNNs using EP, faster convergence.
problem Training spiking neural networks with Expectation-Propagation.
method Message-passing framework for learning marginal distributions of SNN parameters.
result Faster convergence compared to gradient-based methods.
Bayesian model for sparse regression with spatial-temporal structure.
problem Sparse linear regression with spatio-temporal constraints.
method Hierarchical Gaussian process prior and Expectation Propagation algorithm.
result Model successfully applied to real data.
Paper introduces SNEP for efficient Bayesian learning and posterior server for distributed data.
problem Efficient Bayesian learning with distributed data.
method SNEP (stochastic natural gradient expectation propagation) and posterior server architecture.
result SNEP converges and improves Bayesian learning efficiency.
EP method speeds up Bayesian probit regression in high dimensions.
problem Computational challenges in high-dimensional Bayesian probit regression.
method Adapting EP approximation to multivariate Gaussian prior and skew-normal distribution.
result EP routine is computationally feasible in high-dimensional settings.
SEP extends EP for large datasets with reduced memory usage.
problem Large memory overhead in EP for large datasets.
method Stochastic Expectation Propagation (SEP) maintains a global posterior approximation while updating it locally.
result SEP reduces memory consumption by a factor of N compared to full EP.
Dynamic Bayesian networks can perform complex computations.
problem Understanding the computational limits of dynamic Bayesian networks.
method Simulation of Turing-complete computation using modified belief propagation algorithms.
result Dynamic Bayesian networks with continuous variables can perform complex computations.
Develops an efficient approximation for collapsed Gibbs sampling in complex models.
problem Intractability of integrating out variables in collapsed Gibbs sampling for complex models.
method Uses expectation propagation to approximate collapsed Gibbs integrals.
result Approximate sampler enables a runtime-accuracy tradeoff in sampling complex models.
A new method for inferring latent states in Markov jump processes.
problem Intractable exact inference for Markov jump processes.
method Entropic matching within expectation propagation.
result Superior performance in approximating the mean of the posterior process.
Belief propagation recovers backpropagation results.
problem Connection between backpropagation and belief propagation poorly understood.
method Converted backpropagation input to belief propagation input and showed results.
result Backpropagation is a special case of belief propagation.
Bayesian EP solves CS problems more accurately than other methods.
problem Finding sparse solutions to underdetermined linear systems with constraints.
method Bayesian inference with Expectation Propagation (EP) for marginal distribution computation.
result EP outperforms other methods in solving CS problems with correlated sensing matrices.
New EP variants improve inference stability and efficiency.
problem Inference stability and efficiency issues in EP.
method Motivated by natural-gradient optimization, new EP variants are introduced that are robust to Monte Carlo noise and efficient with single samples.
result Improved stability and efficiency in inference tasks.
Many inference problems involving questions of optimality ask for the maximum or the minimum of a finite set of unknown quantities. This technical report derives the first two posterior moments of the maximum of two correlated Gaussian variables and the first two posterior moments of the two generating variables (corre…
This paper analyzes how errors accumulate in PCA's deflation method.
problem Error accumulation in PCA's deflation method.
method Mathematical analysis of inexact Hotelling's deflation method in two scenarios.
result Characterization of error propagation in PCA's deflation method.
A new gradient method reduces variance for non-reparameterizable distributions.
problem Efficient calculation of unbiased gradients for expectation-based objectives.
method GO Gradient, which applies to non-reparameterizable distributions and has low variance.
result GO Gradient reduces variance to the same level as reparameterization trick with one sample.
A new method for Bayesian neural networks using probabilistic backpropagation.
problem Approximating posterior distributions in Bayesian neural networks.
method Variational Expectation Propagation (VEP) with probabilistic backpropagation.
result Efficient algorithm for approximate integration over posterior distributions.
Paper compares hard and soft EM for BN learning from incomplete data.
problem Learning BNs from incomplete data using EM algorithms.
method Investigates the impact of imputation vs. belief propagation in hard and soft EM.
result A decision tree can guide practitioners in choosing the best EM algorithm.
Model for EEG source localization with spatio-temporal structure.
problem Solving MMV problems with spatio-temporal sparsity patterns.
method Generalized spike and slab prior with Expectation Propagation.
result Demonstrated viability of the proposed model and inference scheme.
A method for converting NIW parameters for better estimation.
problem Estimating parameters of multivariate normal distribution.
method Convergent procedure for converting mean parameters to natural parameters in NIW family.
result Maximum likelihood estimation of natural parameters from observed statistics.
We propose to learn a kernel-based message operator which takes as input all expectation propagation (EP) incoming messages to a factor node and produces an outgoing message. In ordinary EP, computing an outgoing message involves estimating a multivariate integral which may not have an analytic expression. Learning suc…
Paper proposes an alternating back-propagation algorithm for generator networks.
problem Learning realistic generator models of natural images, video sequences, and sounds.
method Alternating back-propagation algorithm that iterates inferential and learning steps.
result The alternating back-propagation algorithm can learn realistic generator models of natural images, video sequences, and sounds.
Efficient multi-class classification method for large datasets using EP.
problem Large-scale multi-class classification with Gaussian processes.
method Expectation Propagation (EP) for scalable training.
result EP method performs similarly or better than variational inference methods on large datasets.
New algorithm for Gaussian process classification using posterior linearisation.
problem Improving Gaussian process classification performance.
method Posterior linearisation to approximate posterior density iteratively, accounting for linearisation error.
result PL has better performance than EP in experimental data.
DGPs use EP for scalable regression, outperforming GPs and other methods.
problem Scalable regression with deep Gaussian processes.
method Approximate Expectation Propagation for learning DGPs.
result DGPs outperform GPs and other methods on real-world datasets.
New theory explains signal propagation in normalization-free transformers.
problem Understanding signal propagation in normalization-free transformers.
method Deriving recurrence relations for activation statistics and APJNs across layers.
result Transformers with elementwise tanh-like nonlinearities exhibit subcritical signal propagation.
Two EP frameworks ensure integrable beliefs in Bayesian estimation problems.
problem Non-integrable beliefs in EP can lead to infeasible solutions.
method Proposes two EP frameworks to keep messages non-integrable.
result Ensures integrable beliefs in EP, even with non-integrable messages.