Expectation propagation (EP) is a powerful approximate inference algorithm. However, a critical barrier in applying EP is that the moment matching in message updates can be intractable. Handcrafting approximations is usually tricky, and lacks generalizability. Importance sampling is very expensive. While Laplace propag…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Moment Propagation”
Corrected moment-based methods improve inference in topic model regression.
problem Inferential difficulties in topic model plug-in workflow for regression.
method Corrected spectral moment methods for LDA, response-weighted word moments.
result Direct identification of regression coefficients without estimating topic shares.
Many inference problems involving questions of optimality ask for the maximum or the minimum of a finite set of unknown quantities. This technical report derives the first two posterior moments of the maximum of two correlated Gaussian variables and the first two posterior moments of the two generating variables (corre…
Bayesian learning is often hampered by large computational expense. As a powerful generalization of popular belief propagation, expectation propagation (EP) efficiently approximates the exact Bayesian computation. Nevertheless, EP can be sensitive to outliers and suffer from divergence for difficult cases. To address t…
The Score Kalman Filtereess.SY
A new filter reduces density fitting to a linear solve, improving performance on nonlinear systems.
problem Nonlinear Bayesian filtering challenges in representing belief distributions.
method Combines score matching with Stein's identity to avoid partition function evaluation.
result The Score Kalman Filter (SKF) outperforms existing methods on nonlinear systems.
New methods for uncertainty in neural networks with leaky ReLU activations.
problem Uncertainty in feed-forward neural networks with random input perturbations.
method Analytical expressions for PDF and moments of neural network output, linearization of leaky ReLU, Gaussian copula surrogate models.
result Accurate statistical results for large input perturbations, excellent agreement with Monte Carlo simulations.
To deepen our understanding of graph neural networks, we investigate the representation power of Graph Convolutional Networks (GCN) through the looking glass of graph moments, a key property of graph topology encoding path of various lengths. We find that GCNs are rather restrictive in learning graph moments. Without c…
A new framework for lightweight BNNs learns heteroscedastic uncertainties efficiently.
problem Learning heteroscedastic uncertainties from BNNs for lightweight networks.
method Embedding heteroscedastic variances into BNN parameters and using moment propagation for inference.
result Improves predictive performance for lightweight BNNs without increasing parameter count.
Paper presents a method to accurately quantify neural network uncertainty without sampling.
problem Uncertainty quantification in neural networks for reliability and robustness.
method Sample-free moment propagation technique for mean vectors and covariance matrices.
result Analytic solution for covariance of nonlinear activation functions.
New EP variants improve inference stability and efficiency.
problem Inference stability and efficiency issues in EP.
method Motivated by natural-gradient optimization, new EP variants are introduced that are robust to Monte Carlo noise and efficient with single samples.
result Improved stability and efficiency in inference tasks.
Proposes a new neural network architecture inspired by biology to improve learning and information flow.
problem Improving artificial neural networks to match biological neuron properties like multidirectional propagation and probabilistic modeling.
method Extends KAN approach with joint distribution neurons that can propagate values and distributions, including variance and higher-order moments.
result Proposed architecture can predict and propagate distributions, including expected values and variances.
This paper makes two contributions to Bayesian machine learning algorithms. Firstly, we propose stochastic natural gradient expectation propagation (SNEP), a novel alternative to expectation propagation (EP), a popular variational inference algorithm. SNEP is a black box variational algorithm, in that it does not requi…
Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influences signal propagation in deep neural networks remains limited. By extending recent work based on mean field theory, we develop a new frame…
State Space Expectation Propagation: Efficient Inference Schemes for Temporal Gaussian Processesstat.ML
Unified framework for efficient Gaussian process inference.
problem Efficient inference in non-conjugate Gaussian process models.
method Combines expectation propagation with linearization for improved efficiency.
result Unified view of various inference schemes, including classical smoothers and EP.
Many neural networks use the tanh activation function, however when given a probability distribution as input, the problem of computing the output distribution in neural networks with tanh activation has not yet been addressed. One important example is the initialization of the echo state network in reservoir computing…
Novel methods for splitting Gaussian mixtures improve uncertainty propagation in nonlinear systems.
problem Improving accuracy and efficiency in nonlinear uncertainty propagation.
method Preserving mean and covariance, novel heuristics for selecting splitting direction informed by initial uncertainty and nonlinear function properties.
result Improved accuracy and efficiency in uncertainty propagation compared to existing techniques.
Uniform-in-time analysis for Stein Variational Gradient Descent across various metrics.
problem Understanding long-term behavior of finite-particle systems in relation to their mean-field limits.
method Developed uniform-in-time propagation-of-chaos results for continuous-time SVGD using cutoff strategies and finite-dimensional theories.
result Uniform-in-time propagation-of-chaos bounds in various metrics, including Langevin kernel Stein discrepancy, Wasserstein-1, and Wasserstein-2 distances.
Exponential family distributions are highly useful in machine learning since their calculation can be performed efficiently through natural parameters. The exponential family has recently been extended to the t-exponential family, which contains Student-t distributions as family members and thus allows us to handle noi…
We address the problem of estimating statistics of hidden units in a neural network using a method of analytic moment propagation. These statistics are useful for approximate whitening of the inputs in front of saturating non-linearities such as a sigmoid function. This is important for initialization of training and f…
Wide neural networks learn features under P, identifying weights and decomposing support.
problem Feature learning in wide neural networks under P.
method Proving mean-field limit, characterizing identifiability, sparse-dictionary decomposition, and feature-learning-error decomposition.
result The triple identifies the natural learning cell of the architecture-data pair .
Develops a Gaussian-based message-passing algorithm for noisy matrix completion.
problem Noisy matrix completion problems.
method Gaussian parameterized belief propagation algorithm.
result Advantages over traditional methods in non-Gaussian noise scenarios.
Approximate inference techniques are the cornerstone of probabilistic methods based on Gaussian process priors. Despite this, most work approximately optimizes standard divergence measures such as the Kullback-Leibler (KL) divergence, which lack the basic desiderata for the task at hand, while chiefly offering merely t…
Graph-based Bayesian SSL uses graph theory to propagate labels from a few to many unlabeled features.
problem Efficiently propagating labels from a small set of labeled data to a large set of unlabeled data.
method Probabilistic framework using graph theory and Bayesian statistics.
result Mathematical foundations for improving the accuracy and efficiency of label propagation.
Paper improves spectral learning of HMMs to avoid local optima and improve robustness.
problem Spectral learning of HMMs can get stuck in local optima and degrade due to unchecked error propagation.
method Developed a novel algorithm (PSHMM) and online learning variants to mitigate error propagation and nonstationarity.
result PSHMM provides more robust estimation and forecasting compared to SHMM and B-W algorithm.
A fast method for neural networks that provides uncertainty measures.
problem Inability of neural networks to provide reliable uncertainty measures for new situations.
method Moment propagation (MP) for analytically approximating MC dropout.
result Single-shot MC dropout approximation as fast as NNs with similar uncertainty measures.
Proposes a method to adapt DNNs to drift in data distribution.
problem Adapting to out-of-distribution data and shifting objectives.
method Bayesian Inference, Variational Density Propagation, Evidence Lower Bound (ELBO), Minimum Description Length (MDL) Principle.
result Minimizes catastrophic forgetting by approximating MDL principle.
A new neural network initialization method is proposed for faster and more accurate training.
problem Efficient initialization for training multi-layer feedforward neural networks.
method Initialization based on Stein's identity, using eigenvectors of cross-moment matrix.
result The SteinGLM method is faster and more accurate than other initialization methods.
Generative model captures complex dependence in financial data.
problem Complex dependence structure in business and financial data.
method Multivariate generative model with heterogeneous and asymmetric tail dependence.
result Novel moment learning algorithm for scalable parameter estimation.
Expectation Propagation (EP) provides a framework for approximate inference. When the model under consideration is over a latent Gaussian field, with the approximation being Gaussian, we show how these approximations can systematically be corrected. A perturbative expansion is made of the exact but intractable correcti…
New initialization schemes preserve fractional moments of weights in deep networks, improving training and test performance.
problem Heavy-tailed distribution of stochastic gradients in DNNs during training.
method Developed initialization schemes that preserve any given fractional moment of order s < 2 over layers for various activations.
result The network output admits a heavy-tailed distribution with finite moments, improving training and test performance.
Optimal stock price prediction model using recurrent neural networks with RMSprop optimizer.
problem Stock price prediction using neural networks.
method Comparison of fully connected, convolutional, and recurrent architectures; inclusion of three optimization techniques.
result Single layer recurrent neural network with RMSprop optimizer produces optimal results with validation and test MAE of 0.0150 and 0.0148 respectively.
Favour speeds up variance estimation in BNNs, making them practical for performance-critical tasks.
problem Slow inference in Bayesian Neural Networks (BNNs) hinders their adoption.
method A fast algorithm for updating a diagonal-plus-low-rank matrix approximation under various operations.
result Favour is as fast as 2-3 inference samples but matches the performance of 10-100 samples.
Deep Gaussian Processes (DGPs) were proposed as an expressive Bayesian model capable of a mathematically grounded estimation of uncertainty. The expressivity of DPGs results from not only the compositional character but the distribution propagation within the hierarchy. Recently, [1] pointed out that the hierarchical s…
A new method learns posterior and predictive distributions together, reducing computational cost.
problem Sequential two-stage Bayesian inference is computationally expensive.
method Amortized variational inference targeting posterior-predictive distribution.
result Efficient online inference with more accurate predictive distributions.
SPIDER uses deep neural networks for streaming tensor factorization.
problem Lack of effective approach for deep tensor factorization of streaming data.
method Bayesian neural networks with spike-and-slab prior, Taylor expansions, moment matching, and EPI framework.
result Effective incremental updates for latent factors and NN weights.
Minsky Financial Instability, Interscale Feedback, Percolation and Marshall-Walras Disequilibriumq-fin.GN
We study analytically and numerically Minsky instability as a combination of top-down, bottom-up and peer-to-peer positive feedback loops. The peer-to-peer interactions are represented by the links of a network formed by the connections between firms, contagion leading to avalanches and percolation phase transitions pr…
We address the problem of learning the parameters in graphical models when inference is intractable. A common strategy in this case is to replace the partition function with its Bethe approximation. We show that there exists a regime of empirical marginals where such Bethe learning will fail. By failure we mean that th…
The aim of this paper is to provide new theoretical and computational understanding on two loss regularizations employed in deep learning, known as local entropy and heat regularization. For both regularized losses we introduce variational characterizations that naturally suggest a two-step scheme for their optimizatio…
A network of independently trained Gaussian processes (StackedGP) is introduced to obtain predictions of quantities of interest with quantified uncertainties. The main applications of the StackedGP framework are to integrate different datasets through model composition, enhance predictions of quantities of interest thr…
Two wave fronts and that originated at some points of the manifold are said to be causally related if one of them passed through the origin of the other before the other appeared. We define the causality relation invariant to be the algebraic number of times the earlier born front pass…
Generalized belief propagation converges to optimal solutions on graphs with motifs.
problem Understanding belief propagation on loopy graphs.
method Study of generalized belief propagation on graphs with motifs.
result Generalized belief propagation converges to the global optimum of the Bethe free energy.
Belief propagation recovers backpropagation results.
problem Connection between backpropagation and belief propagation poorly understood.
method Converted backpropagation input to belief propagation input and showed results.
result Backpropagation is a special case of belief propagation.
Susceptibility Propagation by Using Diagonal Consistencycond-mat.stat-mech
A susceptibility propagation that is constructed by combining a belief propagation and a linear response method is used for approximate computation for Markov random fields. Herein, we formulate a new, improved susceptibility propagation by using the concept of a diagonal matching method that is based on mean-field app…
A new method calculates fractional moments using the moment-generating function.
problem Computing fractional moments from probability densities.
method Integral framework based on moment-generating function.
result Exact integral expressions for various types of moments.
FAB-COST improves cold-start recommendation accuracy with less data.
problem Cold-start problem in recommendation systems.
method Contextual bandit algorithm using Expectation Propagation and Assumed Density Filtering.
result FAB-COST outperforms Laplace approximation on real data.
Expectation Propagationstat.ML
Variational inference is a powerful concept that underlies many iterative approximation algorithms; expectation propagation, mean-field methods and belief propagations were all central themes at the school that can be perceived from this unifying framework. The lectures of Manfred Opper introduce the archetypal example…
For a GJR-GARCH specification with a generic innovation distribution we derive analytic expressions for the first four conditional moments of the forward and aggregated returns and variances. Moment for the most commonly used GARCH models are stated as special cases. We also the limits of these moments as the time hori…
This paper proposes an alternating back-propagation algorithm for learning the generator network model. The model is a non-linear generalization of factor analysis. In this model, the mapping from the continuous latent factors to the observed signal is parametrized by a convolutional neural network. The alternating bac…