A new method for target propagation using iterative approximations converges fast and is more biologically plausible.
problem Improving target propagation methods for neural networks.
method Iterative approximate inverses and local auto-encoders.
result The method converges exponentially fast under certain conditions.
The paper analyzes the complexity of sparse label propagation on networks.
problem Computational complexity of sparse label propagation on network data.
method Characterization of iterations for achieving a prescribed accuracy using a first-order oracle model.
result An upper bound on iterations required for accuracy, showing sharpness for chain structures.
Variational inference is a powerful concept that underlies many iterative approximation algorithms; expectation propagation, mean-field methods and belief propagations were all central themes at the school that can be perceived from this unifying framework. The lectures of Manfred Opper introduce the archetypal example…
IIC decouples causal identification into two phases, significantly reducing the HTC gap in linear SEMs.
problem Determining causal effect coefficients in linear SEMs with latent confounders using the Half-Trek Criterion (HTC) leaves a gap of inconclusive causal effects.
method Iterative Identification Closure (IIC) framework that decouples causal identification into two phases: a seed function S_0 and Reduced HTC propagation.
result IIC strictly subsumes both HTC and ancestor decomposition, reducing the HTC gap by over 80% with combined seeds.
Proposes HOPF framework for CC using higher-order propagation.
problem Collective Classification struggles with node information morphing across multiple hops.
method Iterative inference mechanism with differentiable kernels for multi-hop neighborhood information.
result NIP models preserve node information and provide more robust performance.
BBPL uses block updates to learn Markov random fields without full inference.
problem Training Markov random fields requires inference over all variables, scaling with model size.
method Block-coordinate updates of approximate marginals to compute approximate gradients.
result BBPL converges to the same solution as full inference, despite approximations.
This paper analyzes how errors accumulate in PCA's deflation method.
problem Error accumulation in PCA's deflation method.
method Mathematical analysis of inexact Hotelling's deflation method in two scenarios.
result Characterization of error propagation in PCA's deflation method.
Belief propagation quickly converges to global optima for ferromagnetic Ising models.
problem Understanding convergence of belief propagation on graphs with cycles.
method Natural initialization and analysis of Ising models on arbitrary graphs.
result Belief propagation converges quickly to the global optimum of the Bethe free energy for ferromagnetic Ising models.
We propose an original particle-based implementation of the Loopy Belief Propagation (LPB) algorithm for pairwise Markov Random Fields (MRF) on a continuous state space. The algorithm constructs adaptively efficient proposal distributions approximating the local beliefs at each note of the MRF. This is achieved by cons…
This paper makes complex neural graphs nearly convex through iterative decomposition and scale mechanism.
problem Non-convexity in complex neural architectures limits their performance in convex optimization.
method Decompose neural graph into operators, algorithms, and functions; iteratively propagate along edges; introduce scale mechanism to transform non-convex properties.
result Proves neural graph is nearly convex in each variable when others are fixed, validating the scale mechanism.
This paper proposes an alternating back-propagation algorithm for learning the generator network model. The model is a non-linear generalization of factor analysis. In this model, the mapping from the continuous latent factors to the observed signal is parametrized by a convolutional neural network. The alternating bac…
Two EP frameworks ensure integrable beliefs in Bayesian estimation problems.
problem Non-integrable beliefs in EP can lead to infeasible solutions.
method Proposes two EP frameworks to keep messages non-integrable.
result Ensures integrable beliefs in EP, even with non-integrable messages.
A decentralized MARL algorithm for value propagation in non-linear settings.
problem Decentralized coordination in multi-agent systems with different local rewards.
method Decentralized optimization and value propagation algorithm.
result Non-asymptotic convergence rate of 1/T in nonlinear function approximation.
Improved error correction using neural networks and belief propagation.
problem Inference in factor graphs with loops or poor approximations.
method Hybrid model combining FG-GNN and belief propagation.
result Hybrid model outperforms belief propagation in error correction tasks.
A number of problems in statistical physics and computer science can be expressed as the computation of marginal probabilities over a Markov random field. Belief propagation, an iterative message-passing algorithm, computes exactly such marginals when the underlying graph is a tree. But it has gained its popularity as …
Deep learning method for 3D cardiac segmentation with spatial propagation.
problem Cardiac segmentation from MRI stacks with spatial consistency.
method Iterative deep learning with U-net for spatial propagation, training on UK Biobank.
result Comparable or better results than state-of-the-art, enhanced spatial consistency.
Truncated back-propagation improves hyperparameter tuning and meta learning efficiency.
problem Computational challenges in evaluating exact gradients for high-dimensional bilevel optimization problems.
method Use truncated back-propagation to approximate gradients for the lower-level problem.
result Optimization with few-step back-propagation approximations often performs comparably to exact gradients, but with less memory and computation.
Iterative BP-CNN improves channel decoding under correlated noise.
problem Channel decoding under correlated noise.
method Concatenates CNN with BP decoder, iteratively improving SNR.
result Iterative BP-CNN achieves better BER with lower complexity.
Two new algorithms improve Q* approximation in batch RL with linear error propagation.
problem Improving Q* approximation in batch reinforcement learning.
method Two novel algorithms that estimate Bellman error directly, without quadratic dependence.
result Linear-in-horizon error propagation for batch RL algorithms.
We present a joint message passing approach that combines belief propagation and the mean field approximation. Our analysis is based on the region-based free energy approximation method proposed by Yedidia et al. We show that the message passing fixed-point equations obtained with this combination correspond to station…
The paper introduces various gradient descent algorithms for training deep learning models.
problem Training deep neural networks is challenging due to their complexity.
method Gradient descent and its variants are discussed for optimizing deep learning models.
result Gradient descent and its variants improve the training performance of deep learning models.
BPNNs learn to solve combinatorial problems faster and more accurately.
problem Generalizing belief propagation for efficient problem solving.
method BPNNs are parameterized operators that operate on factor graphs, generalizing BP. BPNN-D is a learned iterative operator that provably maintains BP's properties.
result BPNN-D converges 1.7x faster on Ising models and provides tighter bounds.
A neural network model minimizes region-based free energy for faster inference in MRFs.
problem Efficient inference in complex Markov random fields (MRFs).
method Region-based Energy Neural Network (RENN) that directly minimizes region-based free energy.
result RENN outperforms other methods in marginal distribution estimation, partition function estimation, and MRF learning.
DP-SEP privatizes EP by refining a single factor per data point.
problem Private inference of complex models with limited memory.
method Stochastic Expectation Propagation with differential privacy.
result DP-SEP provides better posterior estimates with guaranteed privacy.
Gaussian BP algorithm converges exponentially under walk summability for cyclic graphs.
problem Convergence rate of Gaussian BP for cyclic graphs.
method Extending known results on walk summability, proving exponential convergence rate.
result Gaussian BP converges exponentially under walk summability for cyclic graphs.
Proposes a new method for GNNs that avoids iterative node state convergence.
problem Iterative computation of node states in GNNs is inefficient and requires many epochs.
method Constrained optimization in the Lagrangian framework to learn transition function and node states simultaneously.
result The proposed method compares favorably with existing models on various benchmarks.
Contrastive regularization improves semi-supervised learning by better propagating confident pseudo-labels.
problem Consistency regularization's limitation in high performance and efficiency.
method Proposes contrastive regularization to update model features, pushing confident labels into unlabeled samples.
result Improves semi-supervised learning tasks with fewer training iterations and robust performance.
Optimal trading strategy derived for nonlinear price impact models.
problem Optimal trading with nonlinear price impact induced by alpha signals.
method Variational approach, nonlinear Fredholm equation, iterative scheme.
result Existence and uniqueness of optimal trading strategy under monotonicity condition.
CEP improves inference efficiency and accuracy by conditional moment matching.
problem Intractable moment matching in EP.
method Conditional expectation propagation (CEP) performs conditional moment matching and expectation.
result CEP achieves better inference quality and efficiency.
Community detection is considered for a stochastic block model graph of n vertices, with K vertices in the planted community, edge probability p for pairs of vertices both in the community, and edge probability q for other pairs of vertices. The main focus of the paper is on weak recovery of the community based on the …
Uniform-in-time analysis for Stein Variational Gradient Descent across various metrics.
problem Understanding long-term behavior of finite-particle systems in relation to their mean-field limits.
method Developed uniform-in-time propagation-of-chaos results for continuous-time SVGD using cutoff strategies and finite-dimensional theories.
result Uniform-in-time propagation-of-chaos bounds in various metrics, including Langevin kernel Stein discrepancy, Wasserstein-1, and Wasserstein-2 distances.
EP method speeds up Bayesian probit regression in high dimensions.
problem Computational challenges in high-dimensional Bayesian probit regression.
method Adapting EP approximation to multivariate Gaussian prior and skew-normal distribution.
result EP routine is computationally feasible in high-dimensional settings.
We present a new deep meta reinforcement learner, which we call Deep Episodic Value Iteration (DEVI). DEVI uses a deep neural network to learn a similarity metric for a non-parametric model-based reinforcement learning algorithm. Our model is trained end-to-end via back-propagation. Despite being trained using the mode…
New algorithm for solving minimax problems over distributions converges to Nash equilibrium.
problem Solving minimax problems over probability distributions.
method Symmetric Mean-field Langevin Dynamics (MFL-AG and MFL-ABR) with weighted averaging and best response dynamics.
result Converges to mixed Nash equilibrium with average-iterate and last-iterate convergence.
New algorithm for Gaussian process classification using posterior linearisation.
problem Improving Gaussian process classification performance.
method Posterior linearisation to approximate posterior density iteratively, accounting for linearisation error.
result PL has better performance than EP in experimental data.
Paper uses EXIT analysis for community detection with side information.
problem Community detection in the presence of side information.
method Imported EXIT method from iterative decoding of error control codes.
result Predicts asymptotic phase transition and residual errors for community detection.
Neural networks learn higher-order derivatives for physics problems.
problem Lack of higher-order derivatives in neural networks for theoretical physics.
method Graph-theoretical approach to assign diagrams to partial derivatives, iterative NN perturbation theory.
result NNs can learn higher-order derivatives, improving machine-learned approximations.
Deep vanilla transformers trained without shortcuts achieve similar performance to standard models.
problem Training deep vanilla transformers without shortcuts and normalizations.
method Parameter initializations, bias matrices, and location-dependent rescaling.
result Deep vanilla transformers can train at similar speeds and performance to standard models.
New algorithm for efficient inference over tree-structured graphs.
problem Inference over probabilistic graphical models with aggregate data.
method Optimal transport theory, Sinkhorn/iterative scaling algorithm, belief propagation.
result Global convergence and polynomial computational complexity.
EarlyStopping package helps prevent overfitting in iterative learning procedures.
problem Preventing overfitting in iterative learning procedures.
method Early stopping implemented in a Python package.
result Demonstrates how to use the EarlyStopping package to explore implicit regularization.
New insights into belief propagation and Bethe approximation for factor graphs.
problem Understanding the correctness and efficiency of belief propagation and its relation to partition functions.
method Viewing factor graphs through the lens of polynomials and reformulating Bethe approximation as a polynomial optimization problem.
result For bipartite normal factor graphs, the Bethe approximation is a lower bound to the partition function under certain analytic conditions.
Forest Fire Clustering discovers cell types from single-cell data.
problem Discovering cell types from large-scale single-cell sequencing data.
method Iterative label propagation and parallelized Monte Carlo simulation.
result Forest Fire Clustering outperforms state-of-the-art methods on diverse benchmarks.
RippleNet uses a knowledge graph to improve recommendation by propagating user preferences.
problem Collaborative filtering sparsity and cold start problem.
method End-to-end framework that propagates user preferences over the knowledge graph.
result Ripple Network achieves substantial gains in recommendation performance.
We design iterative receiver schemes for a generic wireless communication system by treating channel estimation and information decoding as an inference problem in graphical models. We introduce a recently proposed inference framework that combines belief propagation (BP) and the mean field (MF) approximation and inclu…
Improved pruning method using iterative sensitivity ranking before training.
problem Improper sensitivity propagation in existing pruning methods.
method Iterative application of SNIP criterion before training.
result State-of-the-art sparsity-performance trade-offs achieved.
We consider a set of learning agents in a collaborative peer-to-peer network, where each agent learns a personalized model according to its own learning objective. The question addressed in this paper is: how can agents improve upon their locally trained model by communicating with other agents that have similar object…
New deep network derived from rate reduction principles, explaining features and efficiency.
problem Understanding and optimizing deep learning architectures.
method Gradient ascent scheme for rate reduction leading to multi-layer deep network.
result Explicitly constructed multi-layer network with precise optimization and interpretation.
Improved PoC for MFLD reduces approximation error and provides model ensemble guarantees.
problem Quantifying optimization complexity in mean-field Langevin dynamics.
method Refined defective log-Sobolev inequality for neural network training.
result Improved PoC result with reduced approximation error and theoretical model ensemble guarantees.