PPL improves on Takacs-Fiksel estimation for Gibbs models.
problem Improving point process estimation methods.
method PPL uses cross-validation and a specific loss function to estimate parameters.
result PPL with specific loss functions and hyperparameters outperforms Takacs-Fiksel estimation in mean square error.
DMVI uses diffusion models for efficient probabilistic inference in PPLs.
problem Efficient probabilistic inference in complex probabilistic programming languages.
method DMVI employs diffusion models as variational approximations to the posterior distribution, optimizing a bound on the marginal likelihood.
result DMVI produces more accurate posterior inferences than existing methods in PPLs with similar computational cost and less manual tuning.
NP-iMCMC algorithm for nonparametric models in universal PPLs.
problem Developing inference algorithms for arbitrary nonparametric models in universal PPLs.
method Unifying involutive MCMC framework with a general procedure for state movement.
result Proves the correctness of the NP-iMCMC sampler and shows significant performance improvements.
Probabilistic programming languages (PPLs) are receiving widespread attention for performing Bayesian inference in complex generative models. However, applications to science remain limited because of the impracticability of rewriting complex scientific simulators in a PPL, the computational cost of inference, and the …
Optimizing an expensive-to-query function is a common task in science and engineering, where it is beneficial to keep the number of queries to a minimum. A popular strategy is Bayesian optimization (BO), which leverages probabilistic models for this task. Most BO today uses Gaussian processes (GPs), or a few other surr…
PClean automates Bayesian data cleaning for specific datasets.
problem Bayesian inference for diverse and complex data cleaning.
method Domain-specific probabilistic programming language with custom models and inference.
result PClean programs outperform general-purpose PPLs in accuracy and runtime.
Study uses PPLs to model and forecast COVID-19 spread and policy interventions.
problem Forecast and control the spread of COVID-19 using policy interventions.
method Compartmental models, probabilistic programming, inference, greedy algorithm.
result Optimal series of policy interventions can control the infected population.
UM trains a neural network to approximate marginal distributions in probabilistic programs.
problem High computational cost and lack of theoretical guarantees in inference methods for probabilistic programs.
method Combining samples from a probabilistic program prior with an augmentation method to train a neural network for any conditional marginal distribution.
result UM trains a single neural network to approximate any conditional marginal distribution, amortizing inference costs.
Probabilistic programming languages (PPLs) are a powerful modeling tool, able to represent any computable probability distribution. Unfortunately, probabilistic program inference is often intractable, and existing PPLs mostly rely on expensive, approximate sampling-based methods. To alleviate this problem, one could tr…
Improves sampling from complex hierarchical models using HMC and automatic marginalization.
problem Sampling from complex hierarchical models is difficult for HMC.
method Proposes automatic marginalization as part of the sampling process using HMC in a graphical model extracted from a PPL.
result Significantly improves sampling from real-world hierarchical models.
Improves Bayesian optimisation for engineering design problems with many variables.
problem Efficiently searching for global minima in high-dimensional design spaces.
method Integrates input and output data to identify a reduced latent subspace using probabilistic partial least squares.
result Significant improvements in convergence to the global minimum compared to existing methods.
We consider probabilistic programming for birth-death models of evolution and introduce a new widely-applicable inference method that combines an extension of the alive particle filter (APF) with automatic Rao-Blackwellization via delayed sampling. Birth-death models of evolution are an important family of phylogenetic…
Hamiltonian Monte Carlo (HMC) is arguably the dominant statistical inference algorithm used in most popular "first-order differentiable" Probabilistic Programming Languages (PPLs). However, the fact that HMC uses derivative information causes complications when the target distribution is non-differentiable with respect…
We develop a new Low-level, First-order Probabilistic Programming Language (LF-PPL) suited for models containing a mix of continuous, discrete, and/or piecewise-continuous variables. The key success of this language and its compilation scheme is in its ability to automatically distinguish parameters the density functio…
Type system captures CI relationships for probabilistic models.
problem Challenges in inference for models with mixed discrete and continuous parameters.
method Information flow type system for probabilistic programming.
result Well-typed programs guarantee certain CI relationships.
LIC compiles probabilistic models to generate efficient MCMC proposals.
problem Creating accurate Metropolis-Hastings proposals for Bayesian inference.
method Integrates probabilistic graphical models and neural networks in an open-source framework to optimize proposal distributions.
result LIC produces more efficient and robust MCMC proposals compared to existing methods.
NP-HMC extends HMC for nonparametric models in probabilistic programming.
problem Inference for nonparametric models in probabilistic programming.
method Introduces NP-HMC, a generalization of HMC for nonparametric models using tree representable functions.
result Empirically shows significant performance improvements over existing approaches.
Deriving conditional and marginal distributions using conjugacy relationships can be time consuming and error prone. In this paper, we propose a strategy for automating such derivations. Unlike previous systems which focus on relationships between pairs of random variables, our system (which we call Autoconj) operates …
A new framework for PPLS combines noise estimation, optimization, and calibration.
problem Probabilistic PLS models need interpretable latent factors and calibrated uncertainty.
method End-to-end pipeline combining noise estimation, constrained optimization, and prediction calibration.
result Achieves near-nominal coverage and native calibrated uncertainty across benchmarks.
This book is a graduate-level introduction to probabilistic programming. It not only provides a thorough background for anyone wishing to use a probabilistic programming system, but also introduces the techniques needed to design and build these systems. It is aimed at people who have an undergraduate-level understandi…
Compressive Transformer learns long-range sequences by compressing past memories.
problem Learning long-range sequences in language and speech models.
method Compressive Transformer compresses past memories for efficient long-range sequence learning.
result State-of-the-art performance on language and speech benchmarks.
New algorithm learns policies without uniform overlap assumption.
problem Learning optimal policies from non-uniformly collected data.
method Pessimistic Policy Learning (PPL) using lower confidence bounds.
result Efficient policy learning for adaptively collected data.
dynestyx: A library for probabilistic programming of dynamical systems
problem integrating state-space models into probabilistic programming languages
method a unified interface for specifying priors and performing inference
result principled uncertainty quantification for state and parameters
The front-end factor analysis (FEFA), an extension of principal component analysis (PPCA) tailored to be used with Gaussian mixture models (GMMs), is currently the prevalent approach to extract compact utterance-level features (i-vectors) for automatic speaker verification (ASV) systems. Little research has been conduc…
ProbDR framework interprets DR algorithms as probabilistic inference.
problem Efficiently compressing high-dimensional data into lower dimensions.
method ProbDR variational framework that treats DR as probabilistic inference.
result ProbDR unifies various DR algorithms and enables probabilistic reasoning.
Unified continuous diffusion model outperforms discrete alternatives in scalability and quality.
problem Continuous diffusion models were perceived as less scalable than discrete models.
method Reconstructed Plaid model and compared it with modern discrete DLMs, optimizing noise schedule and embeddings via likelihood.
result Unified continuous diffusion model (RePlaid) outperforms discrete models in compute efficiency and quality.
This research explores discrete diffusion models for natural language generation.
problem Challenges in applying diffusion models to discrete data, especially natural language.
method Investigates Discrete Denoising Diffusion Probabilistic Model (D3PM) and compares it with autoregressive models.
result Discrete diffusion models achieve better processing speed than autoregressive models.
This paper studies activation sparsity in large language models, finding key trends and implications.
problem Activation sparsity in large language models (LLMs) can be improved for efficiency and interpretability.
method Proposes PPL-p% sparsity, analyzes trends with training data, width-depth ratio, and parameter scale. result ReLU is more efficient for sparsity than SiLU, and deeper architectures can improve sparsity.
Improved autoencoder boosts sequence learning with less memory.
problem Training recurrent networks with high memory costs.
method Sparse Predictive Autoencoder (bRSM) with recurrent connections and boosting rule.
result Near optimal performance on stochastic sequence learning task.
ADAHESSIAN optimizes machine learning models with adaptive second-order methods.
problem Efficiently optimizing machine learning models with second-order methods.
method Dynamic Hessian estimation via adaptive estimates, incorporating fast approximations and moving averages.
result ADAHESSIAN achieves state-of-the-art performance across various tasks.