Transformers interpret as probabilistic mixtures, offering new insights.
problem Understanding Transformers from a probabilistic perspective.
method Modeling Transformers as mixtures of Gaussian models.
result Transformers can be seen as maximum posterior probability estimators.
Transformers interpreted as probabilistic Laplacian Eigenmaps steps.
problem Improving transformer performance through probabilistic interpretation.
method Probabilistic Laplacian Eigenmaps model derivation and graph diffusion step.
result Subtracting identity from attention matrix improves transformer performance.
Study improves probabilistic circuits using transformations for better predictions.
problem Predictive limitations of probabilistic circuits in robotic scenarios.
method Integrates transformations into joint probability trees, extending their capabilities.
result Achieves higher likelihoods with fewer parameters on various data sets.
Probabilistic STNs improve image classification and robustness.
problem Training and robustness issues in STNs.
method Probabilistic extension of STNs that estimates stochastic transformations.
result Improved classification performance, robustness, and model calibration.
Transforms distance-based outlier scores into interpretable probabilistic estimates.
problem Difficult interpretation of distance-based outlier scores.
method Generic transformation of scores into probabilistic estimates using distance probability distributions.
result Probabilistic transformation improves interpretability without impacting detection performance.
Probabilistic inference procedures are usually coded painstakingly from scratch, for each target model and each inference algorithm. We reduce this effort by generating inference procedures from models automatically. We make this code generation modular by decomposing inference algorithms into reusable program-to-progr…
A simple approach speeds up probabilistic programming models.
problem Efficiently implementing probabilistic programming in deep learning.
method Embedding probabilistic programming into TensorFlow with a single random variable abstraction.
result Optimal linear speedup from 1 to 256 TPUv2 chips, and 100x speedup on GPUs.
Paper improves Tm prediction of protein fragments using sparsity and probabilistic models.
problem Improving accuracy of melting temperature prediction for protein fragments.
method Promoting sparsity in pre-trained transformer models and adopting probabilistic frameworks.
result Mean absolute error of 0.23C for predicting melting temperature.
Small neural networks embed arbitrary metric spaces into Gaussian mixtures.
problem Embedding arbitrary metric spaces into a fixed space with low distortion.
method Probabilistic transformers of small depth and width.
result Embeddings with low metric distortion for various metric spaces.
Enformer and GEnformer use Transformers with stochastic learning to forecast multivariate and spatiotemporal data with uncertainty.
problem Uncertainty quantification in multivariate time series and spatiotemporal forecasting.
method Synthesizing Transformer's expressive power with stochastic learning to model conditional distributions directly.
result Enformer and GEnformer yield calibrated probabilistic forecasts and outperform state-of-the-art baselines.
Transformer model for probabilistic dynamical systems.
problem Modeling high-dimensional dynamical systems from noisy observations.
method Parallel between dynamical systems and language modeling; transformer-based model with geometrical properties; iterative training algorithm.
result Fine-grid approximation of conditional probabilities for high-dimensional systems.
Transforms ensemble predictions to maintain interpretability.
problem Loss of interpretability in deep ensembles.
method Proposes transformation ensembles that aggregate predictions while preserving interpretability.
result Transformation ensembles yield better predictions than individual models and maintain interpretability.
Transformer with denoising diffusion improves probabilistic density estimation.
problem Estimating non-Gaussian and multimodal probability distributions for regression problems.
method Training a denoising diffusion head on top of a Transformer model.
result The model provides reasonable probability density estimation for high-dimensional inputs.
Paper uses Transformers to predict intraday volume ratio with high accuracy.
problem Accurate prediction of intraday volume ratio for VWAP strategies.
method Transformer architecture with log-normal transformation and external features.
result Probabilistic forecasting captures mean and standard deviation of volume ratios.
Unified framework for supervised classification with diverse training data.
problem Handling different types of training data for supervised classification.
method Generalized robust risk minimization (GRRM) with probabilistic transformations.
result GRRM can handle various training data types and new supervision schemes.
SPPL simplifies probabilistic programming for exact inference.
problem Efficient exact inference in probabilistic models.
method SPPL translates probabilistic programs into sum-product expressions, leveraging new techniques for scalability.
result SPPL achieves up to 3500x speedups in exact inference.
Deep learning uses layers of transformations to predict structured data with uncertainty.
problem Predicting structured high-dimensional data efficiently and with uncertainty.
method Applying layers of semi-affine input transformations to find features for probabilistic statistical methods.
result Achieves scalable prediction rules with uncertainty quantification and feature selection.
Normalizing flows simplify complex distributions through bijective transformations.
problem Defining expressive probability distributions efficiently.
method Bijective transformations on a base distribution.
result Unified perspective on normalizing flows for modeling and inference.
ACE models allow flexible conditioning and prediction of latent variables.
problem Lack of flexibility in conditioning and prediction of latent variables in probabilistic models.
method Introduces Amortized Conditioning Engine (ACE) that explicitly represents latent variables and allows runtime conditioning and prediction.
result ACE models outperform existing methods in diverse tasks like image completion, classification, Bayesian optimization, and simulation-based inference.
Improves sequential recommendation with relation-aware self-attention.
problem Improving accuracy in sequential recommendation.
method Integrates Transformer's self-attention mechanism with a probabilistic model of recommendation context.
result Significant improvements over recent baseline models.
NumPyro accelerates probabilistic programming with JAX transformations.
problem Efficiently handling probabilistic models with hardware acceleration.
method Composable effects and program transformations in JAX.
result Iterative NUTS formulation JIT compiled for speed.
Transforms input design for probabilistic models into optimal control of a Hamiltonian system.
problem Designing inputs for probabilistic models with intractable posterior distributions.
method Representing posterior as Hamiltonian system trajectories, solving optimal control problem.
result Parameter posterior concentrates around true parameter values.
Transformers model contextual relations using probabilistic measures, revealing their expressive power.
problem Lack of clear understanding of Transformer's ability to model contextual relations.
method Introduced a measure-theoretic framework connecting softmax attention and entropy-regularized optimal transport.
result Transformer architectures can approximate arbitrary contextual relations, and the choice of normalization affects how these relations are represented.
Bayesian Transformer improves probabilistic load forecasting with calibrated uncertainty estimates.
problem Overconfident point predictions from deep learning models fail under extreme weather distributional shifts.
method Integrates three uncertainty mechanisms: MC Dropout, variational layers, and stochastic attention.
result Achieves state-of-the-art performance with CRPS of 0.0289 and 90% PICP across various horizons.
Automates reparameterization in probabilistic programs for better inference.
problem Performance of inference algorithms is affected by parameterization, requiring manual and non-intuitive transformations.
method Automates non-centring and related reparameterizations using interleaved sampling and variational formulation.
result New inference algorithms yield more efficient samplers than fixed parameterizations across various models.
The paper models musical motif transformations in Beethoven's works.
problem Understanding how motifs transform in symbolic music.
method Developed a probabilistic framework using Conditional Random Fields.
result Identified patterns of motif transformations and their co-occurrences.
The TensorFlow Distributions library implements a vision of probability theory adapted to the modern deep-learning paradigm of end-to-end differentiable computation. Building on two basic abstractions, it offers flexible building blocks for probabilistic computation. Distributions provide fast, numerically stable metho…
Paper integrates real data into probabilistic models using Fourier transform.
problem Learning from constrained data sets in high dimensions.
method Functional approach based on weak formulation of Fourier transform of probability measures.
result Estimation of posterior probability measures for QoI and QoI with control parameter.
Paper optimizes score transformation for fair binary classification.
problem Ensuring fairness in binary classification with predicted scores.
method Formulates and solves a convex optimization problem for transforming scores to meet fairness constraints.
result Derives a closed-form expression for optimal transformed scores and provides guarantees for finite sample settings.
A simple framework Probabilistic Multi-view Graph Embedding (PMvGE) is proposed for multi-view feature learning with many-to-many associations so that it generalizes various existing multi-view methods. PMvGE is a probabilistic model for predicting new associations via graph embedding of the nodes of data vectors with …
We solve Bayesian PCA's rotational symmetry issue by rotation-invariant parameterization.
problem Bayesian PCA's rotational symmetry complicates inference and interpretation.
method Rotation-invariant Householder parameterization using random matrix theory.
result Efficient rotation-invariant probabilistic PCA implementation.
This work combines recurrent models with diffusion for probabilistic time series forecasting.
problem Scalability and capturing high-dimensional distributions and cross-feature dependencies in time series forecasting.
method Combines recurrent neural networks' efficiency with diffusion models' probabilistic modeling, using stochastic interpolants and conditional generation.
result Offers scalable probabilistic time series forecasting methods.
Normalizing flows are shown to be equivalent to Bayesian networks, revealing new insights.
problem Understanding the limitations and capabilities of normalizing flows.
method Revisiting normalizing flows as probabilistic graphical models and analyzing their structure.
result Normalizing flows can be reduced to Bayesian networks, revealing new insights into their structure and capabilities.
EMFs combine deep learning and probabilistic models for better density estimation.
problem Combining domain-specific knowledge with general-purpose deep learning.
method Alternating transformations with structured layers that embed domain-specific inductive biases.
result EMFs induce desirable properties like multimodality and hierarchical coupling.
This thesis improves deep sequence models by integrating probabilistic methods for uncertainty quantification.
problem Lack of uncertainty awareness in deep sequence models limits their deployment.
method Develops approximate Bayesian inference methods for Transformers and HiPPOs, leveraging inductive biases.
result Improves predictive and generative performance of deep sequence models by incorporating probabilistic structures.
Improved probabilistic forecasts using behavioral transformations.
problem Improving accuracy and consistency of probabilistic asset price forecasts.
method Behavioral transformation of fundamental expectations to disentangle sentiment-induced biases.
result Substantial forecast gains across various models and risk-preferences.
Efficiently combines autoregressive and set-based models for joint distributions.
problem Joint distributions over multiple predictions from set-based models.
method Causal autoregressive buffer that caches context and captures dependencies.
result Up to 20x faster joint sampling and density evaluation, up to 7x lower memory usage.
New approach improves linear-time attention for language models.
problem Challenges of quadratic attention in long-sequence modelling, especially for discrete data.
method Reinterpreting linear attention through latent probabilistic graphical models, introducing asymmetric structure and recurrent parameterisation.
result Our model achieves competitive performance and outperforms existing linear attention variants on language modelling benchmarks.
The paper improves ODE solvers by integrating diverse information types.
problem Improving accuracy and physical meaningfulness of ODE solutions.
method Leveraging probabilistic solvers to include second-order information and physical conservation laws.
result Solutions become more accurate and physically meaningful with additional information.
Model approximates continuous functions in 1-Wasserstein space.
problem Approximating continuous functions in 1-Wasserstein space.
method Probabilistic Transformer (PT) model with three phases: feature map, deep neural network, and probabilistic extension of attention mechanism.
result Can approximate any continuous function from R^d to P1(R^D) uniformly on compact sets.
We present the first general purpose framework for marginal maximum a posteriori estimation of probabilistic program variables. By using a series of code transformations, the evidence of any probabilistic program, and therefore of any graphical model, can be optimized with respect to an arbitrary subset of its sampled …
A framework uses free probability to analyze Transformer models.
problem Understanding the dynamics and complexity of Transformer-based language models.
method Formal operator-theoretic analysis using free probability theory.
result Entropy-based generalization bounds derived under freeness assumptions.
We introduce a method for using deep neural networks to amortize the cost of inference in models from the family induced by universal probabilistic programming languages, establishing a framework that combines the strengths of probabilistic programming and deep learning methods. We call what we do "compilation of infer…
Hybrid model combines deep features with invertible transformations for accurate predictions and feature modeling.
problem Accurate prediction and feature modeling using deep and invertible transformations.
method Neural hybrid model with a linear model on features from a deep invertible transformation.
result Hybrid model achieves similar accuracy to purely predictive models while maintaining generative capabilities.
The paper introduces models to learn generalized transformation equivariant representations.
problem Capturing intrinsic visual structures equivariant to various transformations.
method Deterministic and probabilistic AutoEncoding Transformations (AET and AVT) models trained to learn visual representations from generic groups of transformations.
result Generalized TERs (GTERs) that are equivariant to transformations in a more general fashion.
Kernel estimator optimally recovers function from noisy exponential Radon transform.
problem Inverting noisy exponential Radon transform of a function.
method Proposed a kernel estimator to estimate the true function.
result The estimator converges to the true function at minimax optimal rate.
Mixtures of Gaussians, factor analyzers (probabilistic PCA) and hidden Markov models are staples of static and dynamic data modeling and image and video modeling in particular. We show how topographic transformations in the input, such as translation and shearing in images, can be accounted for in these models by inclu…
Rigorous uncertainty quantification of probabilistic AI weather forecasts with conformal prediction
problem Calibrated uncertainty quantification in probabilistic weather forecasts
method Conformal prediction
result Calibrated uncertainty at no expense to other probabilistic metrics