Transformers interpret as probabilistic mixtures, offering new insights.
problem Understanding Transformers from a probabilistic perspective.
method Modeling Transformers as mixtures of Gaussian models.
result Transformers can be seen as maximum posterior probability estimators.
Transformers interpreted as probabilistic Laplacian Eigenmaps steps.
problem Improving transformer performance through probabilistic interpretation.
method Probabilistic Laplacian Eigenmaps model derivation and graph diffusion step.
result Subtracting identity from attention matrix improves transformer performance.
Study improves probabilistic circuits using transformations for better predictions.
problem Predictive limitations of probabilistic circuits in robotic scenarios.
method Integrates transformations into joint probability trees, extending their capabilities.
result Achieves higher likelihoods with fewer parameters on various data sets.
Probabilistic STNs improve image classification and robustness.
problem Training and robustness issues in STNs.
method Probabilistic extension of STNs that estimates stochastic transformations.
result Improved classification performance, robustness, and model calibration.
Probabilistic inference procedures are usually coded painstakingly from scratch, for each target model and each inference algorithm. We reduce this effort by generating inference procedures from models automatically. We make this code generation modular by decomposing inference algorithms into reusable program-to-progr…
Transforms distance-based outlier scores into interpretable probabilistic estimates.
problem Difficult interpretation of distance-based outlier scores.
method Generic transformation of scores into probabilistic estimates using distance probability distributions.
result Probabilistic transformation improves interpretability without impacting detection performance.
Paper improves Tm prediction of protein fragments using sparsity and probabilistic models.
problem Improving accuracy of melting temperature prediction for protein fragments.
method Promoting sparsity in pre-trained transformer models and adopting probabilistic frameworks.
result Mean absolute error of 0.23C for predicting melting temperature.
Small neural networks embed arbitrary metric spaces into Gaussian mixtures.
problem Embedding arbitrary metric spaces into a fixed space with low distortion.
method Probabilistic transformers of small depth and width.
result Embeddings with low metric distortion for various metric spaces.
Enformer and GEnformer use Transformers with stochastic learning to forecast multivariate and spatiotemporal data with uncertainty.
problem Uncertainty quantification in multivariate time series and spatiotemporal forecasting.
method Synthesizing Transformer's expressive power with stochastic learning to model conditional distributions directly.
result Enformer and GEnformer yield calibrated probabilistic forecasts and outperform state-of-the-art baselines.
Transformer model for probabilistic dynamical systems.
problem Modeling high-dimensional dynamical systems from noisy observations.
method Parallel between dynamical systems and language modeling; transformer-based model with geometrical properties; iterative training algorithm.
result Fine-grid approximation of conditional probabilities for high-dimensional systems.
Transforms ensemble predictions to maintain interpretability.
problem Loss of interpretability in deep ensembles.
method Proposes transformation ensembles that aggregate predictions while preserving interpretability.
result Transformation ensembles yield better predictions than individual models and maintain interpretability.
Transformer with denoising diffusion improves probabilistic density estimation.
problem Estimating non-Gaussian and multimodal probability distributions for regression problems.
method Training a denoising diffusion head on top of a Transformer model.
result The model provides reasonable probability density estimation for high-dimensional inputs.
Paper uses Transformers to predict intraday volume ratio with high accuracy.
problem Accurate prediction of intraday volume ratio for VWAP strategies.
method Transformer architecture with log-normal transformation and external features.
result Probabilistic forecasting captures mean and standard deviation of volume ratios.
Deep learning uses layers of transformations to predict structured data with uncertainty.
problem Predicting structured high-dimensional data efficiently and with uncertainty.
method Applying layers of semi-affine input transformations to find features for probabilistic statistical methods.
result Achieves scalable prediction rules with uncertainty quantification and feature selection.
SPPL simplifies probabilistic programming for exact inference.
problem Efficient exact inference in probabilistic models.
method SPPL translates probabilistic programs into sum-product expressions, leveraging new techniques for scalability.
result SPPL achieves up to 3500x speedups in exact inference.
ACE models allow flexible conditioning and prediction of latent variables.
problem Lack of flexibility in conditioning and prediction of latent variables in probabilistic models.
method Introduces Amortized Conditioning Engine (ACE) that explicitly represents latent variables and allows runtime conditioning and prediction.
result ACE models outperform existing methods in diverse tasks like image completion, classification, Bayesian optimization, and simulation-based inference.
We describe a simple, low-level approach for embedding probabilistic programming in a deep learning ecosystem. In particular, we distill probabilistic programming down to a single abstraction---the random variable. Our lightweight implementation in TensorFlow enables numerous applications: a model-parallel variational …
Transformers model contextual relations using probabilistic measures, revealing their expressive power.
problem Lack of clear understanding of Transformer's ability to model contextual relations.
method Introduced a measure-theoretic framework connecting softmax attention and entropy-regularized optimal transport.
result Transformer architectures can approximate arbitrary contextual relations, and the choice of normalization affects how these relations are represented.
Bayesian Transformer improves probabilistic load forecasting with calibrated uncertainty estimates.
problem Overconfident point predictions from deep learning models fail under extreme weather distributional shifts.
method Integrates three uncertainty mechanisms: MC Dropout, variational layers, and stochastic attention.
result Achieves state-of-the-art performance with CRPS of 0.0289 and 90% PICP across various horizons.
The paper models musical motif transformations in Beethoven's works.
problem Understanding how motifs transform in symbolic music.
method Developed a probabilistic framework using Conditional Random Fields.
result Identified patterns of motif transformations and their co-occurrences.
The TensorFlow Distributions library implements a vision of probability theory adapted to the modern deep-learning paradigm of end-to-end differentiable computation. Building on two basic abstractions, it offers flexible building blocks for probabilistic computation. Distributions provide fast, numerically stable metho…
Paper integrates real data into probabilistic models using Fourier transform.
problem Learning from constrained data sets in high dimensions.
method Functional approach based on weak formulation of Fourier transform of probability measures.
result Estimation of posterior probability measures for QoI and QoI with control parameter.
This paper considers fair probabilistic binary classification where the outputs of primary interest are predicted probabilities, commonly referred to as scores. We formulate the problem of transforming scores to satisfy fairness constraints that are linear in conditional means of scores while minimizing a cross-entropy…
A simple framework Probabilistic Multi-view Graph Embedding (PMvGE) is proposed for multi-view feature learning with many-to-many associations so that it generalizes various existing multi-view methods. PMvGE is a probabilistic model for predicting new associations via graph embedding of the nodes of data vectors with …
This work combines recurrent models with diffusion for probabilistic time series forecasting.
problem Scalability and capturing high-dimensional distributions and cross-feature dependencies in time series forecasting.
method Combines recurrent neural networks' efficiency with diffusion models' probabilistic modeling, using stochastic interpolants and conditional generation.
result Offers scalable probabilistic time series forecasting methods.
Normalizing flows are shown to be equivalent to Bayesian networks, revealing new insights.
problem Understanding the limitations and capabilities of normalizing flows.
method Revisiting normalizing flows as probabilistic graphical models and analyzing their structure.
result Normalizing flows can be reduced to Bayesian networks, revealing new insights into their structure and capabilities.
EMFs combine deep learning and probabilistic models for better density estimation.
problem Combining domain-specific knowledge with general-purpose deep learning.
method Alternating transformations with structured layers that embed domain-specific inductive biases.
result EMFs induce desirable properties like multimodality and hierarchical coupling.
Different types of training data have led to numerous schemes for supervised classification. Current learning techniques are tailored to one specific scheme and cannot handle general ensembles of training data. This paper presents a unifying framework for supervised classification with general ensembles of training dat…
Recent studies identified that sequential Recommendation is improved by the attention mechanism. By following this development, we propose Relation-Aware Kernelized Self-Attention (RKSA) adopting a self-attention mechanism of the Transformer with augmentation of a probabilistic model. The original self-attention of Tra…
This thesis improves deep sequence models by integrating probabilistic methods for uncertainty quantification.
problem Lack of uncertainty awareness in deep sequence models limits their deployment.
method Develops approximate Bayesian inference methods for Transformers and HiPPOs, leveraging inductive biases.
result Improves predictive and generative performance of deep sequence models by incorporating probabilistic structures.
Improved probabilistic forecasts using behavioral transformations.
problem Improving accuracy and consistency of probabilistic asset price forecasts.
method Behavioral transformation of fundamental expectations to disentangle sentiment-induced biases.
result Substantial forecast gains across various models and risk-preferences.
We consider probabilistic PCA and related factor models from a Bayesian perspective. These models are in general not identifiable as the likelihood has a rotational symmetry. This gives rise to complicated posterior distributions with continuous subspaces of equal density and thus hinders efficiency of inference as wel…
Efficiently combines autoregressive and set-based models for joint distributions.
problem Joint distributions over multiple predictions from set-based models.
method Causal autoregressive buffer that caches context and captures dependencies.
result Up to 20x faster joint sampling and density evaluation, up to 7x lower memory usage.
New approach improves linear-time attention for language models.
problem Challenges of quadratic attention in long-sequence modelling, especially for discrete data.
method Reinterpreting linear attention through latent probabilistic graphical models, introducing asymmetric structure and recurrent parameterisation.
result Our model achieves competitive performance and outperforms existing linear attention variants on language modelling benchmarks.
The paper improves ODE solvers by integrating diverse information types.
problem Improving accuracy and physical meaningfulness of ODE solutions.
method Leveraging probabilistic solvers to include second-order information and physical conservation laws.
result Solutions become more accurate and physically meaningful with additional information.
Model approximates continuous functions in 1-Wasserstein space.
problem Approximating continuous functions in 1-Wasserstein space.
method Probabilistic Transformer (PT) model with three phases: feature map, deep neural network, and probabilistic extension of attention mechanism.
result Can approximate any continuous function from R^d to P1(R^D) uniformly on compact sets.
We present the first general purpose framework for marginal maximum a posteriori estimation of probabilistic program variables. By using a series of code transformations, the evidence of any probabilistic program, and therefore of any graphical model, can be optimized with respect to an arbitrary subset of its sampled …
Transformation Equivariant Representations (TERs) aim to capture the intrinsic visual structures that equivary to various transformations by expanding the notion of {\em translation} equivariance underlying the success of Convolutional Neural Networks (CNNs). For this purpose, we present both deterministic AutoEncoding…
Probabilistic programming has emerged as a powerful paradigm in statistics, applied science, and machine learning: by decoupling modelling from inference, it promises to allow modellers to directly reason about the processes generating data. However, the performance of inference algorithms can be dramatically affected …
A framework uses free probability to analyze Transformer models.
problem Understanding the dynamics and complexity of Transformer-based language models.
method Formal operator-theoretic analysis using free probability theory.
result Entropy-based generalization bounds derived under freeness assumptions.
We introduce a method for using deep neural networks to amortize the cost of inference in models from the family induced by universal probabilistic programming languages, establishing a framework that combines the strengths of probabilistic programming and deep learning methods. We call what we do "compilation of infer…
Kernel estimator optimally recovers function from noisy exponential Radon transform.
problem Inverting noisy exponential Radon transform of a function.
method Proposed a kernel estimator to estimate the true function.
result The estimator converges to the true function at minimax optimal rate.
Mixtures of Gaussians, factor analyzers (probabilistic PCA) and hidden Markov models are staples of static and dynamic data modeling and image and video modeling in particular. We show how topographic transformations in the input, such as translation and shearing in images, can be accounted for in these models by inclu…
Rigorous uncertainty quantification of probabilistic AI weather forecasts with conformal prediction
problem Calibrated uncertainty quantification in probabilistic weather forecasts
method Conformal prediction
result Calibrated uncertainty at no expense to other probabilistic metrics
Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expa…
Probabilistic models often have parameters that can be translated, scaled, permuted, or otherwise transformed without changing the model. These symmetries can lead to strong correlation and multimodality in the posterior distribution over the model's parameters, which can pose challenges both for performing inference a…
Many practical techniques for probabilistic inference require a sequence of distributions that interpolate between a tractable distribution and an intractable distribution of interest. Usually, the sequences used are simple, e.g., based on geometric averages between distributions. When models are expressed as probabili…
A new algorithm converts staged trees into Chain Event Graphs.
problem Creating explicit representations of structural zeros and conditional independences.
method Iterative backward algorithm transforming a staged tree into a CEG.
result No information is lost in the transformation from staged tree to CEG.