Transformers interpret as probabilistic mixtures, offering new insights.
problem Understanding Transformers from a probabilistic perspective.
method Modeling Transformers as mixtures of Gaussian models.
result Transformers can be seen as maximum posterior probability estimators.
Improved text classification performance through conformal transformations of kernels.
problem Text document categorization in high-dimensional spaces.
method Introduced new Gaussian Cosine kernel and two conformal transformations.
result Conformal transformations significantly improve kernel performance, especially for sub-optimal kernels.
Transformers converge linearly to optimal models for Gaussian mixtures classification.
problem Theoretical understanding of transformers' in-context classification.
method Gradient descent training of a single-layer transformer for Gaussian mixtures classification.
result Transformers converge linearly to globally optimal models for Gaussian mixtures classification.
Deep Transformed Gaussian Processes extend TGPs with variational inference for scalable multi-layer modeling.
problem Flexible modeling of complex data distributions.
method DTGPs are a multi-layer model of TGPs using variational inference for scalability.
result DTGPs achieve good scalability and performance in multiple regression datasets.
Study on convergence of transformed metric spaces as dimensions grow.
problem Conditions for convergence of transformed metric spaces.
method Clarifying conditions for convergence of transformed spaces from original sequence and vice versa.
result Spheres and projective spaces converge to Gaussian space and its quotient as dimensions increase.
Computation of moments of transformed random variables is a problem appearing in many engineering applications. The current methods for moment transformation are mostly based on the classical quadrature rules which cannot account for the approximation errors. Our aim is to design a method for moment transformation for …
SGPA calibrates transformer uncertainty for safety-critical tasks.
problem Uncertainty estimation in transformer models for safety-critical domains.
method Bayesian inference in transformer's output space using sparse Gaussian processes.
result SGPA-based Transformers improve in-distribution calibration and out-of-distribution robustness.
The paper shows how to infer conditional independence from non-Gaussian data.
problem Inferring conditional independence from non-Gaussian distributions.
method Developed a method to recover conditional independence structure from the precision matrix of generalized nonparanormal data.
result The conditional independence structure can be inferred from the precision matrix of generalized nonparanormal data.
Transformer-MGK replaces redundant heads with Gaussian key mixtures, improving efficiency and performance.
problem Redundant attention heads in transformers degrade performance and efficiency.
method Transformer-MGK replaces redundant heads with a mixture of Gaussian keys.
result Transformer-MGK accelerates training and inference, reduces parameters and FLOPs, and achieves comparable or better accuracy.
A new method improves likelihood-free Bayesian inference by transforming summary statistics and using efficient Variational Bayes.
problem Incorrectly assuming normally distributed summary statistics in likelihood-free Bayesian inference.
method Wasserstein Gaussianization transformation combined with robust BSL and efficient Variational Bayes.
result Highly efficient and reliable approximate Bayesian inference for likelihood-free problems.
DGPFM uses deep Gaussian processes to map functions accurately and quantify uncertainty.
problem Learning mappings between functional spaces, especially when data are noisy, sparse, or irregularly sampled.
method Constructs a sequence of GP-based linear and nonlinear transformations directly in function space, leveraging kernel integral transforms, GP conditional means, and nonlinear activations sampled from Gaussian processes.
result Empirical results show DGPFM outperforms existing methods in predictive accuracy and uncertainty calibration.
ETGP improves multi-class classification efficiency.
problem Efficiently handling non-stationary, dependent multi-class classification problems.
method ETGP uses transformed Gaussian processes with efficient sparse variational inference.
result ETGPs outperform state-of-the-art methods in multi-class classification tasks.
Gaussianization flows transform any random vector into a Gaussian, enabling efficient computation and sample generation.
problem Transforming any random vector into a Gaussian for efficient computation and sample generation.
method Iterative Gaussianization and normalizing flow model.
result Gaussianization flows are universal approximators and achieve better performance on tabular datasets.
Transformers solve Gaussian Mixture Models without supervision.
problem Solving Gaussian Mixture Models (GMMs) unsupervised.
method Proposes TGMM, a transformer-based framework for GMM tasks.
result Transformers can effectively solve GMM tasks, improving upon classical methods.
Transformers learn to cluster Gaussian mixtures as well as the EM algorithm.
problem Learning guarantees of Transformers in multi-class clustering of Gaussian mixtures.
method Developed a theory connecting Transformer's Softmax Attention layers to the EM algorithm's workflow.
result Transformers achieve minimax optimal rate for clustering Gaussian mixtures with sufficient training samples and initialization.
Algorithm learns polynomial transformations of Gaussian distributions.
problem Learning high-dimensional polynomial transformations of Gaussian distributions.
method Polynomial-time algorithms for smoothed settings, tensor ring decomposition.
result First end-to-end guarantees for learning pushforwards under neural networks.
A new Bayesian modeling method is proposed by combining the maximization of the marginal likelihood with a momentum-space renormalization group transformation for Gaussian graphical models. Moreover, we present a scheme for computint the statistical averages of hyperparameters and mean square errors in our proposed met…
In this paper we consider the Gaussian thermostat ray transform on both closed Riemannian surfaces and compact Riemannian surfaces with boundary. We establish certain results on the injectivity of the thermostat ray transform and the surjectivity of its adjoint.
Transformers can solve complex filtering problems for non-Gaussian signals.
problem Non-linear and non-Markovian filtering problems for conditionally Gaussian signals.
method Continuous-time transformer models called filterformers.
result Filterformers can approximate the conditional law of non-Markovian and conditionally Gaussian signal processes.
Diffusion Transformer captures spatial-temporal dependencies in sequential data.
problem Capturing rich spatial and temporal dependencies in sequential data.
method Established theoretical guarantees for diffusion transformers learning Gaussian process data.
result Spatial-temporal dependencies are captured within attention layers of diffusion transformers.
The paper studies ray transforms on surfaces with negative curvature, proving injectivity and determining connections and Higgs fields.
problem Injectivity of ray transforms on surfaces with negative curvature and determination of connections and Higgs fields.
method Analysis of Gaussian thermostats on compact Riemannian surfaces with negative curvature, proving injectivity and determining connections and Higgs fields.
result Injectivity of the thermostat ray transform and determination of connections and Higgs fields.
Extends Gaussian process regression for non-Gaussian data.
problem Inadequate modeling of uncertainty and over-smoothing in non-Gaussian datasets.
method Time-changed Gaussian processes with Lévy processes.
result Improved modeling of heavy-tailed non-Gaussian behaviors.
GT is a new method for denoising and enhancing datasets using Gaussian density estimates.
problem Improving latent structures in datasets.
method GT is an iterative method that generates a new distance function by computing the ℓ2-Wasserstein distance between Gaussian density estimates. result GT is stable under perturbations and asymptotically ellipsoidal neighborhoods in the continuous case.
New Bäcklund transformations for discrete pseudospherical surfaces of revolution are found.
problem Constructing new non-rotational discrete pseudospherical surfaces.
method Explicit parametrizations and Bäcklund transformations for discrete constant negative Gaussian curvature surfaces of revolution.
result Conditions for Bäcklund transformations to preserve periodicity are provided.
Paper derives a simplified formula for Expected Improvement using log-transformed data.
problem Challenges in enhancing Bayesian optimization with Expected Improvement.
method Derives a closed form of Expected Improvement for Gaussian process trained on log-transformed objective.
result Provides a simplified formula for Expected Improvement.
Paper introduces rational Gaussian wavelets for efficient signal approximation.
problem Efficiently approximating complex signals with few coefficients.
method Continuous wavelet transform using rational Gaussian wavelets with adjustable parameters.
result Proposed rational Gaussian wavelets provide accurate signal approximations.
New pruning method retains model expressiveness for NLP tasks.
problem Pruning large pretrained transformer models for real-world deployment.
method Mixture Gaussian Prior Pruning (MGPP) algorithm.
result MGPP outperforms existing pruning methods in high sparsity settings.
Small neural networks embed arbitrary metric spaces into Gaussian mixtures.
problem Embedding arbitrary metric spaces into a fixed space with low distortion.
method Probabilistic transformers of small depth and width.
result Embeddings with low metric distortion for various metric spaces.
Study finds the number of modes in Gaussian kernel density estimators scales with sqrt(β log β).
problem Determining the number of clusters in Transformers.
method Used Kac-Rice formula and Edgeworth expansion to prove scaling.
result The expected number of modes scales as Θ(√(β log β)).
The inference of deep hierarchical models is problematic due to strong dependencies between the hierarchies. We investigate a specific transformation of the model parameters based on the multivariate distributional transform. This transformation is a special form of the reparametrization trick, flattens the hierarchy a…
Most signal processing problems involve the challenging task of multidimensional probability density function (PDF) estimation. In this work, we propose a solution to this problem by using a family of Rotation-based Iterative Gaussianization (RBIG) transforms. The general framework consists of the sequential applicatio…
Transformers tend to learn more symmetric functions in sequence data.
problem Understanding inductive bias in Transformers with infinitely over-parameterized models.
method Analyzing Transformers in the Gaussian process limit, using representation theory of the symmetric group.
result Transformers are biased towards more permutation symmetric functions, and this can be quantitatively predicted.
Rotates MFVI for better Gaussian approximations.
problem Improving variational approximations for complex distributions.
method Rotated coordinate system, PCA-based rotation, iterative Gaussianization.
result Significantly more accurate approximations with lower computational cost.
UT module refines VAE latent space, improving disentanglement and interpretability.
problem Irregular latent distributions cause posterior collapse and misalignment in VAEs.
method UT module uses G-KDE clustering, GM modeling, and PIT to transform latent space into uniform distribution.
result UT module enhances disentanglement and interpretability of latent representations.
Transformers can cluster data from Gaussian mixtures without supervision.
problem Clustering data from Gaussian mixtures without labeled data.
method Theoretical analysis of attention-based layers, focusing on a simplified two-head attention layer and an identity matrix attention layer.
result Attention-based layers can align with true mixture centroids and adapt to input-specific distributions.
Computing accurate estimates of the Fourier transform of analog signals from discrete data points is important in many fields of science and engineering. The conventional approach of performing the discrete Fourier transform of the data implicitly assumes periodicity and bandlimitedness of the signal. In this paper, we…
We consider a modification of the covariance function in Gaussian processes to correctly account for known linear constraints. By modelling the target function as a transformation of an underlying function, the constraints are explicitly incorporated in the model such that they are guaranteed to be fulfilled by any sam…
We introduce the Wasserstein Transform (WT), a general unsupervised framework for updating distance structures on given data sets with the purpose of enhancing features and denoising. Our framework represents each data point by a probability measure reflecting the neighborhood structure of the point, and then updates t…
Diagonal transformations preserve independence structures in non-Gaussian distributions.
problem Preserving independence structures in non-Gaussian distributions.
method Diagonal nonlinear transformations of multivariate normal variables.
result Independence structures are preserved in non-Gaussian distributions under diagonal transformations.
We investigate geometric aspects of the the Bäcklund transform of principal contact element nets. A Bäcklund transform exists if and only if it the principal contact element net is of constant negative Gaussian curvature (a pseudosphere). We describe an elementary construction of the Bäcklund transform and prove its co…
Novel CMG framework improves financial sentiment forecasting.
problem Challenges in short-term sentiment forecasting of financial OHLC data.
method Integrates chaos theory, Markov chains, and Gaussian processes with transformer models.
result Consistently outperforms traditional models in accuracy and efficiency.
Formulae for Bäcklund transformations of hyperbolic and elliptic sine-Gordon/sinh-Gordon equations.
problem Finding solutions for specific types of equations.
method Providing superposition formulae for Bäcklund transformations.
result Algebraically obtain infinitely many solutions after first integration.
New methods reduce computational cost for Gaussian Markov Random Fields with sparse constraints.
problem Inference and simulation of GMRFs are computationally prohibitive with many constraints.
method Proposes a basis transformation into blocks of constrained and non-constrained subspaces.
result Significantly outperforms existing alternatives in computational cost.
TM-VI uses flexible transformation models to approximate complex posteriors in Bayesian models.
problem Approximating complex posteriors in Bayesian models with limited flexibility.
method Transformation models for variational inference (TM-VI).
result TM-VI allows accurate approximation of complex posteriors in models with one parameter and works in a mean-field fashion for multi-parameter models.
We utilize copulas to constitute a unified framework for constructing and optimizing variational proposals in hierarchical Bayesian models. For models with continuous and non-Gaussian hidden variables, we propose a semiparametric and automated variational Gaussian copula approach, in which the parametric Gaussian copul…
Skyformer uses Gaussian kernel and Nyström method to speed up self-attention in transformers.
problem High computational cost of self-attention in transformers.
method Replaces softmax with Gaussian kernel and applies Nyström method for matrix approximation.
result Skyformer achieves comparable or better performance with fewer computation resources.
There has been growing recent interest in probabilistic interpretations of kernel-based methods as well as learning in Banach spaces. The absence of a useful Lebesgue measure on an infinite-dimensional reproducing kernel Hilbert space is a serious obstacle for such stochastic models. We propose an estimation model for …
Paper introduces a new method for Transformers with linear complexity.
problem No efficient relative positional encoding for linear Transformer models.
method Stochastic Positional Encoding (SPE) that replaces classical RPE.
result SPE behaves like RPE and performs well on benchmarks.