arXiv research

Linear attention in Transformers can be interpreted as dynamic VAR models.

Proposes a non-autoregressive Transformer for time series forecasting.

This work proposes an efficient autoregressive model for text generation.

Transformers become faster by linearizing self-attention.

This paper explores how Transformers predict next tokens in autoregressive tasks.

Transformers learn a mesa-optimizer to implement in-context learning.

New method forecasts multilinear data using tensor autoregression.

Efficiently combines autoregressive and set-based models for joint distributions.

We present the Insertion Transformer, an iterative, partially autoregressive model for sequence generation based on insertion operations. Unlike typical autoregressive models which rely on a fixed, often left-to-right ordering of the output, our approach accommodates arbitrary orderings by allowing for tokens to be ins…

Normalizing flows and autoregressive models have been successfully combined to produce state-of-the-art results in density estimation, via Masked Autoregressive Flows (MAF), and to accelerate state-of-the-art WaveNet-based speech synthesis to 20x faster than real-time, via Inverse Autoregressive Flows (IAF). We unify a…

Parallelizes autoregressive generation using VSSM.

The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast to earlier published flows, scales well to high-dimensional latent spaces. The proposed f…

A normalizing flow models a complex probability density as an invertible transformation of a simple density. The invertibility means that we can evaluate densities and generate samples from a flow. In practice, autoregressive flow-based models are slow to invert, making either density estimation or sample generation sl…

Transformers can simulate MLE for Bayesian network sequences.

PARD generates graphs efficiently and invariantly to node ordering.

A normalizing flow models a complex probability density as an invertible transformation of a simple base density. Flows based on either coupling or autoregressive transforms both offer exact density evaluation and sampling, but rely on the parameterization of an easily invertible elementwise transformation, whose choic…

The fundamental task of general density estimation $p(x)$ has been of keen interest to machine learning. In this work, we attempt to systematically characterize methods for density estimation. Broadly speaking, most of the existing methods can be categorized into either using: \textit{a}) autoregressive models to estim…

Recently very deep transformers have outperformed conventional bi-directional long short-term memory networks by a large margin in speech recognition. However, to put it into production usage, inference computation cost is still a serious concern in real scenarios. In this paper, we study two different non-autoregressi…

WAVE improves time series forecasting by integrating AR and MA components.

Transformer learns long-term dependencies from real-world data.

CHOOSE enhances shallow Transformers for wireless symbol detection.

Transformers can generalize to a large task family with only a few demonstrations.

Transformers encode latent distributions in text, improving performance in out-of-distribution cases.

Flow-based generative models are a family of exact log-likelihood models with tractable sampling and latent-variable inference, hence conceptually attractive for modeling complex distributions. However, flow-based models are limited by density estimation performance issues as compared to state-of-the-art autoregressive…

Monotonic neural networks have recently been proposed as a way to define invertible transformations. These transformations can be combined into powerful autoregressive flows that have been shown to be universal approximators of continuous probability distributions. Architectures that ensure monotonicity typically enfor…

Generative neural samplers estimate quantum spin system properties.

Normalising flows (NFS) map two density functions via a differentiable bijection whose Jacobian determinant can be computed efficiently. Recently, as an alternative to hand-crafted bijections, Huang et al. (2018) proposed neural autoregressive flow (NAF) which is a universal approximator for density functions. Their fl…

New approaches improve uncertainty quantification in autoregressive models for sequence data.

Improved flow-based models capture dependencies better with multi-scale autoregressive priors.

AutoGraph uses transformers to efficiently generate graphs as sequences.

Paper proposes LATC for multivariate time series prediction and missing data imputation.

The study explores how Transformers predict the next token in a sequence.

Flow-based deep generative models learn data distributions by transforming a simple base distribution into a complex distribution via a set of invertible transformations. Due to the invertibility, such models can score unseen data samples by computing their exact likelihood under the learned distribution. This makes fl…

Most machine translation systems generate text autoregressively from left to right. We, instead, use a masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation. This approach allows for efficient iterative…

Flow models have recently made great progress at modeling ordinal discrete data such as images and audio. Due to the continuous nature of flow models, dequantization is typically applied when using them for such discrete data, resulting in lower bound estimates of the likelihood. In this paper, we introduce subset flow…

Paper trains a Transformer to add numbers of any length.

Simple tabular event prediction model outperforms existing methods.

MPP trains a transformer to predict multiple physical systems, improving accuracy across various tasks.

Transformer model predicts train axle vibrations for safer maintenance.

New language model shows context length impacts generation quality and reasoning ability.

PolyGen models 3D meshes directly, predicting vertices and faces sequentially.

AR CI framework handles complex confounders and sequential actions.

New study finds best language model architecture and pretraining objective for zero-shot tasks.

The paper explores how LLMs with CoT improve performance on complex tasks.

LTMs use latent vectors for efficient autoregressive generation.

Reinforcement learning after next-token prediction aids in learning from diverse sequence lengths.

Study bounds VAR model's circuit complexity, showing it's limited to TC^0 circuits.

Stock prices predicted using a Transformer model.