Autoregressive models are among the best performing neural density estimators. We describe an approach for increasing the flexibility of an autoregressive model, based on modelling the random numbers that the model uses internally when generating data. By constructing a stack of autoregressive models, each modelling th…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Masked Autoregressive Flows”
Paper uses GMM and MAF for probabilistic classification, outperforming simpler models.
problem Classifying data with complex distributions.
method Density estimation using Gaussian Mixture Model and Masked Autoregressive Flow.
result Proposed classifiers outperform simpler models like linear discriminant analysis.
Flow-based generative models, conceptually attractive due to tractability of both the exact log-likelihood computation and latent-variable inference, and efficiency of both training and sampling, has led to a number of impressive empirical successes and spawned many advanced variants and theoretical investigations. Des…
Normalizing flows and autoregressive models have been successfully combined to produce state-of-the-art results in density estimation, via Masked Autoregressive Flows (MAF), and to accelerate state-of-the-art WaveNet-based speech synthesis to 20x faster than real-time, via Inverse Autoregressive Flows (IAF). We unify a…
The recently proposed mask-predict decoding algorithm has narrowed the performance gap between semi-autoregressive machine translation models and the traditional left-to-right approach. We introduce a new training method for conditional masked language models, SMART, which mimics the semi-autoregressive behavior of mas…
Most machine translation systems generate text autoregressively from left to right. We, instead, use a masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation. This approach allows for efficient iterative…
Flow-based deep generative models learn data distributions by transforming a simple base distribution into a complex distribution via a set of invertible transformations. Due to the invertibility, such models can score unseen data samples by computing their exact likelihood under the learned distribution. This makes fl…
Unified framework for ICL in causal and masked models.
problem Understanding ICL in masked language models and comparing it to causal models.
method Developed a statistical learning framework representing context by empirical measure and predicting using context and query.
result Upper bounds for masked and autoregressive objectives under Wasserstein-type regularity conditions.
This study compares different types of normalizing flows for generating complex distributions.
problem Comparing different types of normalizing flows for generating complex distributions.
method Real-valued non-Volume preserving (RealNVP), masked autoregressive flow (MAF), coupling rational quadratic spline (C-RQS), and autoregressive rational quadratic spline (A-RQS) were compared using statistical tests.
result A-RQS algorithm outperforms others in terms of accuracy and training speed.
Simplified masked diffusion models improve discrete data generation.
problem Complex model formulations and unclear relationships hinder discrete data generative modeling.
method Developed a simple and general framework for masked diffusion models.
result Models trained on OpenWebText surpass prior diffusion language models and outperform autoregressive models.
LMConv improves autoregressive models for image generation and completion.
problem Limited generation order in autoregressive models restricts their applicability.
method Introduces LMConv, a modified 2D convolution that allows arbitrary masks to be applied to weights.
result LMConv achieves improved performance on image density estimation and coherent completions.
Latent Noise Injection improves synthetic data generation for privacy and statistical alignment.
problem Slow convergence of generative models in high-dimensional settings.
method Latent Noise Injection using Masked Autoregressive Flows (MAF).
result Synthetic data closely reflects the underlying distribution, especially in high-dimensional settings.
Listen and Fill in the Missing Letters: Non-Autoregressive Transformer for Speech Recognitioneess.AS
Recently very deep transformers have outperformed conventional bi-directional long short-term memory networks by a large margin in speech recognition. However, to put it into production usage, inference computation cost is still a serious concern in real scenarios. In this paper, we study two different non-autoregressi…
StrNN uses neural network structures to learn conditional independencies.
problem Learning conditional independencies in neural networks.
method Designing masks for neural networks based on binary matrix factorization.
result StrNN improves density estimation and causal inference.
CoSMIC extends flow-based SVI to transdimensional problems.
problem Bayesian structure learning and model selection with multi-model parameter spaces.
method Normalizing flows with a combined stochastic variational transdimensional inference approach.
result Improved performance on high-cardinality model spaces.
This research compares masked diffusion models to autoregressive language models, focusing on architectural differences.
problem Comparing masked diffusion models to autoregressive language models due to architectural differences.
method Equitably compare MDMs within a decoder-only framework, investigating architectural influences.
result Decoder-only MDMs can achieve significant speedups and comparable perplexity with temperature annealing.
New method reduces discrete flow transitions, improving perplexity estimation.
problem Stochasticity in discrete paths makes rectification strategies ineffective.
method Dynamic-optimal-transport-like minimization objective with minibatch strategies.
result 32 times reduction in transitions for same perplexity.
Paper proposes AXE loss for non-autoregressive machine translation, improving performance.
problem Challenges in training non-autoregressive models due to lack of autoregressive factors and cross entropy loss penalties.
method Proposes aligned cross entropy (AXE) loss function using a differentiable dynamic program for better word order alignment.
result AXE-based training improves performance on major WMT benchmarks and sets a new state of the art for non-autoregressive models.
A new method detects anomalous sounds using self-supervised learning.
problem Detecting anomalous sounds in noisy data.
method Flow-based self-supervised density estimation using Normalizing Flows.
result The method improves anomaly detection performance significantly.
There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder's parameters to respect autoregressive constraints: ea…
While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. In this paper, we show that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-d…
Paper proposes energy objective for training normalizing flows without determinants.
problem Challenges in training normalizing flows due to Jacobian determinants.
method Introduces energy objective based on proper scoring rules, determinant-free.
result Energy objective supports novel model families and competitive performance.
Enhanced Sampling Scheme improves masked generative modeling.
problem Limitations of existing sampling schemes in masked non-autoregressive generative modeling.
method ESS consists of three stages: Naive Iterative Decoding, Critical Reverse Sampling, and Critical Resampling.
result ESS achieves significant performance gains in unconditional and class-conditional sampling.
The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast to earlier published flows, scales well to high-dimensional latent spaces. The proposed f…
Improved lattice field theory simulations with local-Autoregressive Conditional Normalizing Flow.
problem Efficiently sampling lattice field theories with computational challenges.
method Integrates locality into autoregressive conditional normalizing flows.
result Autocorrelation times improved by orders of magnitude for theory on a 2D lattice.
Autoregressive flow models can perform causal discovery and inference tasks.
problem Causal inference tasks such as causal discovery and interventional predictions.
method Using autoregressive flow models to estimate causal directions and make predictions.
result Autoregressive flows can accurately perform causal inference tasks without restrictive assumptions.
Flow-based generative models are a family of exact log-likelihood models with tractable sampling and latent-variable inference, hence conceptually attractive for modeling complex distributions. However, flow-based models are limited by density estimation performance issues as compared to state-of-the-art autoregressive…
Efficiently combines autoregressive and set-based models for joint distributions.
problem Joint distributions over multiple predictions from set-based models.
method Causal autoregressive buffer that caches context and captures dependencies.
result Up to 20x faster joint sampling and density evaluation, up to 7x lower memory usage.
Improved flow-based models capture dependencies better with multi-scale autoregressive priors.
problem Limited expressiveness of flow-based models for long-range data dependencies.
method Introducing channel-wise dependencies through multi-scale autoregressive priors (mAR) in split coupling flow layers (mAR-SCF).
result Achieves state-of-the-art density estimation results on MNIST, CIFAR-10, and ImageNet.
Causal Autoregressive Flowsstat.ML
Causal autoregressive flows enable accurate causal inference and prediction.
problem Causal discovery and interventional predictions in machine learning.
method Autoregressive normalizing flows with fixed variable orderings.
result Causal models derived from autoregressive flows are identifiable and allow for accurate interventional and counterfactual predictions.
Normalising flows (NFS) map two density functions via a differentiable bijection whose Jacobian determinant can be computed efficiently. Recently, as an alternative to hand-crafted bijections, Huang et al. (2018) proposed neural autoregressive flow (NAF) which is a universal approximator for density functions. Their fl…
Cubic-Spline Flowsstat.ML
A normalizing flow models a complex probability density as an invertible transformation of a simple density. The invertibility means that we can evaluate densities and generate samples from a flow. In practice, autoregressive flow-based models are slow to invert, making either density estimation or sample generation sl…
Normalizing flows are a powerful class of generative models for continuous random variables, showing both strong model flexibility and the potential for non-autoregressive generation. These benefits are also desired when modeling discrete random variables such as text, but directly applying normalizing flows to discret…
DTM improves dLLM fine-tuning stability and performance.
problem Intractable sequence-level marginal likelihoods for masked diffusion models.
method Discrete Tilt Matching (DTM) recasts dLLM fine-tuning as state-level matching of local unmasking posteriors under reward tilting.
result DTM yields strong gains on Sudoku and Countdown while remaining competitive on MATH500 and GSM8K.
We introduce Hyper-Conditioned Neural Autoregressive Flow (HCNAF); a powerful universal distribution approximator designed to model arbitrarily complex conditional probability density functions. HCNAF consists of a neural-net based conditional autoregressive flow (AF) and a hyper-network that can take large conditions …
Remasking improves the quality of discrete diffusion models for natural language and image generation.
problem Limited iterative refinement in masked discrete diffusion models.
method Introducing ReMDM sampler that allows remasking during inference.
result Remasking enables better quality outputs with increased sampling steps.
Flow-based generative models are powerful exact likelihood models with efficient sampling and inference. Despite their computational efficiency, flow-based models generally have much worse density modeling performance compared to state-of-the-art autoregressive models. In this paper, we investigate and improve upon thr…
Pretrained masked language models (MLMs) require finetuning for most NLP tasks. Instead, we evaluate MLMs out of the box via their pseudo-log-likelihood scores (PLLs), which are computed by masking tokens one by one. We show that PLLs outperform scores from autoregressive language models like GPT-2 in a variety of task…
EventFlow forecasts event sequences without autoregression, improving accuracy.
problem Forecasting errors in autoregressive models for event sequences.
method EventFlow uses flow matching to learn joint distributions over event times directly.
result EventFlow reduces forecast error by 20%-53% compared to baselines.
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cs.CL
New study finds best language model architecture and pretraining objective for zero-shot tasks.
problem Evaluating which language model architectures and pretraining objectives best enable zero-shot generalization.
method Compared three model architectures and two pretraining objectives across 170 billion tokens, with and without finetuning.
result Causal decoder-only models trained on autoregressive language modeling exhibit strongest zero-shot generalization.
ARDMs are a new model class for autoregressive diffusion that generalize existing models and can compress data efficiently.
problem Efficient data compression and generation.
method Autoregressive Diffusion Models (ARDMs) that generalize existing autoregressive models and discrete diffusion models.
result ARDMs require significantly fewer steps for compression compared to discrete diffusion models.
Improving Variational Auto-Encoders using convex combination linear Inverse Autoregressive Flowstat.ML
In this paper, we propose a new volume-preserving flow and show that it performs similarly to the linear general normalizing flow. The idea is to enrich a linear Inverse Autoregressive Flow by introducing multiple lower-triangular matrices with ones on the diagonal and combining them using a convex combination. In the …
Estimates high-dimensional posterior densities by marginal distributions and neural networks.
problem High-dimensional probability density estimation for inference is difficult.
method Direct estimation of lower-dimensional marginal distributions, using Moment Networks for fast computation of moments.
result Demonstrates estimation of gravitational wave time series and applications in cosmology.
A new neural network for efficient density estimation.
problem Efficient density estimation for high-dimensional data.
method Triangular neural network implementation of neural autoregressive flow (NAF).
result Achieves state-of-the-art bits-per-dimension indices on MNIST and CIFAR-10.
Theoretical analysis shows MDMs can be efficient but not for all metrics.
problem Understanding the efficiency-accuracy trade-off of diffusion language models.
method Theoretical analysis of Masked Diffusion Model (MDM) using perplexity and sequence error rate as metrics.
result MDM achieves near-optimal perplexity but requires linear scaling for sequence error rate, highlighting efficiency-accuracy trade-offs.
AR-Flow VAE improves blind source separation with flexible autoregressive priors.
problem Unsupervised blind source separation of latent signals from mixtures.
method AR-Flow VAE uses autoregressive flows to model latent sources, enhancing flexibility and capturing complex dependencies.
result AR-Flow VAE effectively separates latent sources, demonstrating improved performance over conventional methods.
This review compares various deep generative models.
problem Training deep neural networks to model data distributions.
method Comprehensive comparison of VAEs, GANs, flows, energy models, and autoregressives.
result Trade-offs and interrelationships among different models.
MDMs train to decode tokens in a random order, which affects performance; we show they can be optimized for a favorable order.
problem Performance of MDMs is affected by the random order in which tokens are decoded.
method We show that MDMs can be optimized for a favorable decoding order by equipping their continuous-time variational objective with multivariate noise schedules.
result MDMs can be decomposed into a weighted auto-regressive losses over orders, making them auto-regressive models with learnable orders.