ARCNPs improve CNPs by autoregressively modeling dependencies.
problem CNPs struggle with modeling dependencies in predictions.
method Autoregressive deployment of factorized Gaussian CNPs.
result ARCNPs significantly outperform non-AR CNPs in various tasks.
ARMA cell simplifies neural autoregressive modeling for time series.
problem Complex RNN cells are not always necessary and can be inferior.
method Introduces ARMA cell, a simpler, modular approach for neural time series modeling.
result The ARMA cell is competitive with popular alternatives in performance.
Autoregressive models are among the best performing neural density estimators. We describe an approach for increasing the flexibility of an autoregressive model, based on modelling the random numbers that the model uses internally when generating data. By constructing a stack of autoregressive models, each modelling th…
A new neural network for efficient density estimation.
problem Efficient density estimation for high-dimensional data.
method Triangular neural network implementation of neural autoregressive flow (NAF).
result Achieves state-of-the-art bits-per-dimension indices on MNIST and CIFAR-10.
We propose a conditional non-autoregressive neural sequence model based on iterative refinement. The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task. We extensively evaluate the proposed model on machine…
Estimates causal effects using neural autoregressive density estimators.
problem Estimating causal effects in non-linear systems.
method Neural autoregressive density estimators within Pearl's do-calculus framework.
result Retrieves causal effects from non-linear systems without explicit modeling.
Normalizing flows and autoregressive models have been successfully combined to produce state-of-the-art results in density estimation, via Masked Autoregressive Flows (MAF), and to accelerate state-of-the-art WaveNet-based speech synthesis to 20x faster than real-time, via Inverse Autoregressive Flows (IAF). We unify a…
This work proposes an efficient autoregressive model for text generation.
problem The challenge of generating high-quality text with autoregressive models.
method Introduces a cascaded decoding approach using Markov transformers to achieve sub-linear parallel time generation.
result Shows competitive accuracy/speed tradeoff compared to existing methods on five machine translation datasets.
We introduce Hyper-Conditioned Neural Autoregressive Flow (HCNAF); a powerful universal distribution approximator designed to model arbitrarily complex conditional probability density functions. HCNAF consists of a neural-net based conditional autoregressive flow (AF) and a hyper-network that can take large conditions …
New method estimates spin system mutual information using neural networks.
problem Estimating mutual information in spin systems.
method Monte Carlo sampling enhanced by autoregressive neural networks.
result Area law satisfied for temperatures away from critical temperature.
Normalising flows (NFS) map two density functions via a differentiable bijection whose Jacobian determinant can be computed efficiently. Recently, as an alternative to hand-crafted bijections, Huang et al. (2018) proposed neural autoregressive flow (NAF) which is a universal approximator for density functions. Their fl…
The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast to earlier published flows, scales well to high-dimensional latent spaces. The proposed f…
Neural density estimators are flexible families of parametric models which have seen widespread use in unsupervised machine learning in recent years. Maximum-likelihood training typically dictates that these models be constrained to specify an explicit density. However, this limitation can be overcome by instead using …
NeLLoC improves image compression with parallel decoding.
problem Image compression with OOD generalization.
method Local autoregressive model with parallel decoding.
result Significant gains in compression runtime.
Neural autoregressive models are explicit density estimators that achieve state-of-the-art likelihoods for generative modeling. The D-dimensional data distribution is factorized into an autoregressive product of one-dimensional conditional distributions according to the chain rule. Data completion is a more involved ta…
The paper develops a learning theory for neural network-based CHARME models.
problem Developing a learning theory for CHARME models using neural networks.
method Proves the stationarity and ergodicity of CHARME models under weak conditions, then applies neural networks to derive strong consistency and asymptotic normality of estimators.
result Strong consistency and asymptotic normality of NN-based estimators of CHARME model weights and biases under weak conditions.
CTRNNs improve blood glucose forecasting in ICU, outperforming traditional models.
problem Forecasting blood glucose in ICU with irregular measurements.
method Continuous time autoregressive recurrent neural networks (CTRNNs) using neural ODE or neural flow layers.
result CTRNNs generally outperform traditional autoregressive models in probabilistic forecasting of blood glucose.
Paper proposes a neural network method for fast, interpretable AR model estimation.
problem Computational inefficiency and convergence issues in conventional AR model estimation.
method Embeds autoregressive structure into a feedforward neural network for coefficient estimation via backpropagation.
result Neural network method consistently recovers AR model coefficients, converging in all cases and providing reliable estimates.
BiGG model efficiently generates sparse graphs with reduced complexity.
problem Challenges in scalable deep learning for sparse graphs.
method BiGG model, an autoregressive model that leverages graph sparsity.
result Graph generation time complexity reduced from O(n2) to O((n+m)logn). Paper uses surprisal to dynamically allocate computation between fast and slow models.
problem Dynamic allocation of computation in neural networks.
method Surprisal-based dynamic model selection.
result Model can match baseline performance with 15% fewer FLOPs.
Thermalizer stabilizes autoregressive models for long-term predictions in chaotic systems.
problem Long-term predictions in chaotic spatiotemporal systems are unreliable due to trajectory divergence.
method Diffusion models are used to implicitly estimate the score of an invariant measure, which stabilizes autoregressive emulators by applying denoising during inference.
result Thermalization extends the time horizon of stable predictions by an order of magnitude in chaotic systems.
Efficiently combines autoregressive and set-based models for joint distributions.
problem Joint distributions over multiple predictions from set-based models.
method Causal autoregressive buffer that caches context and captures dependencies.
result Up to 20x faster joint sampling and density evaluation, up to 7x lower memory usage.
Study reveals issues with neural autoregressive models and proposes mode recovery cost.
problem Unreasonable affinity of neural autoregressive models to short and long sequences.
method Investigates modes of ground-truth, empirical, and decoding-induced distributions via mode recovery cost.
result Mode recovery cost varies depending on ground-truth distribution and impacts decoding-induced distribution.
This work maps Boltzmann distributions to ARNNs for better physics-based model approximations.
problem Approximating Boltzmann distributions of binary systems.
method Exact mapping of Boltzmann distribution to autoregressive neural network architecture.
result New ARNN architectures derived from physical models show superior performance.
This paper proposes CF-NADE, a neural autoregressive architecture for collaborative filtering (CF) tasks, which is inspired by the Restricted Boltzmann Machine (RBM) based CF model and the Neural Autoregressive Distribution Estimator (NADE). We first describe the basic CF-NADE model for CF tasks. Then we propose to imp…
EventFlow forecasts event sequences without autoregression, improving accuracy.
problem Forecasting errors in autoregressive models for event sequences.
method EventFlow uses flow matching to learn joint distributions over event times directly.
result EventFlow reduces forecast error by 20%-53% compared to baselines.
New method makes machine learning approximations unbiased and efficient.
problem Efficient sampling of complex probability distributions.
method Uses autoregressive neural networks with cluster updates and physical symmetries.
result Shows unbiased and low-variance approximations for phase transitions.
Study finds optimal vocabulary size for neural machine translation.
problem Imbalanced class distribution in language data affects NMT performance.
method Casts NMT as a classification task, analyzes vocabulary sizes, and tests multiple languages.
result Certain vocabulary sizes outperform others, explaining NMT performance.
We present Sequential Neural Likelihood (SNL), a new method for Bayesian inference in simulator models, where the likelihood is intractable but simulating data from the model is possible. SNL trains an autoregressive flow on simulated data in order to learn a model of the likelihood in the region of high posterior dens…
Neural networks improve gravitational-wave parameter estimation.
problem Estimating parameters of binary black hole systems from gravitational-wave data.
method Autoregressive normalizing flows for likelihood-free inference.
result Performance comparable to current best deep-learning approaches, with fast sampling.
Improved language generation with faster sampling speed.
problem Speed and coherence issues in autoregressive language models.
method Introduces Neural Flow Diffusion Models (NFDM) for discrete state spaces.
result Substantially reduces likelihood gap with autoregressive models.
Improved lattice field theory simulations with local-Autoregressive Conditional Normalizing Flow.
problem Efficiently sampling lattice field theories with computational challenges.
method Integrates locality into autoregressive conditional normalizing flows.
result Autocorrelation times improved by orders of magnitude for φ4 theory on a 2D lattice. Identification of a groundwater contaminant source simultaneously with the hydraulic conductivity in highly-heterogeneous media often results in a high-dimensional inverse problem. In this study, a deep autoregressive neural network-based surrogate method is developed for the forward model to allow us to solve efficien…
Generative neural samplers estimate quantum spin system properties.
problem Estimating observables for quantum spin systems.
method Autoregressive models using Suzuki-Trotter transformation.
result Results for energy, specific heat, and susceptibility are in good agreement with Monte Carlo methods.
LMConv improves autoregressive models for image generation and completion.
problem Limited generation order in autoregressive models restricts their applicability.
method Introduces LMConv, a modified 2D convolution that allows arbitrary masks to be applied to weights.
result LMConv achieves improved performance on image density estimation and coherent completions.
A normalizing flow models a complex probability density as an invertible transformation of a simple base density. Flows based on either coupling or autoregressive transforms both offer exact density evaluation and sampling, but rely on the parameterization of an easily invertible elementwise transformation, whose choic…
PARNN improves ARNN with ARIMA feedback for accurate long-range forecasting.
problem Accurate long-range forecasting of complex time series data.
method Improves ARNN using ARIMA feedback, providing uncertainty quantification.
result PARNN outperforms state-of-the-art forecasters across various horizons.
Efficient methods for answering complex probabilistic queries in sequential data.
problem Complex probabilistic queries in sequential data.
method Broad class of novel approximation techniques for marginalization in sequential models.
result Efficient techniques for answering long-range probabilistic queries.
This review compares various deep generative models.
problem Training deep neural networks to model data distributions.
method Comprehensive comparison of VAEs, GANs, flows, energy models, and autoregressives.
result Trade-offs and interrelationships among different models.
Improved simulation of phase transitions using hierarchical autoregressive networks.
problem Simulating phase transitions in complex systems.
method Hierarchical Autoregressive Neural (HAN) network sampling algorithm.
result Significant improvement in statistical uncertainty compared to the Wolff cluster algorithm.
Non-autoregressive translation (NAT) models remove the dependence on previous target tokens and generate all target tokens in parallel, resulting in significant inference speedup but at the cost of inferior translation accuracy compared to autoregressive translation (AT) models. Considering that AT models have higher a…
Vector autoregressive models characterize a variety of time series in which linear combinations of current and past observations can be used to accurately predict future observations. For instance, each element of an observation vector could correspond to a different node in a network, and the parameters of an autoregr…
ARTree uses deep learning to infer tree topologies efficiently.
problem Efficient phylogenetic inference from tree topologies.
method Deep autoregressive model based on graph neural networks (GNNs).
result ARTree provides a flexible family of distributions over tree topologies.
Paper explores variable skipping to speed up range density estimation.
problem Efficiently estimating range densities over high-dimensional data.
method Variable skipping technique to accelerate range density estimation.
result 10-100x efficiency improvements in challenging high-quantile error metrics.
Single autoregressive model outperforms ensemble methods in offline reinforcement learning.
problem Offline reinforcement learning with limited data and model errors.
method Infer system dynamics from data and optimize policies on model rollouts, using a single autoregressive model.
result Single autoregressive model achieves better performance than ensembles on the D4RL benchmark.
SALT models combine ARHMM and SLDS for efficient, interpretable time-series analysis.
problem Efficient modeling of systems with time-varying dynamics and long-range dependencies.
method Switching autoregressive low-rank tensor models parameterized with a low-rank factorization.
result SALT models provide a balance of interpretability and efficiency, outperforming ARHMMs and SLDSs.
Multivariate binary distributions can be decomposed into products of univariate conditional distributions. Recently popular approaches have modeled these conditionals through neural networks with sophisticated weight-sharing structures. It is shown that state-of-the-art performance on several standard benchmark dataset…
Graph neural networks improve volatility forecasts and portfolio performance.
problem Improving volatility forecasting for better portfolio performance.
method Compared Heterogeneous Autoregressive and Long Short-Term Memory models with GraphSAGE models built on rolling correlation, sector, and Granger-causal graphs.
result GraphSAGE models with macro regime features outperform other models in terms of forecast accuracy, ranking quality, and portfolio Sharpe ratio.