BiGG model efficiently generates sparse graphs with reduced complexity.
problem Challenges in scalable deep learning for sparse graphs.
method BiGG model, an autoregressive model that leverages graph sparsity.
result Graph generation time complexity reduced from O(n2) to O((n+m)logn). This review compares various deep generative models.
problem Training deep neural networks to model data distributions.
method Comprehensive comparison of VAEs, GANs, flows, energy models, and autoregressives.
result Trade-offs and interrelationships among different models.
Missing value imputation is a fundamental problem in spatiotemporal modeling, from motion tracking to the dynamics of physical systems. Deep autoregressive models suffer from error propagation which becomes catastrophic for imputing long-range sequences. In this paper, we take a non-autoregressive approach and propose …
Paper explores variable skipping to speed up range density estimation.
problem Efficiently estimating range densities over high-dimensional data.
method Variable skipping technique to accelerate range density estimation.
result 10-100x efficiency improvements in challenging high-quantile error metrics.
Efficient autoregressive entity linking with correction for faster, more accurate results.
problem High computational cost and non-parallelizable decoding in autoregressive entity linking.
method Parallelizes autoregressive linking across all mentions, uses a shallow decoder, and adds a discriminative correction term.
result 70 times faster and more accurate than previous methods, outperforming state-of-the-art approaches.
Paper proposes a forecasting model combining autoregressive models with spectral attention.
problem Time series forecasting across various domains.
method Combines deep autoregressive models with Spectral Attention (SA) module.
result SAAM consistently demonstrates improved forecasting accuracy compared to state-of-the-art approaches.
ARTree uses deep learning to infer tree topologies efficiently.
problem Efficient phylogenetic inference from tree topologies.
method Deep autoregressive model based on graph neural networks (GNNs).
result ARTree provides a flexible family of distributions over tree topologies.
We introduce a deep, generative autoencoder capable of learning hierarchies of distributed representations from data. Successive deep stochastic hidden layers are equipped with autoregressive connections, which enable the model to be sampled from quickly and exactly via ancestral sampling. We derive an efficient approx…
Identification of a groundwater contaminant source simultaneously with the hydraulic conductivity in highly-heterogeneous media often results in a high-dimensional inverse problem. In this study, a deep autoregressive neural network-based surrogate method is developed for the forward model to allow us to solve efficien…
This paper introduces methods to handle discrete data by dequantization.
problem Handling discrete data in deep learning models.
method Dequantization framework, including importance-weighted and Rényi dequantization objectives, and autoregressive dequantization.
result Improved performance on uniform dequantization distributions and state-of-the-art negative log-likelihood on CIFAR10.
New model handles missing data effectively in autoregressive models.
problem Handling missing data in autoregressive models.
method Reinterpret existing models through missing data lens, introduce principled framework for incomplete datasets, active information acquisition.
result MO-ARM consistently outperforms imputation baselines across real-world benchmarks.
LMConv improves autoregressive models for image generation and completion.
problem Limited generation order in autoregressive models restricts their applicability.
method Introduces LMConv, a modified 2D convolution that allows arbitrary masks to be applied to weights.
result LMConv achieves improved performance on image density estimation and coherent completions.
CARRNN tackles deep learning for sporadic data, improving prediction errors in healthcare.
problem Challenges in learning temporal patterns from sporadic multivariate longitudinal data.
method Developed a novel deep learning architecture combining RNN and CAR models, using a generalized discrete-time autoregressive model.
result CARRNN achieves the lowest prediction errors in multivariate time-series regression tasks.
Neural networks improve gravitational-wave parameter estimation.
problem Estimating parameters of binary black hole systems from gravitational-wave data.
method Autoregressive normalizing flows for likelihood-free inference.
result Performance comparable to current best deep-learning approaches, with fast sampling.
This paper introduces a new learning-based approach for approximately solving the Travelling Salesman Problem on 2D Euclidean graphs. We use deep Graph Convolutional Networks to build efficient TSP graph representations and output tours in a non-autoregressive manner via highly parallelized beam search. Our approach ou…
NVAE improves VAE performance on large image datasets.
problem Improving variational autoencoder performance for large image datasets.
method Deep hierarchical VAE with depth-wise separable convolutions and batch normalization, residual parameterization of Normal distributions, and spectral regularization.
result NVAE achieves state-of-the-art results on MNIST, CIFAR-10, CelebA 64, and CelebA HQ datasets.
Autoregressive feedback is considered a necessity for successful unconditional text generation using stochastic sequence models. However, such feedback is known to introduce systematic biases into the training process and it obscures a principle of generation: committing to global information and forgetting local nuanc…
Behavior cloning training instabilities amplified by SGD noise over long horizons.
problem Training instabilities in behavior cloning with deep neural networks.
method Empirical dissection of minibatch SGD updates and their effects on long-horizon rewards.
result Exponential moving average (EMA) of iterates effectively mitigates gradient variance amplification (GVA).
Single autoregressive model outperforms ensemble methods in offline reinforcement learning.
problem Offline reinforcement learning with limited data and model errors.
method Infer system dynamics from data and optimize policies on model rollouts, using a single autoregressive model.
result Single autoregressive model achieves better performance than ensembles on the D4RL benchmark.
Non-autoregressive transformer improves speech recognition speed and accuracy.
problem Reducing inference computation cost in speech recognition.
method Two non-autoregressive transformer structures (A-CMLM and A-FMLM) trained with masked tokens and iterative prediction during inference.
result Non-autoregressive transformer can match autoregressive transformer performance with 7x speedup.
ARMA cell simplifies neural autoregressive modeling for time series.
problem Complex RNN cells are not always necessary and can be inferior.
method Introduces ARMA cell, a simpler, modular approach for neural time series modeling.
result The ARMA cell is competitive with popular alternatives in performance.
The paper compares DL models to WP curve modeling for forecasting with irregular shutdowns.
problem Forecasting wind power with irregular shutdowns due to redispatching.
method Compared autoregressive DL models to WP curve modeling.
result WP curve modeling achieves lower forecasting errors and is more computationally efficient.
The paper tackles deep learning from dependent data, achieving optimal performance.
problem Deep learning from strongly mixing observations, especially with regularization and optimality.
method Sparse-penalized regularization for deep neural networks, oracle inequality for expected excess risk.
result Deep neural network estimator achieves minimax optimal rate for nonparametric autoregression.
This paper proposes CF-NADE, a neural autoregressive architecture for collaborative filtering (CF) tasks, which is inspired by the Restricted Boltzmann Machine (RBM) based CF model and the Neural Autoregressive Distribution Estimator (NADE). We first describe the basic CF-NADE model for CF tasks. Then we propose to imp…
A new drift detection method based on autoregressive models.
problem Concept drift in real-world data leads to decreased model performance.
method Autoregressive based drift detection method (ADDM).
result ADDM outperforms state-of-the-art drift detection methods.
PARNN improves ARNN with ARIMA feedback for accurate long-range forecasting.
problem Accurate long-range forecasting of complex time series data.
method Improves ARNN using ARIMA feedback, providing uncertainty quantification.
result PARNN outperforms state-of-the-art forecasters across various horizons.
GraphAF generates chemically valid molecules efficiently and accurately.
problem Generating chemically valid molecular structures while optimizing chemical properties.
method Flow-based autoregressive model combining autoregressive and flow-based approaches.
result GraphAF generates 68% chemically valid molecules without chemical knowledge rules and 100% with rules, achieving state-of-the-art performance.
Sparse deep learning improves prediction uncertainty for time series data.
problem Uncertainty quantification for dependent data like time series.
method Sparse recurrent neural networks (RNNs) for time series data.
result Sparse deep learning can consistently estimate and predict time series data with correct uncertainty quantification.
APLC-XLNet improves XMTC by clustering labels and reducing computational time.
problem Efficiently tagging texts with many labels from a large set.
method Fine-tunes XLNet with APLC to approximate cross entropy loss.
result Achieved state-of-the-art results on XMTC benchmarks.
Deep learning models outperform traditional methods in stock price prediction.
problem Improving stock price prediction accuracy using deep learning.
method Comparative analysis of deep learning models (LSTM, GRU) and traditional methods (ARIMA, ARMA) on historical data.
result Deep learning models, particularly LSTM, outperform traditional methods in predicting stock prices across different time horizons.
AR CI framework handles complex confounders and sequential actions.
problem Low-dimensional confounders and singleton actions in causal inference.
method Sequencification to transform data into sequences, enabling CI with complex confounders and sequential actions.
result AR model can estimate multiple causal quantities using a single model, simplifying inference and improving outcome prediction.
Improved time series forecasting with multivariate probabilistic models.
problem Improving accuracy in forecasting time series with statistical dependencies.
method Conditioned Normalizing Flows for autoregressive deep learning models.
result Improved performance over state-of-the-art models on real-world data sets.
We propose a supervised anomaly detection method based on neural density estimators, where the negative log likelihood is used for the anomaly score. Density estimators have been widely used for unsupervised anomaly detection. By the recent advance of deep learning, the density estimation performance has been greatly i…
State-space models (SSMs) provide a flexible framework for modelling time-series data. Consequently, SSMs are ubiquitously applied in areas such as engineering, econometrics and epidemiology. In this paper we provide a fast approach for approximate Bayesian inference in SSMs using the tools of deep learning and variati…
Autoregressive models are among the best performing neural density estimators. We describe an approach for increasing the flexibility of an autoregressive model, based on modelling the random numbers that the model uses internally when generating data. By constructing a stack of autoregressive models, each modelling th…
Efficiently improves non-autoregressive sequence models for better translation performance.
problem Heavy inference latency and inconsistent output sentences in non-autoregressive models.
method Incorporates a structured inference module with an efficient CRF approximation and dynamic transition technique.
result Significantly better translation performance (BLEU score 26.80) compared to previous non-autoregressive models.
This work proposes an efficient autoregressive model for text generation.
problem The challenge of generating high-quality text with autoregressive models.
method Introduces a cascaded decoding approach using Markov transformers to achieve sub-linear parallel time generation.
result Shows competitive accuracy/speed tradeoff compared to existing methods on five machine translation datasets.
Deep learning predicts currency volatility accurately.
problem Predicting future volatility in Forex trading.
method Constructed a deep-learning network using multiscale LSTM with multi-currency pairs.
result Multiscale LSTM model outperforms conventional models.
Deep generative models are rapidly gaining traction in medical imaging. Nonetheless, most generative architectures struggle to capture the underlying probability distributions of volumetric data, exhibit convergence problems, and offer no robust indices of model uncertainty. By comparison, the autoregressive generative…
Bayesian method for multivariate autoregressive models with exogenous inputs.
problem Estimating uncertainties in autoregressive models with exogenous inputs.
method Recursive Bayesian estimation via message passing in a factor graph.
result Produces full posterior distributions for autoregressive coefficients and noise precision.
Autoregressive state transitions, where predictions are conditioned on past predictions, are the predominant choice for both deterministic and stochastic sequential models. However, autoregressive feedback exposes the evolution of the hidden state trajectory to potential biases from well-known train-test discrepancies.…
We propose a general framework for solving statistical mechanics of systems with finite size. The approach extends the celebrated variational mean-field approaches using autoregressive neural networks, which support direct sampling and exact calculation of normalized probability of configurations. It computes variation…
Alternative sampling method for autoregressive models using Langevin dynamics.
problem Efficiently sampling from autoregressive models.
method Initialize sequences with white noise and follow Langevin dynamics on global log-likelihood.
result Parallelizes and generalizes sampling process for autoregressive models.
GAMs combine autoregressive and log-linear components for data-efficient sequence learning.
problem Poor performance of standard autoregressive models under small-data conditions.
method Introduce Global Autoregressive Models (GAMs) combining autoregressive and log-linear components, trained in two steps.
result GAMs show a strong perplexity reduction over standard models in language modelling.
Forecasting time series data is an important subject in economics, business, and finance. Traditionally, there are several techniques to effectively forecast the next lag of time series data such as univariate Autoregressive (AR), univariate Moving Average (MA), Simple Exponential Smoothing (SES), and more notably Auto…
Study forecasts COVID-19 cases in Iran using deep learning.
problem Predicting the spread of COVID-19 in Iran.
method Deep learning models (Prophet, LSTM, AR, ARIMA) for forecasting.
result Deep learning models outperform time series forecasting.
Paper proposes AXE loss for non-autoregressive machine translation, improving performance.
problem Challenges in training non-autoregressive models due to lack of autoregressive factors and cross entropy loss penalties.
method Proposes aligned cross entropy (AXE) loss function using a differentiable dynamic program for better word order alignment.
result AXE-based training improves performance on major WMT benchmarks and sets a new state of the art for non-autoregressive models.
SMART training improves mask-predict translations.
problem Closing the performance gap between semi-autoregressive and autoregressive models.
method SMART training method for conditional masked language models.
result SMART-trained models produce higher-quality translations.