Paper proposes AXE loss for non-autoregressive machine translation, improving performance.
problem Challenges in training non-autoregressive models due to lack of autoregressive factors and cross entropy loss penalties.
method Proposes aligned cross entropy (AXE) loss function using a differentiable dynamic program for better word order alignment.
result AXE-based training improves performance on major WMT benchmarks and sets a new state of the art for non-autoregressive models.
Efficiently improves non-autoregressive sequence models for better translation performance.
problem Heavy inference latency and inconsistent output sentences in non-autoregressive models.
method Incorporates a structured inference module with an efficient CRF approximation and dynamic transition technique.
result Significantly better translation performance (BLEU score 26.80) compared to previous non-autoregressive models.
NAOMI improves imputation accuracy for long-range sequences.
problem Missing value imputation in spatiotemporal data.
method Non-autoregressive deep generative model exploiting multiresolution structure.
result Significant improvement in imputation accuracy (60% reduction in average prediction error).
Paper improves NART models for machine translation with faster inference.
problem Inference latency in NART models is high and translation accuracy is inferior.
method Leverages hints from hidden states and word alignments to train NART models.
result Significant improvement in translation accuracy and comparable inference speed to ART baselines.
Proposes a non-autoregressive Transformer for time series forecasting.
problem Autoregressive errors and spatial-temporal dependencies in time series forecasting.
method Introduces a Non-Autoregressive Transformer with a learned temporal influence map.
result Demonstrates state-of-the-art performance on time series forecasting datasets.
This work improves NAT translation accuracy through fine-tuning with curriculum learning.
problem Improving NAT translation accuracy while maintaining inference speed.
method Curriculum learning applied to fine-tuning of a pre-trained AT model to a NAT model.
result Significant improvement in translation accuracy (over 1 BLEU score) and speedup in inference.
We propose a conditional non-autoregressive neural sequence model based on iterative refinement. The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task. We extensively evaluate the proposed model on machine…
Non-autoregressive transformer improves speech recognition speed and accuracy.
problem Reducing inference computation cost in speech recognition.
method Two non-autoregressive transformer structures (A-CMLM and A-FMLM) trained with masked tokens and iterative prediction during inference.
result Non-autoregressive transformer can match autoregressive transformer performance with 7x speedup.
Non-autoregressive method speeds up protein folding prediction 23 times.
problem Generating protein sequences with higher order interactions.
method Discrete diffusion conditioned on 3D structure using ProteinMPNN.
result 23 times speed up in inference without performance loss.
Diffusion models generate music sequences without autoregressive loops.
problem Generating music sequences from symbolic data using diffusion models.
method Parameterize discrete symbolic data in continuous latent space, train diffusion model, generate sequences through reverse process.
result Strong unconditional generation and post-hoc conditional infilling compared to autoregressive models.
Parallel decoding improves machine translation efficiency and accuracy.
problem Efficiently generating translations from left to right.
method Conditional masked language modeling for parallel decoding.
result Improves translation performance by over 4 BLEU points.
WaveGrad generates high-fidelity audio using gradient estimation.
problem Generating high-fidelity audio efficiently.
method Conditional model using score matching and diffusion models, iteratively refining a Gaussian white noise signal.
result WaveGrad can generate high-fidelity audio samples using as few as six iterations.
Paper tackles NAT translation issues with auxiliary regularization.
problem Improves NAT translation quality by addressing repeated and incomplete translations.
method Improves decoder hidden representations via two auxiliary regularization terms.
result Significant improvement in NAT model accuracy with better inference efficiency.
Flow++ improves flow-based models by dequantizing with variational methods and using expressive architectures.
problem Flow-based models have poor density estimation compared to autoregressive models.
method Variational dequantization, expressive affine flows, and improved architecture design.
result Flow++ is now the state-of-the-art non-autoregressive model for unconditional density estimation.
EventFlow forecasts event sequences without autoregression, improving accuracy.
problem Forecasting errors in autoregressive models for event sequences.
method EventFlow uses flow matching to learn joint distributions over event times directly.
result EventFlow reduces forecast error by 20%-53% compared to baselines.
A new model uses normalizing flows for discrete sequences, improving generation speed.
problem Modeling discrete sequences like text using normalizing flows poses challenges.
method Proposes a VAE-based model with autoregressive and non-autoregressive flow architectures.
result Flow-based models can match or improve on autoregressive baselines for discrete sequence tasks.
New method for data assimilation using score-based models.
problem Bayesian inverse problem of identifying plausible state trajectories.
method Score-based data assimilation, learning a score-based generative model of state trajectories.
result Effective method for zero-shot observation scenarios.
This paper proposes an EM approach to reduce inference latency in NAR sequence generation.
problem High inference latency in NAR models due to multi-modality in sequence generation.
method A unified EM framework that jointly optimizes AR and NAR models, with iterative refinement.
result The proposed approach achieves competitive performance with existing NAR models and significantly reduces inference latency.
ElasTST improves time-series forecasting across varying horizons.
problem Robust forecasting across different time horizons in varied industrial sectors.
method Elastic Time-Series Transformer (ElasTST) with non-autoregressive design, rotary position embedding, and multi-scale patching.
result ElasTST provides robust forecasts across varying horizons without retraining.
Survey of graph learning methods for combinatorial optimization problems.
problem Efficient and effective analysis of graphs for combinatorial optimization problems.
method Two-stage framework: Graph representation learning followed by machine learning.
result Recent studies have shown promise in using machine learning to solve graph-based combinatorial optimization problems.
This paper compares log-likelihood and BLEU scores for sequence generation tasks.
problem The discrepancy between density estimation and sequence generation performance.
method Comparing several density estimators on five machine translation tasks.
result The correlation between log-likelihood and BLEU varies depending on model families.
Autoregressive feedback is considered a necessity for successful unconditional text generation using stochastic sequence models. However, such feedback is known to introduce systematic biases into the training process and it obscures a principle of generation: committing to global information and forgetting local nuanc…
Paper combines latent state space with CRF for improved autoregressive text generation.
problem Autoregressive models expose hidden state trajectory to biases.
method Combines latent state space model with CRF observation model.
result Improved performance on unconditional sentence generation compared to RNN and GAN baselines.
This work proposes an efficient autoregressive model for text generation.
problem The challenge of generating high-quality text with autoregressive models.
method Introduces a cascaded decoding approach using Markov transformers to achieve sub-linear parallel time generation.
result Shows competitive accuracy/speed tradeoff compared to existing methods on five machine translation datasets.
DiffWave generates high-fidelity audio waveforms efficiently.
problem Conditional and unconditional audio waveform generation.
method Non-autoregressive diffusion model using Markov chain synthesis.
result DiffWave produces high-quality audios in various tasks.
HCNAF models complex conditional distributions for probabilistic occupancy forecasting.
problem Modeling complex conditional probability density functions for occupancy forecasting.
method Hyper-Conditioned Neural Autoregressive Flow (HCNAF) combining AF and hyper-network.
result HCNAF achieves state-of-the-art performance in self-driving datasets.
Paper proposes energy objective for training normalizing flows without determinants.
problem Challenges in training normalizing flows due to Jacobian determinants.
method Introduces energy objective based on proper scoring rules, determinant-free.
result Energy objective supports novel model families and competitive performance.
Discrete flows extend normalizing flows to discrete data, improving various applications.
problem Applying normalizing flows to discrete data distributions.
method Developed discrete autoregressive and bipartite flows, showing their effectiveness on various discrete data tasks.
result Discrete autoregressive flows outperform autoregressive baselines on synthetic discrete distributions and Potts models.
Deep neural networks with discrete latent variables offer the promise of better symbolic reasoning, and learning abstractions that are more useful to new tasks. There has been a surge in interest in discrete latent variable models, however, despite several recent improvements, the training of discrete latent variable m…
PixelCNN models can achieve state-of-the-art results on CIFAR-10 with exact likelihood computation.
problem Dequantization gap in modeling discrete data like images.
method Introducing subset flows to allow exact computation of likelihoods for discrete data.
result PixelCNN models trained with exact likelihood computation achieve state-of-the-art results on CIFAR-10.
This work proposes a new method to train models with deep latent hierarchies using Optimal Transport.
problem Training models with deep latent hierarchies using VAEs often leads to the 'latent variable collapse' issue.
method Proposes a novel approach based on Optimal Transport to train models with deep latent hierarchies.
result The method avoids the 'latent variable collapse' issue and provides better sample generations and latent representation.
WaveFit uses fixed-point iteration to create high-quality neural vocoders.
problem Creating high-quality neural vocoders with fast inference.
method Integrates GANs' adversarial training into a DDPM-like iterative framework based on fixed-point iteration.
result WaveFit synthesizes speech with naturalness comparable to human speech, and is significantly faster than existing methods.
Diffusion LLMs can efficiently generate harmful prompts for adversarial testing.
problem Generating harmful prompts for adversarial testing is resource-intensive and costly.
method Transformed adversarial prompt optimization into an efficient inference task using pretrained Diffusion LLMs.
result Only a few conditional samples are required to generate harmful prompts with high reward.
New deep learning method solves TSP faster and more efficiently.
problem Approximately solving the Travelling Salesman Problem on 2D Euclidean graphs.
method Uses Graph Convolutional Networks for efficient TSP graph representations and non-autoregressive beam search.
result Significantly reduces optimality gap for large problem instances.
Enhanced Sampling Scheme improves masked generative modeling.
problem Limitations of existing sampling schemes in masked non-autoregressive generative modeling.
method ESS consists of three stages: Naive Iterative Decoding, Critical Reverse Sampling, and Critical Resampling.
result ESS achieves significant performance gains in unconditional and class-conditional sampling.
Insertion Transformer generates sequences with flexible insertion orders.
problem Generating sequences with arbitrary orderings.
method Iterative, partially autoregressive model with insertion operations.
result Outperforms non-autoregressive approaches in translation tasks.
Unified approach for generating sequences from undirected models.
problem Generating sequences directly from undirected models like BERT.
method Generalized model of sequence generation unifying decoding in directed and undirected models.
result Adapted decoding algorithms for undirected models achieve competitive results.
NVAE improves VAE performance on large image datasets.
problem Improving variational autoencoder performance for large image datasets.
method Deep hierarchical VAE with depth-wise separable convolutions and batch normalization, residual parameterization of Normal distributions, and spectral regularization.
result NVAE achieves state-of-the-art results on MNIST, CIFAR-10, CelebA 64, and CelebA HQ datasets.
FPETS speeds up TTS by 600X and reduces errors.
problem High latency and errors in end-to-end TTS systems.
method Non-autoregressive, fully parallel approach with UFANS and trainable position encoding.
result Significant speed up and better quality audios with fewer errors.
This research explores discrete diffusion models for natural language generation.
problem Challenges in applying diffusion models to discrete data, especially natural language.
method Investigates Discrete Denoising Diffusion Probabilistic Model (D3PM) and compares it with autoregressive models.
result Discrete diffusion models achieve better processing speed than autoregressive models.
Latent Block-Diffusion Temporal Point Processes (LBDTPP) is a semi-autoregressive framework for generating asynchronous event sequences.
problem Generating asynchronous event sequences
method Latent Block-Diffusion Temporal Point Processes
result Outperforms state-of-the-art TPP baselines in both unconditional and conditional generation tasks
Unified theory for semi-implicit variational inference, bridging approximation and optimization.
problem Developing a statistical theory for semi-implicit variational inference.
method Unified theory combining approximation and optimization analyses.
result Unified theory characterizes SIVI's ability to recover target distributions and governs asymptotic behavior.
Study develops time-continuous models and probabilistic descriptions for agent-based economic market models.
problem Formulating and describing agent-based economic market models in a time-continuous and probabilistic manner.
method Derived time-continuous formulations, discussed impact of time-scaling, proved stability, presented probabilistic descriptions using kinetic theory.
result Time-continuous formulations and probabilistic descriptions for agent-based economic market models.
Hybrid model combines interpretable and black-box models for better transparency and performance.
problem Balancing interpretability and predictive performance in machine learning models.
method Proposes a Hybrid Predictive Model (HPM) integrating an interpretable model with a black-box model, using principled objective functions and customized training algorithms.
result Hybrid models achieve an efficient trade-off between transparency and predictive performance.
Boosts generative models by combining multiple meta-models.
problem Challenges in creating a single generative model that accurately represents complex data.
method Cascades multiple meta-models (like RBM and VAE) to create a stronger generative model.
result Derives a decomposable variational lower bound for training and evaluating the boosted model.
The paper introduces BCART models for aggregate claim amount, improving frequency-severity and joint modeling.
problem Modeling aggregate claim amount with frequency-severity and joint dependencies.
method Developed three types of BCART models: frequency-severity, sequential, and joint models. Used various distributions for claim severity data.
result Weibull distribution outperforms gamma and lognormal for right-skewed, heavy-tailed claim severity data.
The paper uses model-based trees to create interpretable surrogate models for complex machine learning models.
problem Interpreting complex machine learning models.
method Using model-based trees to partition feature space and create interpretable models.
result Model-based trees generate optimal surrogate models that balance interpretability and performance.
Gauge Flow Models use a learnable Gauge Field in Generative Flow Models.
problem Improving generative model performance.
method Integrates a learnable Gauge Field into Flow ODEs.
result Gauge Flow Models outperform traditional Flow Models in Flow Matching experiments.