Cactus improves auto-regressive decoding speed without sacrificing quality.
problem Accelerating auto-regressive decoding while maintaining output quality.
method Formalizes speculative sampling as constrained optimization and proposes Cactus for controlled divergence from the verifier distribution.
result Empirically validated effectiveness across various benchmarks.
MDMs train to decode tokens in a random order, which affects performance; we show they can be optimized for a favorable order.
problem Performance of MDMs is affected by the random order in which tokens are decoded.
method We show that MDMs can be optimized for a favorable decoding order by equipping their continuous-time variational objective with multivariate noise schedules.
result MDMs can be decomposed into a weighted auto-regressive losses over orders, making them auto-regressive models with learnable orders.
Improved text generation with constraints using discrete auto-regressive biasing.
problem Balancing fluency and constraint satisfaction in LLM outputs.
method Discrete Auto-regressive Biasing, leveraging gradients in discrete text space.
result Significantly improved constraint satisfaction with comparable fluency.
Transformer adapts to graphs with adaptive attention and auto-regressive decoding.
problem Transformers struggle with graph data due to non-sequential nature.
method Proposes GRAT, a Transformer variant with adaptive attention and auto-regressive decoding.
result GRAT achieves state-of-the-art performance on molecule property predictions and generation tasks.
Variational autoencoders (VAEs) with an auto-regressive decoder have been applied for many natural language processing (NLP) tasks. The VAE objective consists of two terms, (i) reconstruction and (ii) KL regularization, balanced by a weighting hyper-parameter β. One notorious training difficulty is that the KL term ten…
Auto-regressive models improve smoothing efficiency with exponentially tapered windows.
problem Improving time-series smoothing efficiency.
method An auto-regressive formulation for time-series smoothing.
result Auto-regressive models result in moving means with exponentially tapered windows.
In recent years, deep learning has proven to be a viable methodology for surrogate modeling and uncertainty quantification for a vast number of physical systems. However, in their traditional form, such models can require a large amount of training data. This is of particular importance for various engineering and scie…
Proof of Gaussian ML estimator consistency in linear auto-regressive models.
problem Consistency of Gaussian maximum likelihood estimator in linear auto-regressive models.
method Information-theoretic proof without stability assumptions.
result Nearly optimal non-asymptotic rates for parameter recovery.
CAFLOW uses auto-regressive flows to translate images efficiently.
problem Image-to-image translation tasks.
method Transforms conditioning image into latent encodings using normalizing flows, models conditional distribution with auto-regressive distributions.
result Outperforms former conditional flow designs.
Auto-regressive models learn latent states from partially observed linear dynamical systems.
problem Understanding how auto-regressive models learn latent representations from partially observed linear dynamical systems.
method Empirical risk minimization on partially observed linear dynamical systems.
result Two-layer linear auto-regressive models learn to approximate Kalman filtering, coinciding with optimal state estimates.
VEST automates feature engineering for time series forecasting.
problem Challenges in time series forecasting with improved performance.
method VEST combines auto-regression with statistical summarization of recent past dynamics.
result VEST significantly improves forecasting performance.
In this paper, non-linear time series models are used to describe volatility in financial time series data. To describe volatility, two of the non-linear time series are combined into form TAR (Threshold Auto-Regressive Model) with AARCH (Asymmetric Auto-Regressive Conditional Heteroskedasticity) error term and its par…
Paper reduces vocabulary losslessly for language model cooperation.
problem Language models struggle to cooperate with different tokenizations.
method Established a theoretical framework for lossless vocabulary reduction.
result Efficiently converts models with different tokenizations to cooperate with maximal common vocabulary.
TraDE uses self-attention for better density estimation of tabular and image data.
problem Improving density estimation for tabular and image data.
method Self-attention-based architecture trained with a penalized maximum likelihood objective.
result TraDE produces significantly better density estimates than existing methods.
The paper proposes a new auto-regressive model for multivariate distributional time series.
problem Statistical analysis of multivariate time series of probability measures.
method Wasserstein space, auto-regressive model, iterated random function systems.
result Consistent estimator for auto-regressive coefficients with sparse structure.
Proposes a VAE with a discrete bottleneck for better text generation.
problem VAEs struggle with latent variable auto-regressive decoding in text generation.
method Introduces a discretized bottleneck to enforce latent feature matching in a compact space.
result Demonstrates improved text generation capabilities across various tasks.
DSARF models complex spatio-temporal data with deep switching auto-regressive factors.
problem Forecasting complex spatio-temporal data with recurring patterns.
method Deep switching auto-regressive factorization (DSARF) with stochastic variational inference.
result DSARF outperforms state-of-the-art methods in long- and short-term prediction accuracy.
VAR-GPs solve continual learning by updating posteriors sequentially.
problem Catastrophic forgetting in sequential learning tasks.
method Sparse inducing point approximations and auto-regressive variational distribution.
result VAR-GPs prevent catastrophic forgetting and outperform baselines.
BMBO-DARN optimizes expensive functions with varying fidelities.
problem Optimizing expensive, multi-fidelity functions efficiently.
method Batch Multi-fidelity Bayesian Optimization with Deep Auto-Regressive Networks.
result BMBO-DARN improves surrogate learning and optimization performance.
A new method estimates expectations from subtractive mixture models without sampling.
problem Estimating expectations from multimodal distributions using SMMs.
method Difference representation of SMMs to create unbiased IS estimator (ΔextEx). result Demonstrates that ΔextEx can achieve comparable estimation quality to auto-regressive sampling but is faster. Auto-regressive diffusion models improve capturing conditional dependence in data.
problem Vanilla diffusion models struggle to capture important, high-level relationships in real-world data.
method Developed auto-regressive diffusion models to better capture conditional dependence structures.
result AR diffusion models produce samples with a reduced gap in approximating the data conditional distribution.
Machine learning maps knots to embeddings, revealing topological invariants.
problem Learning topological invariance in knot theory.
method Contrastive and generative machine learning techniques, auto-regressive decoder Transformer network.
result Neural networks can map different knots to the same point in an embedding vector space.
A new method improves text generation quality and diversity.
problem Exposure bias in Maximum Likelihood Estimation for text generation.
method ψ-MLE, a new training scheme based on density ratio estimation.
result ψ-MLE outperforms Maximum Likelihood Estimation and other models in text generation quality and diversity.
Hybrid model combines VAR and neural network for OFI prediction.
problem Accurate prediction of Order Flow Imbalance (OFI) in high frequency trading.
method Combines Vector Auto Regression (VAR) and a simple feedforward neural network (FNN).
result Hybrid model achieves superior predictive accuracy compared to standalone models.
We introduce an auto-regressive model which captures the growing nature of realistic markets. In our model agents do not trade with other agents, they interact indirectly only through a market. Change of their wealth depends, linearly on how much they invest, and stochastically on how much they gain from the noisy mark…
Numerous empirical proofs indicate the adequacy of the time discrete auto-regressive stochastic volatility models introduced by Taylor in the description of the log-returns of financial assets. The pricing and hedging of contingent products that use these models for their underlying assets is a non-trivial exercise due…
Improved language models by incorporating statistical discriminators.
problem Distinguishing model-generated text from real text reliably.
method Energy-Based Model framework to incorporate discriminators.
result Improves language model performance in perplexity and human evaluation.
Enhanced multi-fidelity models improve digital twin accuracy and uncertainty quantification.
problem Lack of detailed application-specific data and inaccurate sensor data hinder surrogate model learning for digital twins.
method Proposes a multi-fidelity surrogate model framework integrating PCFE and GP, and deep-HPCFE with auto-regression schemes.
result Demonstrates improved accuracy and uncertainty quantification in digital twin systems.
BF-VAE estimates uncertainty from LF and HF QoI samples.
problem Balancing computational efficiency and numerical accuracy in uncertainty quantification.
method Bi-fidelity formulation of VAEs in latent space.
result BF-VAE improves accuracy with limited HF data.
New approach predicts tokens in context, explaining how ICL emerges.
problem Limited understanding of in-context learning emergence.
method Auto-regressive next-token prediction (AR-NTP) with prompt token-dependency and a two-level expectation.
result ICL emerges from the generalization of sequences and topics.
We propose a generic confidence-based approximation that can be plugged in and simplify the auto-regressive generation process with a proved convergence. We first assume that the priors of future samples can be generated in an independently and identically distributed (i.i.d.) manner using an efficient predictor. Given…
Generative models speed up complex system simulations.
problem Accurately forecasting the dynamics of complex systems at reduced cost.
method Generative Learning of Effective Dynamics (G-LED) using auto-regressive attention and Bayesian diffusion models.
result Generative models can accurately forecast complex system dynamics at lower computational cost.
As a technology to read brain states from measurable brain activities, brain decoding are widely applied in industries and medical sciences. In spite of high demands in these applications for a universal decoder that can be applied to all individuals simultaneously, large variation in brain activities across individual…
Novel low-rank neural decoder improves μ-ECoG neural decoding.
problem Challenging neural decoding from high-dimensional μ-ECoG data. method Low-rank structure in neural network decoder.
result Low-rank decoder outperforms standard PCA.
A novel deep probabilistic model for dynamic systems forecasting.
problem Probabilistic forecasting in dynamic systems.
method Combining deep generative models and state space models with recurrent neural networks and variational sequence models.
result Outperforms existing models in system identification benchmarks and real-world centrifugal compressor forecasting.
The paper introduces FMCI and hybrid decoding for hidden Markov models.
problem Computing distributions and decoding hidden state sequences in HMMs.
method Finite Markov chain imbedding (FMCI) and hybrid decoding.
result Hybrid decoding improves performance over traditional methods.
Study examines how decoding algorithms affect fairness in language generation models.
problem Impact of decoding algorithms on fairness in open-ended language generation.
method Systematic analysis of top-p, top-k, and temperature decoding algorithms. result Decoding algorithms significantly impact fairness across demographic groups.
We present RL-VAE, a graph-to-graph variational autoencoder that uses reinforcement learning to decode molecular graphs from latent embeddings. Methods have been described previously for graph-to-graph autoencoding, but these approaches require sophisticated decoders that increase the complexity of training and evaluat…
In this paper, we use reinforcement learning to find effective decoding strategies for binary linear codes. We start by reviewing several iterative decoding algorithms that involve a decision-making process at each step, including bit-flipping (BF) decoding, residual belief propagation, and anchor decoding. We then ill…
This paper analyzes speculative decoding, a method to speed up large language model inferences.
problem Theoretical understanding of speculative decoding is lacking.
method Conceptualizes speculative decoding as a markov chain problem and studies its key properties.
result Reveals fundamental connections between LLM components and their impact on decoding efficiency.
Finding optimal correction of errors in generic stabilizer codes is a computationally hard problem, even for simple noise models. While this task can be simplified for codes with some structure, such as topological stabilizer codes, developing good and efficient decoders still remains a challenge. In our work, we syste…
A new RG approach connects discrete and continuous time descriptions of Gaussian processes.
problem Discretization of continuous stochastic processes for accurate simulation or model inference.
method Renormalization Group (RG) approach for Gaussian time series generated by auto-regressive models.
result RG fixed points correspond to discretizations of linear SDEs, providing insights into process accuracy.
We present a new classification method for quasar identification in the EROS-2 and MACHO datasets based on a boosted version of Random Forest classifier. We use a set of variability features including parameters of a continuous auto regressive model. We prove that continuous auto regressive parameters are very importan…
Improving the interpretability of brain decoding approaches is of primary interest in many neuroimaging studies. Despite extensive studies of this type, at present, there is no formal definition for interpretability of brain decoding models. As a consequence, there is no quantitative measure for evaluating the interpre…
Study error bounds and optimal schedules for Masked Diffusions with factorized approximations.
problem Analyzing trade-offs between computation and accuracy in Masked Diffusion Models.
method Provided general error bounds and identified optimal schedules based on data distribution information profiles.
result Identified optimal schedule sizes for Masked Diffusion Models.
With latent variables, stochastic recurrent models have achieved state-of-the-art performance in modeling sound-wave sequence. However, opposite results are also observed in other domains, where standard recurrent networks often outperform stochastic models. To better understand this discrepancy, we re-examine the role…
Recent developments in the field of deep learning have motivated many researchers to apply these methods to problems in quantum information. Torlai and Melko first proposed a decoder for surface codes based on neural networks. Since then, many other researchers have applied neural networks to study a variety of problem…
Breaks down complex nonlinear dynamics into simpler components.
problem Control of nonlinear dynamical systems remains challenging.
method Inspired by hybrid switching systems, decomposes dynamics into simpler stochastic switching linear dynamical systems.
result Extracts hierarchies of Markovian and auto-regressive locally linear controllers from nonlinear experts.