Transformers can simulate MLE for Bayesian network sequences.
problem Understanding transformers' capabilities in Bayesian network sequence generation.
method In-context maximum likelihood estimation (MLE) for autoregressive sequence generation.
result A simple transformer model can estimate Bayesian network probabilities and generate new samples.
Planar neural networks learn image transformations from sequences.
problem Learning image transformations for mental simulation.
method Using planar neural networks, the study investigates various factors affecting the learning of image transformations.
result The approach can effectively learn and transfer image transformations, including translation, rotation, and scaling.
Transformers can outperform feedforward and recurrent networks due to dynamic sparsity.
problem Understanding when and why Transformers outperform other neural network architectures.
method Analyzing a sequence-to-sequence data generating model with dynamic sparsity, proving sample complexity differences between feedforward, recurrent, and Transformers.
result Transformers can learn dynamic sparsity models with lower sample complexity than feedforward and recurrent networks.
This paper studies mutual information in sequence models and finds Transformers excel in capturing long-range dependencies.
problem Understanding the expressive power of sequence models in capturing temporal dependencies.
method Theoretical and empirical analysis of linear and nonlinear RNNs, including Transformers.
result Transformers can capture long-range mutual information more efficiently than RNNs.
Sparse Transformers reduce memory and time requirements for long sequence modeling.
problem Quadratic growth in memory and time with transformer sequence length.
method Sparse factorizations of attention matrix, deeper network training, attention matrix recomputation, fast attention kernels.
result Sparse Transformers can model sequences up to tens of thousands of timesteps with hundreds of layers.
Transformer networks approximate Hölder and Sobolev functions with fixed-depth networks.
problem Nonparametric regression with dependent observations.
method Established novel upper bounds for Transformer networks approximating Hölder and Sobolev functions under various β-mixing data assumptions. result Explicit convergence rates for nonparametric regression problems under β-mixing data assumptions. IGLOO slices features space to process long sequences efficiently.
problem Efficiency in processing long sequences with neural networks.
method Slicing features space into non-local patches and using their relationships to build sequence representations.
result IGLOO can handle dependencies up to 20,000 steps efficiently.
Improves sequence modeling with a flow-based recurrent mixture density network.
problem Sequence modeling and sequence-to-sequence mapping applications.
method Generalized recurrent mixture density networks using normalized flow transformations.
result Significantly improved fit to image sequences measured by log-likelihood.
Proposes a Complex Transformer for complex-valued sequence modeling.
problem Lack of deep learning models for complex-valued data.
method Develops a Complex Transformer using transformer backbone with specialized attention and encoder-decoder networks.
result Achieves state-of-the-art performance on complex-valued datasets.
Linear attention model outperforms full attention in language tasks.
problem Quadratic scaling of full attention limits model size.
method Introduced a linear attention mechanism.
result Linear attention models achieve comparable performance to full attention models.
Sequence Transformer Networks improve mortality prediction in clinical time-series data.
problem Predicting in-hospital mortality from clinical time-series data.
method End-to-end trainable Sequence Transformer Networks that learn temporal and scaling invariances.
result Improves AUROC from 0.838 to 0.851 compared to a baseline CNN.
Single-layer Transformer can approximate any sequence mapping.
problem Lack of theoretical understanding of Transformers.
method Review of linear algebra, probability, and optimization concepts; detailed analysis of Transformer architecture.
result A single-layer Transformer can approximate any continuous sequence-to-sequence mapping to arbitrary precision.
Universal Transformer improves sequence modeling accuracy.
problem Feed-forward sequence models fail on simple tasks.
method Parallel-in-time self-attentive recurrent model combining feed-forward and RNN properties.
result UTs outperform standard Transformers on various tasks.
Sparse Transformers can approximate dense Transformers with only O(n) connections.
problem Can sparse Transformers approximate arbitrary sequence-to-sequence functions?
method Proposed sufficient conditions for universal approximation and proved that sparse Transformers with O(n) connections can approximate dense models.
result Sparse Transformers with O(n) connections can approximate the same function class as dense models with n^2 connections.
Paper explores EEG-based speech recognition using transformers, showing faster training and better performance for smaller vocabularies.
problem Continuous speech recognition using EEG features.
method Transformer-based ASR model compared to RNN-based models.
result Transformer models perform better for smaller vocabularies but RNN models outperform them for larger vocabularies.
New analysis shows RPE-based Transformers can't approximate all functions.
problem Understanding the limitations of RPE-based Transformers in approximating continuous functions.
method Mathematical analysis and development of a novel attention module (URPE) to overcome limitations.
result RPE-based Transformers can't approximate all continuous sequence-to-sequence functions, even with depth and width.
TCT learns multimodal sequence representations by translating from related sequences.
problem Challenges in learning semantic representations from multimodalities.
method Transformer based Cross-modal Translator (TCT) combined with Multimodal Transformer Network (MTN).
result Proposed method achieves new state-of-the-art performance on video-grounded dialogue.
Compact Recurrent Transformer (CRT) improves Transformer efficiency for long sequences.
problem Efficiently scaling Transformer architecture to long sequences with limited compute resources.
method Combines shallow Transformer models with recurrent neural networks and persistent memory.
result CRT achieves comparable or superior performance to full-length Transformers with shorter segments and reduced FLOPs.
Transformers handle infinite dimensional inputs effectively by feature extraction and dynamic feature selection.
problem Understanding the approximation and estimation ability of Transformers with infinite dimensional inputs.
method Anisotropic smoothness analysis and feature extraction properties of Transformers.
result Transformers avoid the curse of dimensionality and dynamically select important features.
Researchers prove Transformers are Turing-complete and analyze their components.
problem Understanding the computational power and limitations of Transformers.
method Analyzed Turing-completeness of vanilla and modified Transformers, and necessity of components.
result Transformers with positional masking and positional encodings are Turing-complete.
Many-to-Many VTN improves voice conversion across multiple speakers.
problem Voice conversion across multiple speakers.
method Sequence-to-sequence learning framework with many-to-many VTN architecture.
result Improved sound quality and speaker similarity compared to baseline methods.
A new transformer model uses Monte Carlo methods for sequence prediction.
problem Predicting sequences from observations with uncertainty.
method Integrates Monte Carlo methods into a transformer architecture to model stochastic sequences.
result Proposes a generative model with predictive distributions.
Transformers become faster by linearizing self-attention.
problem Quadratic complexity of transformers makes them slow for long sequences.
method Expressed self-attention as a linear dot-product and used matrix product associativity to reduce complexity.
result Linear transformers are up to 4000x faster on long sequences.
Many machine learning tasks can be expressed as the transformation---or \emph{transduction}---of input sequences into output sequences: speech recognition, machine translation, protein secondary structure prediction and text-to-speech to name but a few. One of the key challenges in sequence transduction is learning to …
Paper learns dynamic generator models for video sequences.
problem Modeling spatial-temporal processes like dynamic textures and actions.
method Alternating back-propagation through time algorithm to learn latent state vectors and generator model.
result Trains realistic models for dynamic textures and actions.
Spacetimeformer learns spatiotemporal relationships from data alone.
problem Forecasting multivariate time series with distinct spatial relationships.
method Transformers with dynamic graph connections learning interactions between space, time, and value.
result Competitive results on various time series prediction benchmarks.
Neural network iteratively refines image registration, achieving compactness and speed.
problem Non-compact representation of deformations in image registration.
method Recurrent registration neural network that computes local deformations iteratively.
result Our method achieves similar accuracy but is more compact and faster.
Auto-sizing improves machine translation efficiency by 3.9 BLEU points with fewer parameters.
problem Optimizing neural network architecture and hyperparameters for low-resource machine translation is computationally expensive.
method Auto-sizing uses regularization to dynamically adjust network size during training.
result Auto-sizing improves BLEU scores by up to 3.9 points with one-third fewer parameters.
This work makes neural sequence models more efficient by controlling computation.
problem Fixed compute for all examples in neural networks.
method Conditional computation to adapt compute to example complexity.
result Conditional Computation Transformer (CCT) improves efficiency and performance.
Transformers can approximate any sequence-to-sequence function, surprising given their complexity.
problem Understanding the expressive power of Transformer models for sequence-to-sequence functions.
method Established that Transformers are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, and extended this to arbitrary functions using positional encodings.
result Transformers are universal approximators of arbitrary continuous sequence-to-sequence functions on a compact domain.
THP model captures complex dependencies in event sequences efficiently.
problem Inability of existing models to capture long-term dependencies in event sequences.
method Transformer Hawkes Process (THP) model using self-attention mechanism.
result THP outperforms existing models in likelihood and event prediction accuracy.
Graded Transformers embed algebraic structure in neural networks through graded transformations.
problem Efficiently modeling hierarchical and structured data in neural networks.
method Introduces Linearly Graded Transformer (LGT) and Exponentially Graded Transformer (EGT) with graded scaling operators.
result Establishes rigorous guarantees and improved efficiency for structured data.
FineMorphs models smooth transformations for multivariate regression.
problem Efficiently modeling complex transformations for multivariate regression.
method Optimal control of affine and diffeomorphic transformations using smooth vector fields.
result FineMorphs can reduce dimensionality and adapt to large datasets.
Transformers tend to learn more symmetric functions in sequence data.
problem Understanding inductive bias in Transformers with infinitely over-parameterized models.
method Analyzing Transformers in the Gaussian process limit, using representation theory of the symmetric group.
result Transformers are biased towards more permutation symmetric functions, and this can be quantitatively predicted.
Self-attention prefers sparse functions of input sequences, reducing sample complexity.
problem Understanding the inductive biases of self-attention in modeling long-range dependencies.
method Theoretical analysis and synthetic experiments to probe sample complexity of learning sparse functions with Transformers.
result Bounded-norm Transformer networks can represent sparse functions of the input sequence with logarithmic sample complexity.
B-cos transforms improve neural network interpretability by aligning weights.
problem Improving interpretability of deep neural networks.
method Replacing linear transforms with B-cos transforms that promote weight-input alignment during training.
result B-cos transforms lead to highly interpretable and task-relevant linear summaries of neural network computations.
Novel graph neural network combines random walks with local message passing.
problem Graph neural networks struggle with long-range dependencies.
method Combines random walks with local message passing in a novel architecture.
result Significant performance improvements on graph benchmarks.
New sequences prove some link diagrams can't be transformed by specific moves.
problem Proving some link diagrams can't be transformed by specific sequences of Reidemeister moves.
method Proved the existence of I-generalized ordered sequences to create simple transformations.
result Some link diagrams cannot be transformed by sequences of moves that increase, then decrease crossings.
Transformers' self-attention mechanism is mapped to a generalized Potts model.
problem Uncertainty in what type of data distribution self-attention can efficiently learn.
method Decouple word positions and embeddings, then show self-attention learns a generalized Potts model.
result Training self-attention is equivalent to solving the inverse Potts problem.
Compressive Transformer learns long-range sequences by compressing past memories.
problem Learning long-range sequences in language and speech models.
method Compressive Transformer compresses past memories for efficient long-range sequence learning.
result State-of-the-art performance on language and speech benchmarks.
Generative Link Sequence Modeling predicts future links in evolving networks.
problem Predicting future links in networks with evolving structures.
method Sequence modeling framework with self-tokenization to capture temporal link formation patterns.
result GLSM achieves best performance on AUC metrics compared to existing methods.
Softmax attention approximates complex functions and subsumes many known universal approximators.
problem Universal approximation of continuous sequence-to-sequence functions.
method Interpolation-based analysis of attention's internal mechanism, showing its ability to approximate ReLU functions.
result Softmax attention is a universal approximator for continuous sequence-to-sequence functions.
Recurrent neural networks (RNNs) have been drawing much attention with great success in many applications like speech recognition and neural machine translation. Long short-term memory (LSTM) is one of the most popular RNN units in deep learning applications. LSTM transforms the input and the previous hidden states to …
Study on convergence of transformed metric spaces as dimensions grow.
problem Conditions for convergence of transformed metric spaces.
method Clarifying conditions for convergence of transformed spaces from original sequence and vice versa.
result Spheres and projective spaces converge to Gaussian space and its quotient as dimensions increase.
Enhanced transformer converts whispered speech to natural speech.
problem Machine recognition of whispered speech is challenging.
method Proposes an enhanced transformer architecture trained end-to-end using supervised learning.
result Similar formant distributions of converted speech to groundtruth.
Improved multi-step prediction of drivable space for autonomous vehicles.
problem Accurate prediction of drivable space for safer, more comfortable navigation.
method Recurrent Neural Network (RNN) architectures trained on KITTI dataset, incorporating motion features.
result Significant improvement in prediction accuracy over state-of-the-art methods.
Neural networks struggle with long sequences, but a new method improves their performance.
problem Neural networks struggle to generalize to longer sequences and unseen data.
method Proposed a learned conditional masking mechanism and binary encoding for numbers.
result Models can now generalize far outside their training range with near-perfect accuracy.
The paper offers generalization bounds for Transformers that ignore sequence length.
problem Developing generalization bounds for Transformers that are independent of sequence length.
method Covering number approach to upper bound Rademacher complexity of bounded linear transformations.
result Theoretical bounds for Transformer generalization are independent of sequence length.