arXiv research

One-layer transformers can't solve induction heads task efficiently.

Introduces Causal Energy Minimization to understand Transformer layers.

Learned data models based on sparsity are widely used in signal processing and imaging applications. A variety of methods for learning synthesis dictionaries, sparsifying transforms, etc., have been proposed in recent years, often imposing useful structures or properties on the models. In this work, we focus on sparsif…

This study analyzes how one-layer transformers learn regular language recognition tasks.

A new model improves CT image quality from low-dose scans.

Transformers learn to perform logistic regression in-context.

Study on SignGD optimization of two-layer transformer on noisy data.

Transformers can approximate Newton's method for logistic regression.

SSMs combined with neural networks match Transformers in dynamic token selection.

Deep learning uses layers of transformations to predict structured data with uncertainty.

Deep Transformed Gaussian Processes extend TGPs with variational inference for scalable multi-layer modeling.

Transformers with multiple layers learn to estimate bigram distributions, while single-layer models often get stuck in unigram local minima.

A method predicts GNS of transformer layers using normalization layer norms.

Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of…

Looped transformers with LN converge to power method for principal component prediction.

Study Transformer layers under cross-entropy training using mean field control.

Signal models based on sparsity, low-rank and other properties have been exploited for image reconstruction from limited and corrupted data in medical imaging and other computational imaging applications. In particular, sparsifying transform models have shown promise in various applications, and offer numerous advantag…

A framework for transformer attention layers derived from SVR.

New invertible transformations improve flow-based generative models.

Single-layer Transformer can approximate any sequence mapping.

Data augmentations are important ingredients in the recipe for training robust neural networks, especially in computer vision. A fundamental question is whether neural network features encode data augmentation transformations. To answer this question, we introduce a systematic approach to investigate which layers of ne…

Transformers show strengths and weaknesses in complexity analysis.

Random Transformers behave like polynomial models in ICL with asymptotic growth.

The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we fir…

New method enforces orthogonality in convolutional layers for improved robustness.

Sparse Transformers can approximate dense Transformers with only O(n) connections.

Transformers can learn noisy linear systems with depth and IID data.

This paper compares Transformers and RNNs in various tasks, showing size differences.

Investigates optimal parameter allocation in Transformers for efficiency and expressivity.

Paper analyzes why deeper layers of ViTs perform worse on out-of-distribution tasks.

Researchers analyze the geometric and statistical properties of transformer model representations.

The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Or…

The paper analyzes the training dynamics of a transformer for next-token prediction.

Transformers can learn spectral methods and perform unsupervised learning.

Novel neural network layer improves long-range interactions in point clouds.

Despite the widespread adoption of Transformer models for NLP tasks, the expressive power of these models is not well-understood. In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, which is quite surp…

Sparse Transformers degrade semantic information first, with early layers encoding more.

Study reveals Transformer's expressive power and mechanisms.

Deep Graph Neural Networks (GNNs) are useful models for graph classification and graph-based regression tasks. In these tasks, graph pooling is a critical ingredient by which GNNs adapt to input graphs of varying size and structure. We propose a new graph pooling operation based on compressive Haar transforms -- HaarPo…

Minimal token perturbations reveal how Transformer models process information.

Study examines infinite limits of transformer dynamics, identifying key parameterizations.

B-cos transformers explain Vision Transformers' decisions.

Wavelets help compress neural networks efficiently.

Linear attention in Transformers can be interpreted as dynamic VAR models.

Paper proposes Multi-Transformer for more accurate stock volatility forecasts.

Visualizing the details of different cellular structures is of great importance to elucidate cellular functions. However, it is challenging to obtain high quality images of different structures directly due to complex cellular environments. Fluorescence staining is a popular technique to label different structures but …

Graph transformers outperform graph convolutions by preserving community information.

Fine-tuning normalization layers can reconstruct smaller networks.