Task-agnostic data augmentation shows little benefit for pretrained transformers.
problem Evaluating the effectiveness of task-agnostic data augmentation on pretrained transformers.
method Conducted a systematic examination of two data augmentation techniques (Easy Data Augmentation and Back-Translation) across 5 tasks, 6 datasets, and 3 pretrained transformer models.
result Data augmentation techniques previously effective for non-pretrained models fail to consistently improve performance for pretrained transformers, even with limited training data.
Transformers learn to make decisions in new contexts from offline data.
problem Understanding when and how transformers can perform in-context reinforcement learning.
method Theoretical framework analyzing supervised pretraining for ICRL, including algorithm distillation and decision-pretrained transformers.
result Transformers can efficiently approximate optimal reinforcement learning algorithms for various environments.
Transformers can learn new tasks from diverse pretraining data but struggle with out-of-domain tasks.
problem Transformer models' ability to learn new tasks in-context is limited by their pretraining data coverage.
method Investigation of transformer models trained on ( x , f ( x ) ) (x, f(x)) ( x , f ( x )) pairs, comparing in-context learning capabilities across different task families. result Transformers can identify and learn within task families in their pretraining data but fail with out-of-domain tasks.
WeatherFormer learns robust weather features from small datasets.
problem Modeling complex weather dynamics from limited data.
method Pretrained transformer encoder on large satellite dataset, with spatiotemporal encoding.
result State-of-the-art performance in county-level soybean yield prediction and influenza forecasting.
This paper enhances language models with knowledge awareness.
problem Understanding how much knowledge pretrained language models grasp.
method Inserting explicit knowledge layers into pretraining without changing transformer architecture.
result Significantly more knowledge packed into transformer parameters.
Transformers learn to generalize out-of-distribution with diverse pretraining tasks.
problem Conditions for pretrained transformers to generalize out-of-distribution.
method Empirical study of task diversity and pretraining distribution.
result As task diversity increases, transformers transition from specialized to generalized solutions.
Transformers fine-tuned on synthetic data boost tabular data classification performance.
problem Improving tabular data classification accuracy.
method Fine-tuning ICL-transformers on synthetic datasets with complex decision boundaries.
result Fine-tuned ICL-transformers outperform regular neural networks on real-world datasets.
Transformers learn to adapt to different task difficulties and resist distribution shifts.
problem Understanding and optimizing a Transformer's performance across various task difficulties and distribution shifts.
method Analyzing a pretrained Transformer on a mixture distribution of tasks, proving optimal convergence rates.
result Transformers achieve optimal convergence rates on tasks of specific difficulty levels, robust to distribution shifts.
Transformer pretraining yields strong EB performance without explicit adaptation.
problem Empirical Bayes problems with unknown test distributions.
method Indirect analysis of pretrained transformer's performance under universal priors.
result Near-optimal regret bound of O ~ ( 1 n ) \widetilde{O}(\frac{1}{n}) O ( n 1 ) for arbitrary test distributions. MPP trains a transformer to predict multiple physical systems, improving accuracy across various tasks.
problem Training models for specific physical systems is inefficient and requires fine-tuning.
method MPP trains a shared transformer on multiple heterogeneous physical systems, projecting fields into a shared embedding space.
result A single MPP-pretrained transformer outperforms task-specific models on all pretraining sub-tasks and downstream tasks.
Theoretical analysis of data quality and synergies in LLMs.
problem Understanding why different training methods require different amounts of data.
method Theoretical analysis of transformers trained on a weight prediction task for linear regression.
result SFT excels on smaller datasets challenging for the pretrained model, while RL benefits from large, not overly difficult data.
Transformer models can solve complex math problems with less data.
problem Solving complex symbolic mathematics problems with limited data.
method Pretrain transformer models on language translation tasks and fine-tune for symbolic math.
result Pretrained transformer models achieve comparable accuracy to state-of-the-art models with less data.
Transformers learn low-dimensional target functions efficiently in-context.
problem Efficiently learning nonlinear target functions in-context using transformers.
method Nonlinear MLP layer in transformers optimized by gradient descent, focusing on single-index target functions.
result Transformers can learn target functions with low-dimensional structures efficiently in-context.
Transformers can efficiently approximate nonparametric regression with minimal parameters and sequences.
problem Efficiently approximating nonparametric regression functions with transformers.
method Kernel-weighted polynomial basis and gradient descent.
result Achieves minimax optimal rate of convergence with fewer parameters and sequences.
The study provides precise asymptotic theory for in-context learning by Transformers.
problem Understanding the sample complexity, pretraining task diversity, and context length for successful in-context learning.
method An exactly solvable model of linear regression task by linear attention, deriving sharp asymptotics.
result Double-descent learning curve with increasing pretraining examples, phase transition between low and high task diversity regimes.
New study reveals task alignment is key to ICL performance.
problem Understanding how task alignment affects generalization in in-context learning.
method Derived an exact expression for ICL generalization error in high dimensions under task covariance mismatch.
result Identified train-test task alignment as a key determinant of generalization in ICL.
Transformers learn to use induction heads or shortcuts based on data diversity.
problem How data diversity influences the behavior of transformers.
method Gradient-based training of a single-layer transformer on a minimal task.
result Data diversity steers transformers toward induction heads or shortcuts.
Transformer model pretrains on synthetic graphs for AD detection.
problem Limited labeled data and class imbalance in AD diagnosis.
method Diffusion-generated synthetic graphs, Graph Transformers, transfer learning.
result Framework outperforms baselines in AD diagnosis metrics.
Funnel-Transformer reduces computation by compressing sequence data.
problem Redundant token-level representations in language processing.
method Gradually compresses sequence of hidden states to a shorter one.
result Funnel-Transformer outperforms standard Transformer with fewer FLOPs.
Attention temperature improves robustness of ICL in high-dimensional settings.
problem ICL robustness failure under distribution shift in high dimensions.
method Analyzed a Transformer with approximate softmax attention, derived a closed-form error expression, and showed optimal temperature minimizes error.
result Optimal attention temperature minimizes ICL generalization error under distribution shift.
Paper studies how few pretraining tasks are needed for a linear model to solve new tasks.
problem How many pretraining tasks are needed for a linear model to solve new tasks?
method Pretrained a linear attention model for linear regression with a Gaussian prior.
result Effective pretraining requires a small number of independent tasks, and the model closely matches Bayes optimal.
New study finds best language model architecture and pretraining objective for zero-shot tasks.
problem Evaluating which language model architectures and pretraining objectives best enable zero-shot generalization.
method Compared three model architectures and two pretraining objectives across 170 billion tokens, with and without finetuning.
result Causal decoder-only models trained on autoregressive language modeling exhibit strongest zero-shot generalization.
Transformers learn new tasks from few examples via optimal approximation.
problem Learning new tasks from limited examples using large language models.
method Developed approximation and generalization error bounds for transformers trained on nonparametric regression tasks.
result Transformers achieve minimax optimal estimation risk in context.
Adversarially trained transformers can learn robustly across tasks with minimal tuning.
problem Adversarial attacks and the high cost of adversarial training.
method Adversarial pretraining followed by in-context learning.
result Single-layer linear transformers can generalize robustly to unseen tasks.
A new method uses a frozen language model to improve sample efficiency in reinforcement learning.
problem Improving sample efficiency in reinforcement learning with partially observable environments.
method FROZEN Hopfield network and HELM (History Embedding Language Model) method.
result HELM achieves new state-of-the-art results on Minigrid and Procgen environments.
Study shows how specialized attention circuits emerge during transformer training.
problem Understanding the mechanisms of transformer training dynamics at large scales.
method Controlled sparse modular addition task; monitoring token evolution via visual sandbox.
result Specialized attention circuits (clustering heads) naturally emerge during training.
Paper analyzes why deeper layers of ViTs perform worse on out-of-distribution tasks.
problem Performance degradation of intermediate layers in ViTs under distribution shift.
method Extensive linear probing experiments across various benchmarks and fine-grained analysis of transformer modules.
result Probing feedforward network activations yields best performance under significant distribution shift.
Test-time training adapts a pretrained model to each prompt via parameter updates, improving accuracy under pretraining-to-test distribution shifts.
problem Improving accuracy of pretrained models under distribution shifts.
method Explaining TTT behavior through a decision-theoretic lens.
result TTT reduces prediction error when updates are spectrally matched to the prompt's signal-to-noise ratio and aligned with query-relevant eigen-directions.
Random Transformers behave like polynomial models in ICL with asymptotic growth.
problem Understanding in-context learning capabilities of pretrained Transformers.
method Asymptotic analysis of a random Transformer with a fixed first layer and a trained second layer, considering growth in context length, input dimension, hidden dimension, and training parameters.
result The random Transformer's ICL error is equivalent to a finite-degree Hermite polynomial model.
Training-free looped transformers improve model performance without additional training.
problem Improving model performance without additional training or fine-tuning.
method A lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional fine-tuning.
result Our method improves model performance across various model families.
Optimal feature transfer identified through bias-variance analysis.
problem Optimizing feature transfer in transfer learning.
method Simple linear model with fine-grained bias-variance decomposition.
result Optimal pretrained feature transform is naturally sparse.
CERT improves language understanding by contrastively learning sentence-level semantics.
problem Lack of sentence-level semantics in existing pretraining tasks.
method Contrastive self-supervised learning at the sentence level using back-translation augmentations.
result CERT outperforms BERT on 7 out of 11 GLUE benchmark tasks, achieving the same performance as BERT on 2 tasks.
Single neural network predicts ImageNet model parameters for faster training.
problem Training diverse ImageNet models requires significant resources and time.
method Trained a neural network to predict ImageNet model parameters and used them for initialization.
result Models initialized with predicted parameters converge faster and achieve competitive performance.
This study examines how large language models learn in-context and provides insights into their performance and architecture.
problem Understanding how large language models learn in-context and their performance metrics.
method Bayesian model averaging, parameterization, and statistical analysis of transformer architecture.
result Transformer architecture enables in-context learning through attention mechanisms and fine-grained statistical analysis.
TAMO optimizes multiple objectives in-context using transformers.
problem Balancing competing objectives in expensive, black-box problems.
method Fully amortized, transformer-based policy for multi-objective optimization.
result Significant speedup in proposal time with improved Pareto quality.
We consider the extreme multi-label text classification (XMC) problem: given an input text, return the most relevant labels from a large label collection. For example, the input text could be a product description on Amazon.com and the labels could be product categories. XMC is an important yet challenging problem in t…
SUPE combines unlabeled data with RL to efficiently explore tasks.
problem Efficient exploration in reinforcement learning with sparse rewards.
method Extract low-level skills using VAE, pseudo-label unlabeled data, and use as off-policy data for online RL.
result SUPE outperforms prior methods across 42 long-horizon tasks.
Develops methods for training models with few annotations in computer vision tasks.
problem Training models with limited labeled data in computer vision.
method Theoretical, algorithmic, and experimental contributions for Meta-Learning and Semi-Supervised Learning.
result Improves Contrastive Learning and proposes a semi-supervised method for transformer-based object detectors.
New pruning method retains model expressiveness for NLP tasks.
problem Pruning large pretrained transformer models for real-world deployment.
method Mixture Gaussian Prior Pruning (MGPP) algorithm.
result MGPP outperforms existing pruning methods in high sparsity settings.
Transformer models show robustness across domains with domain adversarial training.
problem Domain adaptation from multiple sources with no labeled data.
method Domain adversarial training and mixture of experts.
result Domain adversarial training improves representation but not performance.
Efficiently compress pretrained models using RSI for improved predictive accuracy.
problem Efficiently compressing large pretrained models for practical deployment.
method Randomized subspace iteration (RSI) for low-rank approximation of pretrained models.
result RSI achieves near-optimal approximation quality and outperforms RSVD in predictive accuracy.
The paper proves how Transformers learn from context and generalize well.
problem Understanding how Transformers generalize from diverse tasks.
method Developed a statistical theory for in-context learning, separating risk into Bayes Gap and Posterior Variance.
result The Posterior Variance is task-independent, and the Bayes Gap decreases with more in-context examples.
This work analyzes CoT prompting methods from a statistical estimation perspective.
problem Improving the effectiveness of LLMs in solving multi-step reasoning problems.
method Introducing a multi-step latent variable model to characterize CoT prompting from a statistical estimation viewpoint.
result The CoT estimator is equivalent to a Bayesian estimator when the pretraining dataset is large.
Unified framework for ICL in causal and masked models.
problem Understanding ICL in masked language models and comparing it to causal models.
method Developed a statistical learning framework representing context by empirical measure and predicting using context and query.
result Upper bounds for masked and autoregressive objectives under Wasserstein-type regularity conditions.
Transformers can learn optimal variable selection in group-sparse classification.
problem Understanding how transformers leverage attention to select relevant variables in group-sparse classification.
method Training a one-layer transformer using gradient descent to select variables from one group of input variables.
result A one-layer transformer can correctly leverage the attention mechanism to select variables, disregarding irrelevant ones.
This paper improves test-time adaptation for distribution shifts using confidence maximization and input transformation.
problem Improving deep networks' performance on data shifted from the training distribution.
method Proposes a novel loss function combining confidence maximization and batch-wise entropy maximization with an input transformation module.
result Significantly improves robustness of pretrained networks to corruptions on benchmarks like ImageNet-C.
This study improves sentence embeddings from BERT models.
problem Capturing the underlying meaning of sentences using BERT models.
method Comprehensive review and testing of various sentence embedding extraction and refinement methods.
result Representation-shaping techniques significantly improve sentence embeddings from BERT-based and simple baseline models.
We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure. Our Tensor-Product Transformer (TP-Transformer) sets a new state of the art on the recently-introduced Mathematics Dataset containing 56 categories of free-form math word-pr…