Proving that next-token prediction makes language models generate coherent long documents.
problem Understanding why language models generate coherent documents despite focusing on next-token prediction.
method Proving the power of next-token prediction in learning longer-range structure using Recurrent Neural Networks (RNN).
result Optimizing next-token prediction in RNNs yields a model that closely approximates the training distribution, even for long-range coherence.
Reinforcement learning after next-token prediction aids in learning from diverse sequence lengths.
problem Learning from sequences of varying lengths and complexity.
method Introducing a framework to study reinforcement learning with autoregressive transformers, focusing on next-token prediction and mixture distributions of short and long sequences.
result Reinforcement learning after next-token prediction enables autoregressive transformers to generalize from long sequences, even when they are rare.
This paper examines how different decoding algorithms for LLMs align with various goals.
problem Consistency of decoding algorithms with different goals in LLMs.
method Analysis of greedy, lookahead, random sampling, and temperature-scaled random sampling algorithms.
result Random sampling is consistent with the true probability distribution, but other goals require optimal algorithms for specific probability distributions.
Gradient descent biases linear models in next-token prediction towards data entropy.
problem Optimization bias in next-token prediction models.
method Analysis of gradient descent on linear models with sparse conditional distributions.
result Gradient descent selects parameters that equate token logits differences to log-odds in the data subspace.
Law explains how LLMs learn to predict next tokens.
problem Understanding how LLMs process input data internally.
method Introduced a precise law governing token embeddings in LLMs.
result Each layer equally contributes to next-token prediction accuracy.
A new method improves language model extrapolation without changing training.
problem Improving language model performance on sequences not seen during training.
method Perturbation-based training procedure for language models.
result The method consistently improves out-of-support prediction.
The paper analyzes the training dynamics of a transformer for next-token prediction.
problem Understanding the non-asymptotic performance of transformers in next-token prediction.
method Characterizes training dataset properties, designs a two-stage training algorithm, and analyzes attention gradient properties.
result Trained transformers converge sub-linearly to max-margin solutions and exhibit linear convergence in cross-entropy loss.
This paper shows ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.
problem Understanding the lookahead capabilities of next-token prediction models.
method Unified view of ARMs and EBMs, establishing a bijection and deriving equivalence.
result ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.
Study quantifies how LLMs capture higher-order statistical structure using cumulant expansion.
problem Understanding how LLMs internalize statistical structure during next-token prediction.
method Cumulant-expansion framework treating softmax entropy as perturbation around center distribution.
result Cumulants reveal distinct signatures for mathematical vs. general text prompts, quantifying feature-learning dynamics.
Language models can predict numeric values as strings.
problem Regression tasks with numeric predictions.
method Causal sequence decoding models trained for next-token prediction.
result Decoder-based heads perform as well as standard heads in numeric regression tasks.
The study explores how Transformers predict the next token in a sequence.
problem Understanding the mechanism behind Transformers' autoregressive learning ability.
method Exploring the approximation ability of Transformers for next-token prediction through specific instances and a causal kernel descent method.
result Transformer models can learn context-dependent functions f for next-token prediction based on past and current observations. Unified theory for neural scaling laws in hierarchically compositional data.
problem Understanding neural scaling laws in hierarchically compositional data.
method Probabilistic context-free grammars and power-law distributed production rules.
result Unified learning curve behavior for classification and next-token prediction tasks.
This work explains how linear representations in large language models arise from training objectives and gradient descent.
problem Understanding the origins of linear representations in large language models.
method A latent variable model to abstract and formalize concept dynamics, combined with analysis of the softmax cross-entropy objective and gradient descent.
result Linear representations emerge when learning from data matching the latent variable model, and this simple structure suffices to yield linear representations.
Mathematical framework for language models processes text and predicts next tokens.
problem Understanding and optimizing the performance of large language models.
method Describes encoding, prediction models, learning from data, and deployment of LLMs.
result Demonstrates remarkable empirical successes and provides a platform for further research.
This paper explores how Transformers predict next tokens in autoregressive tasks.
problem Understanding the success of Transformers in autoregressive learning.
method Trained a Transformer on a next-token prediction task, focusing on commuting orthogonal matrices.
result Trained Transformers can be seen as implementing gradient descent for a specific objective function.
Timer-XL predicts multidimensional time series using a unified Transformer approach.
problem Unified time series forecasting across various tasks and contexts.
method Decoder-only Transformers with a universal TimeAttention mechanism and deft position embedding.
result State-of-the-art performance across multiple forecasting benchmarks.
This paper studies transformer learning dynamics and initialization.
problem Understanding how transformers learn Markov chains and the role of initialization.
method First-order Markov chains and single-layer transformers, proving learning dynamics and conditions for convergence.
result Transformer parameters can converge to global or local minima based on initialization and Markovian data properties.
Study reveals AI's spontaneous topic changes in text prediction.
problem AI's inability to spontaneously switch topics like humans.
method Defined topic as Token Priority Graphs (TPGs) and analyzed self-attention models.
result AI can only switch topics if lower-priority tokens outnumber higher-priority ones.
New approach predicts tokens in context, explaining how ICL emerges.
problem Limited understanding of in-context learning emergence.
method Auto-regressive next-token prediction (AR-NTP) with prompt token-dependency and a two-level expectation.
result ICL emerges from the generalization of sequences and topics.
In-Place TTT enhances LLMs with dynamic parameter updates at inference time.
problem Static training limits LLMs from adapting to new information.
method In-Place TTT updates a subset of model parameters (fast weights) at inference time.
result In-Place TTT enables 4B-parameter models to outperform on tasks with up to 128k contexts.
Discrete Flow Maps bypass sequential prediction limits for parallel text generation.
problem Sequential autoregressive prediction limits large language model speed.
method Flow Maps compress generative trajectories into single-step mappings.
result Discrete Flow Maps surpass previous state-of-the-art results in discrete flow modeling.
LoRA-MCL improves language models by generating diverse sentence continuations.
problem Language models struggle with generating diverse, plausible sentence continuations.
method Low-Rank Adaptation combined with Multiple Choice Learning (MCL) to handle ambiguity.
result LoRA-MCL generates high-diversity and relevant outputs in various tasks.
The study compares feed-forward and attention layers in language models.
problem Understanding the role of feed-forward and attention layers in language models.
method Empirical and theoretical analysis in a synthetic setting.
result Feed-forward layers learn simple distributional associations, while attention layers focus on in-context reasoning.
Analysis of ridge regression under concept shift reveals nontrivial effects on generalization performance.
problem Understanding and mitigating the impact of distribution shift in machine learning models.
method Derivation of exact prediction risk expression in the thermodynamic limit for ridge regression under concept shift.
result Reveals a phase transition and nonmonotonic data dependence of test performance under concept shift.
New L2D framework allows deferring specific parts of a sequence prediction to experts.
problem Current L2D methods defer entire predictions, which is not ideal for long sequences.
method Proposes token-level and one-time rejectors to defer specific outputs of a model prediction to experts.
result Granular deferrals achieve better cost-accuracy tradeoffs than whole deferrals.
A new autoregressive model learns the order of graph generation tasks.
problem Generating graphs in a meaningful order when the canonical order is not obvious.
method Introduces a variant of autoregressive models that dynamically decides the autoregressive order based on data.
result Achieves state-of-the-art results on molecular graph generation benchmarks.
Transformer models perform slower than convolutional networks in learning hierarchical language structures.
problem Understanding how neural networks learn hierarchical language structures.
method Theoretical scaling laws and empirical validation of neural network performance.
result Convolutional networks outperform transformers in learning hierarchical language structures.
Noise stability improves understanding of Transformer models.
problem Lack of robustness metrics for real-valued domains and junta-like input dependence in modern LLMs.
method Proposed noise stability as a new metric and developed a practical regularization method.
result Noise stability regularization method accelerates training by 35-75%.
GPT learns a causal world model from token predictions, validated in game sequences.
problem Does GPT implicitly learn a causal world model from token predictions?
method Derived a causal interpretation of GPT's attention mechanism and proposed zero-shot causal structure learning.
result GPT can generate legal next moves with high confidence for sequences with encoded causal structures, but fails for illegal moves.
Estimates proportions of LLM-generated text in mixed documents.
problem Estimating the proportion of text generated by a pre-specified LLM in mixed documents.
method Developed estimators for two observation regimes: full observation and pivotal reduction, and established sample complexity bounds.
result Full observation estimators require fewer samples than pivotal reduction estimators.
Pre-training improves model coverage, crucial for downstream performance.
problem Understanding why pre-training enhances model performance.
method Coverage principle, focusing on next-token prediction and model quality.
result Coverage generalizes faster than cross-entropy, improving downstream performance.
The paper shows how training with synthetic data can lead to model improvement, not degradation, under certain conditions.
problem Model collapse in iterative training on contaminated sources.
method Statistical analysis of iterative training on a mixture of true and synthetic data.
result Training with synthetic data can lead to model improvement, not degradation, under specific conditions.
Language models allocate information storage, not collapsing into uniform representations.
problem Incomplete neural collapse in language model representations.
method Analyzing variance and information sharing across 14 models, proving an information floor.
result Within-class variance is allocated information storage, not collapsed into uniform representations.
Study on estimating Gumbel--Max watermark proportions in edited documents.
problem Estimating the proportion of a document generated from a watermarked LLM.
method Comparison of full observation and pivotal reduction observation regimes; development of estimators and information-theoretic lower bounds.
result Full observation yields a substantially smaller sample complexity compared to pivotal reduction.
New approach uses inverse reinforcement learning to improve language model training.
problem Training large language models using imitation learning methods.
method Developed a new method of inverse reinforcement learning to optimize sequences directly.
result IRL-based fine-tuning leads to better performance and diversity in language generation.
Study shows how transformers classify symbols without naming them, proving a margin-versus-collision criterion.
problem How transformers classify symbols without naming them.
method Logistic classification analysis of transformer-kernel regime, colored collision graph.
result Decomposes learned predictor into ideal template-level classifier and finite-sample perturbation.
Paper develops large time series models using pre-trained transformers.
problem Performance bottlenecks in small models on data-scarce scenarios.
method Large-scale pre-training, unified time series format, GPT-style architecture.
result Generative pre-trained Time Series Transformer (Timer) for diverse tasks.
Compact Recurrent Transformer (CRT) improves Transformer efficiency for long sequences.
problem Efficiently scaling Transformer architecture to long sequences with limited compute resources.
method Combines shallow Transformer models with recurrent neural networks and persistent memory.
result CRT achieves comparable or superior performance to full-length Transformers with shorter segments and reduced FLOPs.
Generative synthetic data can preserve predictive accuracy but distort causal inference.
problem Distortion of average treatment effect estimates in synthetic data.
method Hybrid synthetic-data framework that generates covariates while modeling treatment and outcome mechanisms separately.
result Hybrid synthesis improves causal fidelity compared to fully generative baselines.
New loss functions based on f-divergences improve language model performance.
problem Improving multiclass classification and language modeling performance.
method Constructing new convex loss functions using f-divergences and deriving an operator for computation.
result The α-divergence loss function with α=1.5 performs well across various tasks. LLMs show surprising confidence in their answers, beyond just tokens.
problem LLMs lack meaningful confidence estimates for their responses.
method Semantic calibration test based on local loss optimality and equivalence classes.
result Base LLMs are semantically calibrated across tasks, contrary to expectations.
The paper distinguishes between conditional and marginal processes in language models and discusses conditions for usefulness.
problem The conditional nature of language models trained on observed sequences and the need for marginal text-only processes.
method Distinguishing between full conditional language process, marginal text-only process, and model-induced distribution; analyzing assumptions of stationarity and ergodicity.
result The marginal text-only law is useful only when the observed prefix is an approximately sufficient statistic for the latent circumstances relevant to continuation.
A single algebraic identity unifies information-theoretic variational results.
problem Deriving and generalizing classical information-theoretic variational results
method Proving a single algebraic mixed coincidence identity
result Unified derivation of classical cornerstones of information theory
Transformers learn a mesa-optimizer to implement in-context learning.
problem Understanding the convergence of autoregressive training to a mesa-optimizer.
method Investigated a one-layer linear causal self-attention model autoregressively trained by gradient flow.
result Proved that autoregressive training converges to a gradient descent step for an OLS problem, validating the mesa-optimizer hypothesis.
Randomly initialized transformers show extreme token preferences.
problem Structural biases in randomly initialized transformers.
method Dissection of transformer architecture at initialization.
result Initialization-induced biases persist throughout training.
Mid-training improves RL by identifying compact action abstractions.
problem Unlocking full potential of RL with large language models.
method RA3 algorithm that optimizes sequential variational lower bound and discovers latent structures via RL.
result Improves performance by 8 points on HumanEval and 4 points on MBPP.
Adam optimizer's bias is influenced by mini-batch size and momentum hyperparameters.
problem Understanding how Adam's implicit bias is affected by mini-batch size and momentum parameters.
method Theoretical framework to analyze mini-batch noise's impact on Adam's memory and bias.
result The magnitude of anti-regularization by memory depends on batch size and momentum hyperparameters.
Transformers learn linear models in-context without updates.
problem Understanding how transformers mimic linear models in-context.
method Gradient flow on linear regression tasks with random initialization.
result Transformers achieve prediction error competitive with best linear predictors.