Proving that next-token prediction makes language models generate coherent long documents.
problem Understanding why language models generate coherent documents despite focusing on next-token prediction.
method Proving the power of next-token prediction in learning longer-range structure using Recurrent Neural Networks (RNN).
result Optimizing next-token prediction in RNNs yields a model that closely approximates the training distribution, even for long-range coherence.
Law explains how LLMs learn to predict next tokens.
problem Understanding how LLMs process input data internally.
method Introduced a precise law governing token embeddings in LLMs.
result Each layer equally contributes to next-token prediction accuracy.
Reinforcement learning after next-token prediction aids in learning from diverse sequence lengths.
problem Learning from sequences of varying lengths and complexity.
method Introducing a framework to study reinforcement learning with autoregressive transformers, focusing on next-token prediction and mixture distributions of short and long sequences.
result Reinforcement learning after next-token prediction enables autoregressive transformers to generalize from long sequences, even when they are rare.
This paper examines how different decoding algorithms for LLMs align with various goals.
problem Consistency of decoding algorithms with different goals in LLMs.
method Analysis of greedy, lookahead, random sampling, and temperature-scaled random sampling algorithms.
result Random sampling is consistent with the true probability distribution, but other goals require optimal algorithms for specific probability distributions.
The study explores how Transformers predict the next token in a sequence.
problem Understanding the mechanism behind Transformers' autoregressive learning ability.
method Exploring the approximation ability of Transformers for next-token prediction through specific instances and a causal kernel descent method.
result Transformer models can learn context-dependent functions f for next-token prediction based on past and current observations. This paper explores how Transformers predict next tokens in autoregressive tasks.
problem Understanding the success of Transformers in autoregressive learning.
method Trained a Transformer on a next-token prediction task, focusing on commuting orthogonal matrices.
result Trained Transformers can be seen as implementing gradient descent for a specific objective function.
Gradient descent biases linear models in next-token prediction towards data entropy.
problem Optimization bias in next-token prediction models.
method Analysis of gradient descent on linear models with sparse conditional distributions.
result Gradient descent selects parameters that equate token logits differences to log-odds in the data subspace.
A new method improves language model extrapolation without changing training.
problem Improving language model performance on sequences not seen during training.
method Perturbation-based training procedure for language models.
result The method consistently improves out-of-support prediction.
Study reveals AI's spontaneous topic changes in text prediction.
problem AI's inability to spontaneously switch topics like humans.
method Defined topic as Token Priority Graphs (TPGs) and analyzed self-attention models.
result AI can only switch topics if lower-priority tokens outnumber higher-priority ones.
Mathematical framework for language models processes text and predicts next tokens.
problem Understanding and optimizing the performance of large language models.
method Describes encoding, prediction models, learning from data, and deployment of LLMs.
result Demonstrates remarkable empirical successes and provides a platform for further research.
Paper reduces vocabulary losslessly for language model cooperation.
problem Language models struggle to cooperate with different tokenizations.
method Established a theoretical framework for lossless vocabulary reduction.
result Efficiently converts models with different tokenizations to cooperate with maximal common vocabulary.
The paper analyzes the training dynamics of a transformer for next-token prediction.
problem Understanding the non-asymptotic performance of transformers in next-token prediction.
method Characterizes training dataset properties, designs a two-stage training algorithm, and analyzes attention gradient properties.
result Trained transformers converge sub-linearly to max-margin solutions and exhibit linear convergence in cross-entropy loss.
Study quantifies how LLMs capture higher-order statistical structure using cumulant expansion.
problem Understanding how LLMs internalize statistical structure during next-token prediction.
method Cumulant-expansion framework treating softmax entropy as perturbation around center distribution.
result Cumulants reveal distinct signatures for mathematical vs. general text prompts, quantifying feature-learning dynamics.
New approach predicts tokens in context, explaining how ICL emerges.
problem Limited understanding of in-context learning emergence.
method Auto-regressive next-token prediction (AR-NTP) with prompt token-dependency and a two-level expectation.
result ICL emerges from the generalization of sequences and topics.
GPT learns a causal world model from token predictions, validated in game sequences.
problem Does GPT implicitly learn a causal world model from token predictions?
method Derived a causal interpretation of GPT's attention mechanism and proposed zero-shot causal structure learning.
result GPT can generate legal next moves with high confidence for sequences with encoded causal structures, but fails for illegal moves.
Language models can predict numeric values as strings.
problem Regression tasks with numeric predictions.
method Causal sequence decoding models trained for next-token prediction.
result Decoder-based heads perform as well as standard heads in numeric regression tasks.
Timer-XL predicts multidimensional time series using a unified Transformer approach.
problem Unified time series forecasting across various tasks and contexts.
method Decoder-only Transformers with a universal TimeAttention mechanism and deft position embedding.
result State-of-the-art performance across multiple forecasting benchmarks.
This paper shows ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.
problem Understanding the lookahead capabilities of next-token prediction models.
method Unified view of ARMs and EBMs, establishing a bijection and deriving equivalence.
result ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.
LLMs show surprising confidence in their answers, beyond just tokens.
problem LLMs lack meaningful confidence estimates for their responses.
method Semantic calibration test based on local loss optimality and equivalence classes.
result Base LLMs are semantically calibrated across tasks, contrary to expectations.
New L2D framework allows deferring specific parts of a sequence prediction to experts.
problem Current L2D methods defer entire predictions, which is not ideal for long sequences.
method Proposes token-level and one-time rejectors to defer specific outputs of a model prediction to experts.
result Granular deferrals achieve better cost-accuracy tradeoffs than whole deferrals.
Linear properties are either universal or absent across language models.
problem Explaining the prevalence of linear properties in language models.
method Proved identifiability of distribution-equivalent next-token predictors and analyzed various notions of linearity.
result Linear properties either hold in all or none distribution-equivalent next-token predictors.
The paper distinguishes between conditional and marginal processes in language models and discusses conditions for usefulness.
problem The conditional nature of language models trained on observed sequences and the need for marginal text-only processes.
method Distinguishing between full conditional language process, marginal text-only process, and model-induced distribution; analyzing assumptions of stationarity and ergodicity.
result The marginal text-only law is useful only when the observed prefix is an approximately sufficient statistic for the latent circumstances relevant to continuation.
This work explains how linear representations in large language models arise from training objectives and gradient descent.
problem Understanding the origins of linear representations in large language models.
method A latent variable model to abstract and formalize concept dynamics, combined with analysis of the softmax cross-entropy objective and gradient descent.
result Linear representations emerge when learning from data matching the latent variable model, and this simple structure suffices to yield linear representations.
Generative model predicts NFT collection transactions based on early history.
problem Predict future transactions of newly minted NFT collections.
method Unsupervised learning to extract contexts, then generate future transactions.
result Projected market value of new NFT collections.
Unified theory for neural scaling laws in hierarchically compositional data.
problem Understanding neural scaling laws in hierarchically compositional data.
method Probabilistic context-free grammars and power-law distributed production rules.
result Unified learning curve behavior for classification and next-token prediction tasks.
Theory predicts neural scaling exponents from language statistics.
problem No existing theory could quantitatively predict neural scaling exponents.
method Isolated two key statistical properties of language.
result Derives a simple formula predicting neural scaling exponents.
New bounds for large language models using token properties.
problem Vacuous generalization bounds for large language models.
method Martingale properties and Monarch matrices.
result Non-vacuous generalization bounds for LLMs up to 70B parameters.
In complex processes, various events can happen in different sequences. The prediction of the next event given an a-priori process state is of importance in such processes. Recent methods have proposed deep learning techniques such as recurrent neural networks, developed on raw event logs, to predict the next event fro…
Language models allocate information storage, not collapsing into uniform representations.
problem Incomplete neural collapse in language model representations.
method Analyzing variance and information sharing across 14 models, proving an information floor.
result Within-class variance is allocated information storage, not collapsed into uniform representations.
A new sampler for FLMs improves token-level decoding controls.
problem Sampling from FLMs using standard methods collapses marginals and produces invalid sequences.
method Samples clean one-hot endpoints from FLM token marginals and uses Ornstein-Uhlenbeck bridges conditioned on these endpoints.
result The method preserves token-wise posterior-predictive marginals and improves quality-diversity tradeoff.
Randomly initialized transformers show extreme token preferences.
problem Structural biases in randomly initialized transformers.
method Dissection of transformer architecture at initialization.
result Initialization-induced biases persist throughout training.
Two new estimators reduce costs and improve accuracy for EHR outcome prediction.
problem Sparse estimate distributions, high computational cost, and high sampling variance in EHR outcome prediction.
method Proposed SCOPE and REACH estimators that leverage next-token probability distributions.
result SCOPE and REACH match Monte Carlo accuracy with token reductions of 2.5-3.4 times and variance guarantees.
The paper uncovers symmetries in large language models through layer-peeled optimization.
problem Understanding geometric structure in large language model weights and context embeddings.
method Constrained layer-peeled optimization program to analyze symmetries in next-token distributions.
result Symmetries in target next-token distributions are transferred to optimal model weights and context embeddings.
This paper studies transformer learning dynamics and initialization.
problem Understanding how transformers learn Markov chains and the role of initialization.
method First-order Markov chains and single-layer transformers, proving learning dynamics and conditions for convergence.
result Transformer parameters can converge to global or local minima based on initialization and Markovian data properties.
Study predicts success of crypto-tokens on Pump.fun platform.
problem Identify factors affecting the success of new crypto-tokens.
method Build predictive models using bonding curve mechanism and structural/behavioral variables.
result Conditional variables significantly improve the predictive power of token success.
A method for learning with autoregressive chain-of-thoughts.
problem Learning prompt-to-answer mappings from sequence-to-next-token generators.
method Iterating a fixed, time-invariant generator for multiple steps to generate a chain-of-thought, then taking the final token as the answer.
result Universal representability and computationally tractable chain-of-thought learning for a simple base class.
Study shows how transformers classify symbols without naming them, proving a margin-versus-collision criterion.
problem How transformers classify symbols without naming them.
method Logistic classification analysis of transformer-kernel regime, colored collision graph.
result Decomposes learned predictor into ideal template-level classifier and finite-sample perturbation.
Tokenized RWAs face liquidity issues despite promising markets.
problem Low trading volumes and limited investor participation in tokenized assets.
method Empirical analysis of tokenized real estate, private credit, and treasury funds.
result Most tokenized assets exhibit low transfer activity and limited secondary trading.
ByteGen models LOB dynamics without tokenization, achieving realistic market metrics.
problem Modeling high-frequency LOB dynamics in finance.
method Autoregressive next-byte prediction on packed binary data, using H-Net architecture.
result Successfully reproduces stylized facts of financial markets.
Global watermark for diffusion language models decouples detection from local contexts.
problem Watermarking in diffusion language models is challenging due to joint sampling of distributions over many unresolved positions.
method Proposes a global vector-valued sketch representation to control watermarking in masked diffusion language models.
result The method decouples detection from local contexts, resulting in an order-agnostic statistic and robustness.
Transformers learn linear models in-context without updates.
problem Understanding how transformers mimic linear models in-context.
method Gradient flow on linear regression tasks with random initialization.
result Transformers achieve prediction error competitive with best linear predictors.
New insights show stochastic initialization prevents token clustering in deep Transformers.
problem Understanding token dynamics in deep stochastic Transformers.
method Analysis of deep Transformers with random initialization noise, proving convergence to an interacting-particle system on the sphere.
result Initialization noise prevents token clustering, leading to antipodal formations.
In-Place TTT enhances LLMs with dynamic parameter updates at inference time.
problem Static training limits LLMs from adapting to new information.
method In-Place TTT updates a subset of model parameters (fast weights) at inference time.
result In-Place TTT enables 4B-parameter models to outperform on tasks with up to 128k contexts.
We empirically verify that the market capitalisations of coins and tokens in the cryptocurrency universe follow power-law distributions with significantly different values, with the tail exponent falling between 0.5 and 0.7 for coins, and between 1.0 and 1.3 for tokens. We provide a rationale for this, based on a simpl…
Thinking LLMs struggle with stock prediction, especially as data complexity increases.
problem Evaluating the performance of 'thinking' LLMs in stock prediction, especially under varying levels of cross-sectional complexity.
method Rolling 48m/1m walk-forward evaluation, comparing direct LLMs, TLLMs, and classical learners on cross-sectional ranking loss, MSE, and backtests with transaction costs.
result TLLMs' ranking quality deteriorates as cross-sectional complexity grows, while direct LLMs remain stable.
StockTime predicts stock prices more accurately using LLMs and time series data.
problem Challenges in integrating time series data and natural language for stock price prediction.
method StockTime is a specialized LLM architecture that integrates textual and time series data to predict stock prices.
result StockTime outperforms recent LLMs in predicting stock prices with more accuracy.
The paper identifies patterns in language model weights used for memorizing paragraphs.
problem Locating the specific mechanisms and weights used by language models to memorize paragraphs.
method Examined gradients and attention patterns in language models to identify memorized paragraphs.
result Gradients of memorized paragraphs have a distinguishable spatial pattern, and localized attention heads are involved in paragraph memorization.
LLMs encode latent topic distributions, suggesting Bayesian inference.
problem Capturing topic structure from large language models.
method Connecting LLM optimization to implicit Bayesian inference and de Finetti's theorem.
result LLMs recover latent topic distributions, matching LDA-generated topics.