InfoSFT improves LLMs by focusing on informative, medium-confidence tokens.
problem Overfitting to unlikely samples and degradation of prior capabilities in SFT.
method InfoSFT uses a principled weighting scheme to concentrate learning signals on medium-confidence tokens.
result InfoSFT improves generalization and preserves pre-existing capabilities over vanilla SFT and likelihood-weighted baselines.
The paper identifies patterns in language model weights used for memorizing paragraphs.
problem Locating the specific mechanisms and weights used by language models to memorize paragraphs.
method Examined gradients and attention patterns in language models to identify memorized paragraphs.
result Gradients of memorized paragraphs have a distinguishable spatial pattern, and localized attention heads are involved in paragraph memorization.
The p-Laplacian Transformer improves transformer models by assigning higher attention weights to tokens in close proximity.
problem The self-attention mechanism in transformers does not effectively distinguish attention weights between tokens in close and non-close proximity.
method Proposes a novel class of transformers, p-Laplacian Transformers, that use p-Laplacian regularization to assign higher attention weights to tokens in close proximity. result Empirically demonstrates that p-Laplacian Transformers outperform baseline transformers on various benchmark datasets.
LLM-as-a-service prices vary arbitrarily due to tokenization multiplicity.
problem Arbitrary price variation in LLM-as-a-service due to multiple tokenizations of the same output.
method Introduce canonical generation to restrict LLMs to unique tokenizations and develop an efficient sampling algorithm.
result Our sampling algorithm for canonical generation solves tokenization multiplicity and maintains comparable performance and runtime to standard sampling.
New insights show stochastic initialization prevents token clustering in deep Transformers.
problem Understanding token dynamics in deep stochastic Transformers.
method Analysis of deep Transformers with random initialization noise, proving convergence to an interacting-particle system on the sphere.
result Initialization noise prevents token clustering, leading to antipodal formations.
This research integrates attention into XAI frameworks for better model explanations.
problem Improving the interpretability of transformer models.
method Developed two novel explanation methods: Shapley value decomposition and token-level directional derivatives.
result Attention weights can be meaningfully incorporated into XAI frameworks, enhancing transformer explainability.
Language models allocate information storage, not collapsing into uniform representations.
problem Incomplete neural collapse in language model representations.
method Analyzing variance and information sharing across 14 models, proving an information floor.
result Within-class variance is allocated information storage, not collapsed into uniform representations.
Particles representing tokens cluster in Transformers, influenced by initial tokens and matrix spectrum.
problem Understanding the geometry of learned representations in Transformers.
method Viewing Transformers as particle systems, applying dynamical systems and partial differential equations.
result Particles cluster towards limiting objects, confirming context-awareness and the emergence of leaders.
Transformer models align words through attention weights, closely approximating Optimal Transport.
problem Understanding the internal mechanism of transformer models in language processing.
method Empirical evidence and theoretical analysis of attention weights and their relation to Optimal Transport.
result Transformer models can simulate gradient descent on the dual of entropy-regularized OT problem, providing a theoretical foundation for token alignment.
Attention mechanism learns to focus on sparse tokens efficiently.
problem Detecting weak, rare, and sparsely located features in long sequences.
method Theoretical analysis and training of a single-layer attention classifier in a sparse-token classification model.
result A single-layer attention classifier can achieve vanishing test error with logarithmic signal strength growth, unlike linear classifiers requiring linear growth.
GOTabPFN improves tabular model performance with compact tokenization for HDLSS data.
problem Making tabular models effective for high-dimensional, low-sample size data without retraining.
method Introducing Graph-guided Ordering with Local Refinement (GO-LR) and Neuro-Inspired Subunit Compression (NSC) to create compact meta-features.
result GOTabPFN improves stability and accuracy in tabular benchmarks with compact tokenization.
Prefix consistency improves model reliability by weighting answers based on their reproducibility.
problem Improving the reliability of large language models' reasoning traces.
method Use prefix consistency to weight candidate answers based on their reproducibility during regeneration.
result Prefix consistency is the best correctness predictor, reaching Standard MV plateau accuracy with up to 21x fewer tokens.
Study shows attention-style models learn pairwise interactions efficiently.
problem Learning pairwise interactions in attention-style models.
method Proved minimax rate of convergence for learning pairwise interactions.
result Minimax rate is M−2β+12β independent of embedding dimension and token number. Shampoo achieves higher token efficiency than Muon in language models.
problem Understanding the relationship and relative data efficiency of Shampoo and Muon compared to Adam and Signum.
method Extensive experiments on language models, demonstrating Shampoo's higher efficiency and decomposing its updates.
result Shampoo's benefits are attributed to its application to weight matrices, challenging interpretations based on variance adaptation and whitening.
Develops STC for sequential data with missing labels.
problem Learning from partially labeled and unsegmented sequential data.
method Introduces Star Temporal Classification (STC) using a star token and GTN framework.
result Recover most of supervised baseline performance with up to 70% missing labels.
nGPT learns to transfer learning rates across model dimensions and token horizons.
problem nGPT does not transfer learning rates across model size and token horizon.
method Combining numerical experiments with alignment exponents, a novel nGPT parameterization νGPT is developed.
result νGPT exhibits learning rate transfer across width, depth, and token horizon.
DFRot improves LLMs by reducing outlier and massive activation effects.
problem Reducing outlier and massive activation effects in rotated LLMs.
method Weighted loss function and orthogonal Procrustes transforms for rotation matrix refinement.
result DFRot achieves dual free (Outlier-Free and Massive Activation-Free) with significant improvements in perplexity.
Derives token price process for AMM tokens, finds leverage effect and pricing discrepancies.
problem Derives token price process for AMM tokens.
method Derives CEV process for token price, derives closed-form option prices, introduces liquidity-adjusted Greeks.
result Token price process is CEV, with leverage effect and pricing discrepancies.
Power-SMC reduces inference latency for training-free LLM reasoning.
problem Training-free LLM reasoning with low latency.
method Power-SMC, a training-free Sequential Monte Carlo scheme targeting sequence-level power distribution.
result Power-SMC reduces inference latency from 16-28× to 1.4-3.3× over baseline decoding.
Attention tokens are group elements, negated algebra norms.
problem Attention mechanism for matrix Lie groups
method Lie-Algebra Attention
result Score matches learned kernel on invariant poses, outperforms vector-token baseline
Dynamic-weight AMMs outperform traditional CEX rebalancing in tokenized funds, especially on L2s.
problem Improving asset allocation efficiency in decentralized finance (DeFi) protocols.
method Block-level arbitrage analysis and long-term performance benchmarks on two live pools.
result Dynamic-weight AMMs can achieve performance comparable to or better than traditional CEX rebalancing, especially on Layer 2 (L2) networks.
Transformer models waste resources on long-context tasks.
problem Redundant attention computations in Transformer models for long-context tasks.
method Reformulate sequence modeling as supervised learning, analyze attention sparsity, formulate attention optimization as linear coding problem, propose Dynamic Group Attention.
result DGA reduces computational costs while maintaining performance.
Unified framework for sequence models using test-time regression.
problem Designing efficient sequence models with associative memory.
method Formalizing associative recall as regression over input tokens, deriving various sequence models.
result Clarifies the effectiveness of query-key normalization in softmax attention and offers new generalizations.
Introduces Causal Energy Minimization to understand Transformer layers.
problem Empirical parameterization of Transformer blocks remains largely unexplored.
method Causal Energy Minimization framework that recasts Transformer layers as optimization steps on conditional energy functions.
result Identifies design space for Transformer layers including weight sharing and energy-based interpretations.
We show how to convert ICL in linearized transformers into model weights.
problem Making in-context learning interpretable and permanent in large language models.
method Demonstrates equivalence between ICL and bias terms in linearized transformers, and develops ICLCA for exact conversion.
result Exact conversion of in-context learning into model weights is possible for linearized transformers.
Chronicals boosts LLM fine-tuning speed by 3.51x over Unsloth.
problem Memory bottleneck in fine-tuning large language models.
method Four optimizations: fused kernels, cross-entropy reduction, LoRA+, and sequence packing.
result 3.51x speedup on A100-40GB with Qwen2.5-0.5B.
Elliptical Attention improves transformer performance by focusing on contextually relevant features.
problem Transformer models suffer from representation collapse and are vulnerable to contaminated samples.
method Uses Mahalanobis distance to define hyper-ellipsoidal neighborhoods for attention weights.
result Elliptical Attention reduces representation collapse and enhances model robustness.
Neural sequence generation is typically performed token-by-token and left-to-right. Whenever a token is generated only previously produced tokens are taken into consideration. In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into consideration, …
Detects crypto pump-and-dump schemes with a thresholding-based model.
problem Detecting genuine anomalies from minor trading fluctuations.
method Combining threshold-based criteria with EWMA and volatility measures.
result Balances high true-positive detection with minimal noise.
Study reveals how attention helps in signal recovery from sequence models using random matrix theory.
problem Signal recovery from sequence models with attention mechanisms.
method Analysis of sample covariance matrices constructed from pooled sequence representations with attention weights.
result Optimal attention weights maximize signal-to-noise ratio and improve signal recovery.
Paper reduces vocabulary losslessly for language model cooperation.
problem Language models struggle to cooperate with different tokenizations.
method Established a theoretical framework for lossless vocabulary reduction.
result Efficiently converts models with different tokenizations to cooperate with maximal common vocabulary.
This research improves capital efficiency and impermanent loss in cryptocurrency markets using multi-token trading pools.
problem Poor impermanent loss and capital efficiency in automated market makers.
method Analysis and construction of a multi-token token proactive market maker (MPMM).
result MPMM shows better impermanent loss and capital efficiency than comparable market makers.
This study examines whether tokenized assets improve liquidity and finds significant differences across categories.
problem Improving liquidity for real-world assets through tokenization.
method Examined tokenized real-world assets using Ethereum-based data, measuring liquidity through turnover, active addresses, and active-month indicator.
result Gold-backed tokens show more persistent on-chain activity than Treasury and private-credit-related products, but asset value alone does not reliably predict liquidity.
Study examines revenue from scam tokens on Ethereum, revealing key characteristics and market factors.
problem Revenue from scam tokens on Ethereum blockchain.
method Empirical analysis of Uniswap, examining characteristics and market factors.
result Revenue from scam tokens is influenced by market economic factors and community engagement.
This paper explains how to optimize prompts for model adaptation.
problem Understanding and optimizing prompt tuning for model adaptation.
method Bayesian view and meta-learning to explain prompt optimization.
result Optimal prompting can be studied formally as conditioning Bayesian predictors.
Blockchain-based exchanges adopt based on token pair volatility and personal use.
problem Token value loss and arbitrage issues in decentralized exchanges.
method Investigation of Automated Market Makers (AMMs) using transaction-level data.
result AMMs are adopted for high personal use or highly correlated token price movements.
Blockchain fan tokens boost sports fan engagement by 50%.
problem Low fan engagement in sports decisions.
method Analyzed 3,576 fan token polls to identify determinants of participation.
result Fan tokens engage 4,003 participants per poll, 50% of token holders.
CATR rationalizes text data to stabilize causal effect estimation.
problem Observational positivity violation in high-dimensional text data.
method Confounding-Aware Token Rationalization (CATR) selects necessary subset of tokens.
result CATR yields more accurate and stable causal effect estimates.
Study analyzes AI's impact on firms, markets, and workers using large language model data.
problem Understanding AI's effect on firms, markets, and workers.
method Used 380 trillion tokens from 400+ large language models to analyze AI's impact.
result Firms with higher AI exposure earn higher returns, creating an AI premium.
Paper introduces a method to assess liquidity risk in meme tokens using entity-linked address analysis.
problem High market volatility and vulnerability to manipulation in meme tokens.
method Multi-dimensional approach integrating fund flow analysis, behavioral similarity, and anomalous transaction detection.
result Significant disparities between apparent and actual liquidity in meme token markets.
Survey of diffusion and optimal transport methods in machine learning.
problem Design and analysis of time-evolving probability distributions in machine learning.
method Switch from Eulerian to Lagrangian representation through vector fields.
result Both diffusion methods and optimal transport offer computational advantages.
DOS improves language model generation by considering inter-token dependencies.
problem Lack of sequence-level information and inter-token dependencies in existing decoding strategies.
method Dependency-Oriented Sampler (DOS) that uses attention matrices to approximate inter-token dependencies.
result DOS consistently achieves superior performance on code generation and mathematical reasoning tasks.
Study predicts success of crypto-tokens on Pump.fun platform.
problem Identify factors affecting the success of new crypto-tokens.
method Build predictive models using bonding curve mechanism and structural/behavioral variables.
result Conditional variables significantly improve the predictive power of token success.
Proving that next-token prediction makes language models generate coherent long documents.
problem Understanding why language models generate coherent documents despite focusing on next-token prediction.
method Proving the power of next-token prediction in learning longer-range structure using Recurrent Neural Networks (RNN).
result Optimizing next-token prediction in RNNs yields a model that closely approximates the training distribution, even for long-range coherence.
Expands MLM by masking token positions, improving performance and convergence.
problem Improving language model performance and convergence.
method Masking token positions along with [MASK] tokens, using a fully connected classifier stage.
result Shows .3% improvement and 50% faster convergence for BERT Base with position masking.
This paper compares token and equity financing for startups.
problem Understanding differences in return rates between token and equity financing.
method Developed a three-period model to analyze liquidity and return differences.
result Entrepreneurs can achieve higher payoffs by issuing tokens, especially for risk-averse investors with liquidity needs.
QA-Token improves tokenization for noisy data, boosting model performance.
problem Tokenization ignores data quality, limiting model effectiveness on noisy corpora.
method QA-Token combines signal quality with vocabulary construction through bilevel optimization and reinforcement learning.
result QA-Token achieves state-of-the-art performance on genomic and financial datasets.
Minimal token perturbations reveal how Transformer models process information.
problem Understanding information propagation in Transformer models for interpretability.
method Study of minimal token perturbations on embedding space.
result Rare tokens cause larger shifts, and input information mixes deeper.