DOS improves language model generation by considering inter-token dependencies.
problem Lack of sequence-level information and inter-token dependencies in existing decoding strategies.
method Dependency-Oriented Sampler (DOS) that uses attention matrices to approximate inter-token dependencies.
result DOS consistently achieves superior performance on code generation and mathematical reasoning tasks.
New approach predicts tokens in context, explaining how ICL emerges.
problem Limited understanding of in-context learning emergence.
method Auto-regressive next-token prediction (AR-NTP) with prompt token-dependency and a two-level expectation.
result ICL emerges from the generalization of sequences and topics.
New method improves combinatorial optimization by capturing dependencies among solution variables.
problem Performance limitations in solving combinatorial optimization problems using independent solution variables.
method Subgraph tokenization and variational annealing to capture dependencies and improve learning efficiency.
result Empirical evidence shows superior performance of autoregressive methods with tokenization and annealed entropy regularization.
Particles representing tokens cluster in Transformers, influenced by initial tokens and matrix spectrum.
problem Understanding the geometry of learned representations in Transformers.
method Viewing Transformers as particle systems, applying dynamical systems and partial differential equations.
result Particles cluster towards limiting objects, confirming context-awareness and the emergence of leaders.
LADD models improve discrete diffusion for faster language generation.
problem Practical discrete diffusion models ignore cross-token dependencies, degrading performance.
method Introduces a learnable auxiliary latent channel, diffusing over the joint (token, latent) space.
result LADD models yield improvements on unconditional generation metrics.
Token-adaptive FFN design improves LLM expressivity.
problem Fixed activation functions limit FFN expressivity in LLMs.
method Mixture of Activations (MoA) and learnable activations (LA).
result MoA achieves lower terminal loss and better scaling than baselines.
Timer-XL predicts multidimensional time series using a unified Transformer approach.
problem Unified time series forecasting across various tasks and contexts.
method Decoder-only Transformers with a universal TimeAttention mechanism and deft position embedding.
result State-of-the-art performance across multiple forecasting benchmarks.
New method adapts DLMs to intrinsic data dependence without prior knowledge.
problem Understanding how unmasking schedules affect DLM generation quality.
method Adapts unmasking schedule to target data distribution's dependence structure.
result Sampling convergence guarantees improve for low-complexity distributions.
Recurrent Neural Networks (RNNs) are among the most popular models in sequential data analysis. Yet, in the foundational PAC learning language, what concept class can it learn? Moreover, how can the same recurrent unit simultaneously learn functions from different input tokens to different output tokens, without affect…
New bounds for large language models using token properties.
problem Vacuous generalization bounds for large language models.
method Martingale properties and Monarch matrices.
result Non-vacuous generalization bounds for LLMs up to 70B parameters.
KLD token adjusts supply based on macroeconomic debt index, creating deflationary effect.
problem Managing deflationary pressures in digital assets.
method Debt-indexed supply adjustments linked to macroeconomic data.
result Deflationary mechanism strengthens as debt rises.
Study reveals AI's spontaneous topic changes in text prediction.
problem AI's inability to spontaneously switch topics like humans.
method Defined topic as Token Priority Graphs (TPGs) and analyzed self-attention models.
result AI can only switch topics if lower-priority tokens outnumber higher-priority ones.
A new sampler for FLMs improves token-level decoding controls.
problem Sampling from FLMs using standard methods collapses marginals and produces invalid sequences.
method Samples clean one-hot endpoints from FLM token marginals and uses Ornstein-Uhlenbeck bridges conditioned on these endpoints.
result The method preserves token-wise posterior-predictive marginals and improves quality-diversity tradeoff.
Global watermark for diffusion language models decouples detection from local contexts.
problem Watermarking in diffusion language models is challenging due to joint sampling of distributions over many unresolved positions.
method Proposes a global vector-valued sketch representation to control watermarking in masked diffusion language models.
result The method decouples detection from local contexts, resulting in an order-agnostic statistic and robustness.
The study explores how Transformers predict the next token in a sequence.
problem Understanding the mechanism behind Transformers' autoregressive learning ability.
method Exploring the approximation ability of Transformers for next-token prediction through specific instances and a causal kernel descent method.
result Transformer models can learn context-dependent functions f for next-token prediction based on past and current observations. Randomly initialized transformers show extreme token preferences.
problem Structural biases in randomly initialized transformers.
method Dissection of transformer architecture at initialization.
result Initialization-induced biases persist throughout training.
Derives token price process for AMM tokens, finds leverage effect and pricing discrepancies.
problem Derives token price process for AMM tokens.
method Derives CEV process for token price, derives closed-form option prices, introduces liquidity-adjusted Greeks.
result Token price process is CEV, with leverage effect and pricing discrepancies.
Transformer model improves source code summarization.
problem Generating readable summaries of source code.
method Transformer model with self-attention mechanism for code representation.
result Transformer model outperforms state-of-the-art techniques.
Analyzes how BPE tokenisation affects corpus statistics and model entropy in transformer models.
problem Understanding how natural language properties relate to tokenisation schemes in transformer models.
method Analyzes Shannon entropy of corpora under Zipfian distribution, investigates BPE transformations, trains language models, and uses attention diagnostics.
result Transformer models trained on BPE-tokenised corpora increasingly agree with Zipfian predictions as BPE depth increases, indicating reduced local token dependencies.
CoT improves transformer sample efficiency by reducing input token dependencies and attention sparsity.
problem Transformer sample inefficiency in simple tasks.
method Demonstrated through parity-learning setup, showing CoT reduces required samples from exponential to polynomial.
result Transformer learns function within polynomial samples with CoT, requiring exponential samples without CoT.
SSMs combined with neural networks match Transformers in dynamic token selection.
problem Understanding the capabilities of SSMs in dynamic token selection.
method Exploring SSMs combined with fully connected neural networks.
result SSMs combined with nonlinear layers can efficiently solve challenging tasks and estimate functions.
Generating logical form equivalents of human language is a fresh way to employ neural architectures where long short-term memory effectively captures dependencies in both encoder and decoder units. The logical form of the sequence usually preserves information from the natural language side in the form of similar token…
Neural sequence generation is typically performed token-by-token and left-to-right. Whenever a token is generated only previously produced tokens are taken into consideration. In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into consideration, …
Study quantifies how LLMs capture higher-order statistical structure using cumulant expansion.
problem Understanding how LLMs internalize statistical structure during next-token prediction.
method Cumulant-expansion framework treating softmax entropy as perturbation around center distribution.
result Cumulants reveal distinct signatures for mathematical vs. general text prompts, quantifying feature-learning dynamics.
Transformer models waste resources on long-context tasks.
problem Redundant attention computations in Transformer models for long-context tasks.
method Reformulate sequence modeling as supervised learning, analyze attention sparsity, formulate attention optimization as linear coding problem, propose Dynamic Group Attention.
result DGA reduces computational costs while maintaining performance.
Sandbox LAND prices differ based on unit of account, affecting investment returns.
problem Investment returns vary based on how prices are denominated.
method Analyzed over 71,000 transactions to compare different units of account.
result Users are willing to pay more in SAND and less in wETH for transactions.
Transformers learn to recall with non-orthogonal embeddings in realistic settings.
problem Understanding how transformers store and retrieve knowledge in practical scenarios.
method Analyzing a single-layer transformer with random embeddings trained on a token-retrieval task.
result Explicit formulas for the model's storage capacity reveal a multiplicative dependence on sample size, embedding dimension, and sequence length.
Paper reduces vocabulary losslessly for language model cooperation.
problem Language models struggle to cooperate with different tokenizations.
method Established a theoretical framework for lossless vocabulary reduction.
result Efficiently converts models with different tokenizations to cooperate with maximal common vocabulary.
This research improves capital efficiency and impermanent loss in cryptocurrency markets using multi-token trading pools.
problem Poor impermanent loss and capital efficiency in automated market makers.
method Analysis and construction of a multi-token token proactive market maker (MPMM).
result MPMM shows better impermanent loss and capital efficiency than comparable market makers.
This study examines whether tokenized assets improve liquidity and finds significant differences across categories.
problem Improving liquidity for real-world assets through tokenization.
method Examined tokenized real-world assets using Ethereum-based data, measuring liquidity through turnover, active addresses, and active-month indicator.
result Gold-backed tokens show more persistent on-chain activity than Treasury and private-credit-related products, but asset value alone does not reliably predict liquidity.
Study examines revenue from scam tokens on Ethereum, revealing key characteristics and market factors.
problem Revenue from scam tokens on Ethereum blockchain.
method Empirical analysis of Uniswap, examining characteristics and market factors.
result Revenue from scam tokens is influenced by market economic factors and community engagement.
Blockchain-based exchanges adopt based on token pair volatility and personal use.
problem Token value loss and arbitrage issues in decentralized exchanges.
method Investigation of Automated Market Makers (AMMs) using transaction-level data.
result AMMs are adopted for high personal use or highly correlated token price movements.
Blockchain fan tokens boost sports fan engagement by 50%.
problem Low fan engagement in sports decisions.
method Analyzed 3,576 fan token polls to identify determinants of participation.
result Fan tokens engage 4,003 participants per poll, 50% of token holders.
EPSTE: A geometric token and deep learning approach to estimating transfer entropy in neuroimaging time series
problem Inferring directed interactions between neural systems from EEG and MEG
method Reframing TE estimation as a learnable problem operating on structured symbolic representations
result EPSTE achieves near-perfect recovery of ground-truth directed structure and significantly lower absolute error than the baseline
Paper introduces a method to assess liquidity risk in meme tokens using entity-linked address analysis.
problem High market volatility and vulnerability to manipulation in meme tokens.
method Multi-dimensional approach integrating fund flow analysis, behavioral similarity, and anomalous transaction detection.
result Significant disparities between apparent and actual liquidity in meme token markets.
Study predicts success of crypto-tokens on Pump.fun platform.
problem Identify factors affecting the success of new crypto-tokens.
method Build predictive models using bonding curve mechanism and structural/behavioral variables.
result Conditional variables significantly improve the predictive power of token success.
Proving that next-token prediction makes language models generate coherent long documents.
problem Understanding why language models generate coherent documents despite focusing on next-token prediction.
method Proving the power of next-token prediction in learning longer-range structure using Recurrent Neural Networks (RNN).
result Optimizing next-token prediction in RNNs yields a model that closely approximates the training distribution, even for long-range coherence.
Recurrent neural networks have become ubiquitous in computing representations of sequential data, especially textual data in natural language processing. In particular, Bidirectional LSTMs are at the heart of several neural models achieving state-of-the-art performance in a wide variety of tasks in NLP. However, BiLSTM…
Expands MLM by masking token positions, improving performance and convergence.
problem Improving language model performance and convergence.
method Masking token positions along with [MASK] tokens, using a fully connected classifier stage.
result Shows .3% improvement and 50% faster convergence for BERT Base with position masking.
This paper compares token and equity financing for startups.
problem Understanding differences in return rates between token and equity financing.
method Developed a three-period model to analyze liquidity and return differences.
result Entrepreneurs can achieve higher payoffs by issuing tokens, especially for risk-averse investors with liquidity needs.
QA-Token improves tokenization for noisy data, boosting model performance.
problem Tokenization ignores data quality, limiting model effectiveness on noisy corpora.
method QA-Token combines signal quality with vocabulary construction through bilevel optimization and reinforcement learning.
result QA-Token achieves state-of-the-art performance on genomic and financial datasets.
Minimal token perturbations reveal how Transformer models process information.
problem Understanding information propagation in Transformer models for interpretability.
method Study of minimal token perturbations on embedding space.
result Rare tokens cause larger shifts, and input information mixes deeper.
LLM-as-a-service prices vary arbitrarily due to tokenization multiplicity.
problem Arbitrary price variation in LLM-as-a-service due to multiple tokenizations of the same output.
method Introduce canonical generation to restrict LLMs to unique tokenizations and develop an efficient sampling algorithm.
result Our sampling algorithm for canonical generation solves tokenization multiplicity and maintains comparable performance and runtime to standard sampling.
Unified theory for neural scaling laws in hierarchically compositional data.
problem Understanding neural scaling laws in hierarchically compositional data.
method Probabilistic context-free grammars and power-law distributed production rules.
result Unified learning curve behavior for classification and next-token prediction tasks.
Traditional centralized energy systems have the disadvantages of difficult management and insufficient incentives. Blockchain is an emerging technology, which can be utilized in energy systems to enhance their management and control. Integrating token economy and blockchain technology, token economic systems in energy …
New insights show stochastic initialization prevents token clustering in deep Transformers.
problem Understanding token dynamics in deep stochastic Transformers.
method Analysis of deep Transformers with random initialization noise, proving convergence to an interacting-particle system on the sphere.
result Initialization noise prevents token clustering, leading to antipodal formations.
The paper analyzes risk spillovers between AI ETFs, AI tokens, and green markets.
problem Risk spillovers among AI ETFs, AI tokens, and green markets.
method R2 decomposition method
result AI ETFs and clean energy act as risk transmitters, while AI tokens and green assets act as receivers.
Study reveals risks of investing in new crypto-tokens in decentralized exchanges.
problem Risks associated with investing in newly created tokens in decentralized exchanges.
method Analysis of financial impact, market dynamics, profitability, and liquidity manipulations.
result Significant market liquidity trapped in honeypots, reducing market efficiency and misleading investors.