Randomly initialized transformers show extreme token preferences.
problem Structural biases in randomly initialized transformers.
method Dissection of transformer architecture at initialization.
result Initialization-induced biases persist throughout training.
A new algorithm RTO integrates DPO and PPO for RLHF, improving performance.
problem Learning from sparse, sentence-level rewards in RLHF.
method Modeling RLHF as MDP, introducing RTO that learns token-wise reward function.
result RTO outperforms PPO on benchmarks by 7.5 points on AlpacaEval 2 and 4.1 points on Arena-Hard.
Model analyzes Proof-of-Stake network dynamics and speculative capital effects on token prices.
problem Understanding and managing price dynamics in Proof-of-Stake networks.
method Developed an open-economy macroeconomic model to analyze Proof-of-Stake dynamics and speculative capital effects.
result Speculative capital can shift staked-token ownership, potentially improving consensus decentralization.
Proof of Stake (PoS) is a burgeoning Sybil resistance mechanism that aims to have a digital asset ("token") serve as security collateral in crypto networks. However, PoS has so far eluded a comprehensive threat model that encompasses both Byzantine attacks from distributed systems and financial attacks that arise from …
New statistical models for predicting ranked preferences from partial orders.
problem Statistical models overlook information in list length.
method Composite and augmented ranking models for joint modeling of partial orders and list lengths.
result Augmented ranking models best predict both length and preferences.
Global Style Tokens (GSTs) are a recently-proposed method to learn latent disentangled representations of high-dimensional data. GSTs can be used within Tacotron, a state-of-the-art end-to-end text-to-speech synthesis system, to uncover expressive factors of variation in speaking style. In this work, we introduce the T…
Neural sequence generation is typically performed token-by-token and left-to-right. Whenever a token is generated only previously produced tokens are taken into consideration. In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into consideration, …
Paper reduces vocabulary losslessly for language model cooperation.
problem Language models struggle to cooperate with different tokenizations.
method Established a theoretical framework for lossless vocabulary reduction.
result Efficiently converts models with different tokenizations to cooperate with maximal common vocabulary.
This research improves capital efficiency and impermanent loss in cryptocurrency markets using multi-token trading pools.
problem Poor impermanent loss and capital efficiency in automated market makers.
method Analysis and construction of a multi-token token proactive market maker (MPMM).
result MPMM shows better impermanent loss and capital efficiency than comparable market makers.
This study examines whether tokenized assets improve liquidity and finds significant differences across categories.
problem Improving liquidity for real-world assets through tokenization.
method Examined tokenized real-world assets using Ethereum-based data, measuring liquidity through turnover, active addresses, and active-month indicator.
result Gold-backed tokens show more persistent on-chain activity than Treasury and private-credit-related products, but asset value alone does not reliably predict liquidity.
Study examines revenue from scam tokens on Ethereum, revealing key characteristics and market factors.
problem Revenue from scam tokens on Ethereum blockchain.
method Empirical analysis of Uniswap, examining characteristics and market factors.
result Revenue from scam tokens is influenced by market economic factors and community engagement.
Blockchain-based exchanges adopt based on token pair volatility and personal use.
problem Token value loss and arbitrage issues in decentralized exchanges.
method Investigation of Automated Market Makers (AMMs) using transaction-level data.
result AMMs are adopted for high personal use or highly correlated token price movements.
Blockchain fan tokens boost sports fan engagement by 50%.
problem Low fan engagement in sports decisions.
method Analyzed 3,576 fan token polls to identify determinants of participation.
result Fan tokens engage 4,003 participants per poll, 50% of token holders.
Paper introduces a method to assess liquidity risk in meme tokens using entity-linked address analysis.
problem High market volatility and vulnerability to manipulation in meme tokens.
method Multi-dimensional approach integrating fund flow analysis, behavioral similarity, and anomalous transaction detection.
result Significant disparities between apparent and actual liquidity in meme token markets.
DOS improves language model generation by considering inter-token dependencies.
problem Lack of sequence-level information and inter-token dependencies in existing decoding strategies.
method Dependency-Oriented Sampler (DOS) that uses attention matrices to approximate inter-token dependencies.
result DOS consistently achieves superior performance on code generation and mathematical reasoning tasks.
Study predicts success of crypto-tokens on Pump.fun platform.
problem Identify factors affecting the success of new crypto-tokens.
method Build predictive models using bonding curve mechanism and structural/behavioral variables.
result Conditional variables significantly improve the predictive power of token success.
Proving that next-token prediction makes language models generate coherent long documents.
problem Understanding why language models generate coherent documents despite focusing on next-token prediction.
method Proving the power of next-token prediction in learning longer-range structure using Recurrent Neural Networks (RNN).
result Optimizing next-token prediction in RNNs yields a model that closely approximates the training distribution, even for long-range coherence.
Expands MLM by masking token positions, improving performance and convergence.
problem Improving language model performance and convergence.
method Masking token positions along with [MASK] tokens, using a fully connected classifier stage.
result Shows .3% improvement and 50% faster convergence for BERT Base with position masking.
This paper compares token and equity financing for startups.
problem Understanding differences in return rates between token and equity financing.
method Developed a three-period model to analyze liquidity and return differences.
result Entrepreneurs can achieve higher payoffs by issuing tokens, especially for risk-averse investors with liquidity needs.
QA-Token improves tokenization for noisy data, boosting model performance.
problem Tokenization ignores data quality, limiting model effectiveness on noisy corpora.
method QA-Token combines signal quality with vocabulary construction through bilevel optimization and reinforcement learning.
result QA-Token achieves state-of-the-art performance on genomic and financial datasets.
Minimal token perturbations reveal how Transformer models process information.
problem Understanding information propagation in Transformer models for interpretability.
method Study of minimal token perturbations on embedding space.
result Rare tokens cause larger shifts, and input information mixes deeper.
LLM-as-a-service prices vary arbitrarily due to tokenization multiplicity.
problem Arbitrary price variation in LLM-as-a-service due to multiple tokenizations of the same output.
method Introduce canonical generation to restrict LLMs to unique tokenizations and develop an efficient sampling algorithm.
result Our sampling algorithm for canonical generation solves tokenization multiplicity and maintains comparable performance and runtime to standard sampling.
Traditional centralized energy systems have the disadvantages of difficult management and insufficient incentives. Blockchain is an emerging technology, which can be utilized in energy systems to enhance their management and control. Integrating token economy and blockchain technology, token economic systems in energy …
New approach uses inverse reinforcement learning to improve language model training.
problem Training large language models using imitation learning methods.
method Developed a new method of inverse reinforcement learning to optimize sequences directly.
result IRL-based fine-tuning leads to better performance and diversity in language generation.
New insights show stochastic initialization prevents token clustering in deep Transformers.
problem Understanding token dynamics in deep stochastic Transformers.
method Analysis of deep Transformers with random initialization noise, proving convergence to an interacting-particle system on the sphere.
result Initialization noise prevents token clustering, leading to antipodal formations.
The paper analyzes risk spillovers between AI ETFs, AI tokens, and green markets.
problem Risk spillovers among AI ETFs, AI tokens, and green markets.
method R2 decomposition method
result AI ETFs and clean energy act as risk transmitters, while AI tokens and green assets act as receivers.
Study reveals risks of investing in new crypto-tokens in decentralized exchanges.
problem Risks associated with investing in newly created tokens in decentralized exchanges.
method Analysis of financial impact, market dynamics, profitability, and liquidity manipulations.
result Significant market liquidity trapped in honeypots, reducing market efficiency and misleading investors.
Tokenized RWAs face liquidity issues despite promising markets.
problem Low trading volumes and limited investor participation in tokenized assets.
method Empirical analysis of tokenized real estate, private credit, and treasury funds.
result Most tokenized assets exhibit low transfer activity and limited secondary trading.
Fan tokens surged before World Cup matches, but declined during them, revealing cognitive biases.
problem Analyzing the impact of FIFA World Cup matches on fan tokens.
method Event study and intraday analysis of blockchain-based fan tokens.
result Fan tokens experienced a surge in returns six months before the World Cup, followed by a decline during the matches, revealing asymmetries in performance.
We empirically verify that the market capitalisations of coins and tokens in the cryptocurrency universe follow power-law distributions with significantly different values, with the tail exponent falling between 0.5 and 0.7 for coins, and between 1.0 and 1.3 for tokens. We provide a rationale for this, based on a simpl…
In distributed function computation, each node has an initial value and the goal is to compute a function of these values in a distributed manner. In this paper, we propose a novel token-based approach to compute a wide class of target functions to which we refer as "Token-based function Computation with Memory" (TCM) …
Decoding strategies often exclude human-like tokens, creating a detectable gap in generated text.
problem Decoding strategies exclude contextually appropriate but statistically rare tokens, creating a detectable gap in generated text.
method Analysis of 1.8 million texts across 8 language models, 5 decoding strategies, and 53 hyperparameter configurations.
result 8-18% of human-selected tokens fall outside typical truncation boundaries, indicating a detectable gap.
Method reveals hidden token embeddings of large language models.
problem Understanding hidden token embeddings in large language models.
method Structured prompts to expose token input embeddings up to homeomorphism.
result Mathematical proof for generic LLMs shows effectiveness of method.
Paper develops a risk scoring framework for tokenized RWA markets.
problem Tokenized assets may not reflect true risk due to illiquidity and concentration.
method Develops a risk scoring framework based on observable indicators.
result Assets with limited transfer activity and concentrated ownership have high empirical risk.
Transformers can predict new tokens based on any number of context tokens, approximating continuous mappings with fixed resources.
problem Handling an arbitrarily large number of context tokens in transformers.
method Mathematical analysis of transformer's expressivity using Wasserstein distance and continuous mappings.
result Deep transformers are universal and can approximate continuous in-context mappings to arbitrary precision, uniformly over compact token domains.
SOM-VQ tokenizes discrete models with semantic structure and navigable topology.
problem Lack of semantic structure in vector quantized representations limits interpretable human control.
method Combines vector quantization with Self-Organizing Maps to learn discrete codebooks with explicit topology.
result SOM-VQ produces more learnable token sequences and provides an explicit navigable geometry in code space.
STRATA generates code adversarial examples efficiently without gradients.
problem Generating adversarial examples for code that retains functional meaning.
method Uses token frequency statistics to construct gradient-free adversarial examples.
result Empirically outperforms gradient-based methods with less information and effort.
Gradient-trained embeddings select important tokens in text.
problem Understanding the theoretical structure of token embeddings.
method One-layer softmax attention model with gradient descent.
result Gradient-trained embeddings align with important tokens based on frequency.
User surveys for Quality of Experience (QoE) are a critical source of information. In addition to the common "star rating" used to estimate Mean Opinion Score (MOS), more detailed survey questions (problem tokens) about specific areas provide valuable insight into the factors impacting QoE. This paper explores two aspe…
Analyzes how economic policies affect wealth distribution in Bitcoin token economy.
problem Impact of economic policies on wealth distribution in token economies.
method Eliminated noise in wealth distribution data using macroeconomic and microeconomic time series. Causality analysis between BIPs and wealth distribution data.
result Proposed a structure for economic policy taxonomy in token economies.
Researchers analyze the structure of token spaces in large language models.
problem Understanding the geometric and topological structure of token spaces in large language models.
method Developed estimators for the dimension and Ricci scalar curvature of the token subspace and applied them to three large language models.
result The token subspace is not a manifold but a stratified manifold with significantly negative Ricci curvature.
Law explains how LLMs learn to predict next tokens.
problem Understanding how LLMs process input data internally.
method Introduced a precise law governing token embeddings in LLMs.
result Each layer equally contributes to next-token prediction accuracy.
Attention mechanism learns to focus on sparse tokens efficiently.
problem Detecting weak, rare, and sparsely located features in long sequences.
method Theoretical analysis and training of a single-layer attention classifier in a sparse-token classification model.
result A single-layer attention classifier can achieve vanishing test error with logarithmic signal strength growth, unlike linear classifiers requiring linear growth.
Enhanced Sampling Scheme improves masked generative modeling.
problem Limitations of existing sampling schemes in masked non-autoregressive generative modeling.
method ESS consists of three stages: Naive Iterative Decoding, Critical Reverse Sampling, and Critical Resampling.
result ESS achieves significant performance gains in unconditional and class-conditional sampling.
Reinforcement Patching optimizes dynamic sequence patching for efficient time series forecasting.
problem Efficiently learning data-adaptive representations for long-horizon sequence data, especially continuous sequences.
method Reinforcement Patching (ReinPatch) uses reinforcement learning to optimize dynamic patching policies and sequence backbones.
result ReinPatch achieves compelling performance in time-series forecasting compared to state-of-the-art methods.
Detects potential rug pulls in Uniswap tokens before they occur.
problem Rug pulls in Uniswap, a decentralized exchange, leading to token scams.
method Collects and analyzes 20K transactions, proposes machine learning algorithms with new features.
result Achieved an accuracy of 0.9936 in detecting potential scams before they happen.
Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.
problem Sparse token selection task
method One-layer transformer trained with gradient descent
result Transformers learn sparse token selection and exhibit strong out-of-distribution length generalization
New bounds for large language models using token properties.
problem Vacuous generalization bounds for large language models.
method Martingale properties and Monarch matrices.
result Non-vacuous generalization bounds for LLMs up to 70B parameters.