Bidirectional sequence generation improves performance in conversational tasks.
problem Neural sequence generation typically considers only past tokens, limiting performance.
method Introduced placeholder tokens that can consider both past and future tokens in sequence generation.
result Bidirectional model outperforms competitive baselines on conversational tasks.
DOS improves language model generation by considering inter-token dependencies.
problem Lack of sequence-level information and inter-token dependencies in existing decoding strategies.
method Dependency-Oriented Sampler (DOS) that uses attention matrices to approximate inter-token dependencies.
result DOS consistently achieves superior performance on code generation and mathematical reasoning tasks.
This research improves capital efficiency and impermanent loss in cryptocurrency markets using multi-token trading pools.
problem Poor impermanent loss and capital efficiency in automated market makers.
method Analysis and construction of a multi-token token proactive market maker (MPMM).
result MPMM shows better impermanent loss and capital efficiency than comparable market makers.
LLM-as-a-service prices vary arbitrarily due to tokenization multiplicity.
problem Arbitrary price variation in LLM-as-a-service due to multiple tokenizations of the same output.
method Introduce canonical generation to restrict LLMs to unique tokenizations and develop an efficient sampling algorithm.
result Our sampling algorithm for canonical generation solves tokenization multiplicity and maintains comparable performance and runtime to standard sampling.
Paper reduces vocabulary losslessly for language model cooperation.
problem Language models struggle to cooperate with different tokenizations.
method Established a theoretical framework for lossless vocabulary reduction.
result Efficiently converts models with different tokenizations to cooperate with maximal common vocabulary.
Proving that next-token prediction makes language models generate coherent long documents.
problem Understanding why language models generate coherent documents despite focusing on next-token prediction.
method Proving the power of next-token prediction in learning longer-range structure using Recurrent Neural Networks (RNN).
result Optimizing next-token prediction in RNNs yields a model that closely approximates the training distribution, even for long-range coherence.
SOM-VQ tokenizes discrete models with semantic structure and navigable topology.
problem Lack of semantic structure in vector quantized representations limits interpretable human control.
method Combines vector quantization with Self-Organizing Maps to learn discrete codebooks with explicit topology.
result SOM-VQ produces more learnable token sequences and provides an explicit navigable geometry in code space.
Method reveals hidden token embeddings of large language models.
problem Understanding hidden token embeddings in large language models.
method Structured prompts to expose token input embeddings up to homeomorphism.
result Mathematical proof for generic LLMs shows effectiveness of method.
Decoding strategies often exclude human-like tokens, creating a detectable gap in generated text.
problem Decoding strategies exclude contextually appropriate but statistically rare tokens, creating a detectable gap in generated text.
method Analysis of 1.8 million texts across 8 language models, 5 decoding strategies, and 53 hyperparameter configurations.
result 8-18% of human-selected tokens fall outside typical truncation boundaries, indicating a detectable gap.
Study examines revenue from scam tokens on Ethereum, revealing key characteristics and market factors.
problem Revenue from scam tokens on Ethereum blockchain.
method Empirical analysis of Uniswap, examining characteristics and market factors.
result Revenue from scam tokens is influenced by market economic factors and community engagement.
STRATA generates code adversarial examples efficiently without gradients.
problem Generating adversarial examples for code that retains functional meaning.
method Uses token frequency statistics to construct gradient-free adversarial examples.
result Empirically outperforms gradient-based methods with less information and effort.
New bounds for large language models using token properties.
problem Vacuous generalization bounds for large language models.
method Martingale properties and Monarch matrices.
result Non-vacuous generalization bounds for LLMs up to 70B parameters.
Enhanced Sampling Scheme improves masked generative modeling.
problem Limitations of existing sampling schemes in masked non-autoregressive generative modeling.
method ESS consists of three stages: Naive Iterative Decoding, Critical Reverse Sampling, and Critical Resampling.
result ESS achieves significant performance gains in unconditional and class-conditional sampling.
Generative Link Sequence Modeling predicts future links in evolving networks.
problem Predicting future links in networks with evolving structures.
method Sequence modeling framework with self-tokenization to capture temporal link formation patterns.
result GLSM achieves best performance on AUC metrics compared to existing methods.
This paper investigates how the order of token generation affects language model quality.
problem The influence of token generation order on model quality in autoregressive language models.
method A novel two-pass language model that produces partially-filled sentence templates and then fills in missing tokens. Various strategies for structuring these two passes are compared.
result The most effective strategy generates function words in the first pass followed by content words in the second.
Improved RL for TBGs by pruning irrelevant tokens and bootstrapping.
problem RL methods fail to generalize in TBGs with small data.
method CREST for irrelevant token removal, bootstrapped Q-learning.
result Improved generalization in unseen TextWorld games.
Researchers analyze the structure of token spaces in large language models.
problem Understanding the geometric and topological structure of token spaces in large language models.
method Developed estimators for the dimension and Ricci scalar curvature of the token subspace and applied them to three large language models.
result The token subspace is not a manifold but a stratified manifold with significantly negative Ricci curvature.
xVal tokenizes numbers continuously for better scientific model training.
problem Lack of continuous numerical tokenization for scientific datasets in LLMs.
method xVal: Continuous numerical tokenization strategy.
result xVal outperforms other numerical tokenization methods on scientific datasets.
Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.
problem Sparse token selection task
method One-layer transformer trained with gradient descent
result Transformers learn sparse token selection and exhibit strong out-of-distribution length generalization
Generative model predicts financial market order flow with high accuracy.
problem Creating realistic order flow models for financial markets.
method Token-level autoregressive generative model using deep state space layers.
result Model generates high-quality order flow data with low perplexity.
This paper analyzes shallow ViTs, providing sample complexity and SGD behavior insights.
problem Theoretical understanding of shallow ViTs, especially their sample complexity and SGD behavior.
method Data model with label-relevant and label-irrelevant tokens, theoretical analysis of shallow ViT training.
result Characterization of sample complexity for zero generalization error in shallow ViTs.
This study examines whether tokenized assets improve liquidity and finds significant differences across categories.
problem Improving liquidity for real-world assets through tokenization.
method Examined tokenized real-world assets using Ethereum-based data, measuring liquidity through turnover, active addresses, and active-month indicator.
result Gold-backed tokens show more persistent on-chain activity than Treasury and private-credit-related products, but asset value alone does not reliably predict liquidity.
Paper presents token-level explanations for NLI without annotated data.
problem Generating explanations for NLI models on sentence pairs.
method Simple LSTM architecture with LIME, Anchor, and MIL methods.
result MIL-based method faster but less accurate than black-box methods.
New approach predicts tokens in context, explaining how ICL emerges.
problem Limited understanding of in-context learning emergence.
method Auto-regressive next-token prediction (AR-NTP) with prompt token-dependency and a two-level expectation.
result ICL emerges from the generalization of sequences and topics.
Telescope detects LLM generated text by measuring token repetition probability.
problem Distinguishing LLM generated text from human writing.
method Telescope Perplexity, evaluating token repetition probability.
result Telescope Perplexity enables effective zero-shot LLM detection.
Blockchain-based exchanges adopt based on token pair volatility and personal use.
problem Token value loss and arbitrage issues in decentralized exchanges.
method Investigation of Automated Market Makers (AMMs) using transaction-level data.
result AMMs are adopted for high personal use or highly correlated token price movements.
This paper explores how Transformers predict next tokens in autoregressive tasks.
problem Understanding the success of Transformers in autoregressive learning.
method Trained a Transformer on a next-token prediction task, focusing on commuting orthogonal matrices.
result Trained Transformers can be seen as implementing gradient descent for a specific objective function.
Blockchain fan tokens boost sports fan engagement by 50%.
problem Low fan engagement in sports decisions.
method Analyzed 3,576 fan token polls to identify determinants of participation.
result Fan tokens engage 4,003 participants per poll, 50% of token holders.
Token economics improves energy systems with incentives and efficiency.
problem Traditional energy systems have inefficiencies and lack incentives.
method Integrating token economy and blockchain technology.
result Token economic systems enhance energy efficiency and reduce emissions.
Paper introduces a method to assess liquidity risk in meme tokens using entity-linked address analysis.
problem High market volatility and vulnerability to manipulation in meme tokens.
method Multi-dimensional approach integrating fund flow analysis, behavioral similarity, and anomalous transaction detection.
result Significant disparities between apparent and actual liquidity in meme token markets.
A new sampler for FLMs improves token-level decoding controls.
problem Sampling from FLMs using standard methods collapses marginals and produces invalid sequences.
method Samples clean one-hot endpoints from FLM token marginals and uses Ornstein-Uhlenbeck bridges conditioned on these endpoints.
result The method preserves token-wise posterior-predictive marginals and improves quality-diversity tradeoff.
LADD models improve discrete diffusion for faster language generation.
problem Practical discrete diffusion models ignore cross-token dependencies, degrading performance.
method Introduces a learnable auxiliary latent channel, diffusing over the joint (token, latent) space.
result LADD models yield improvements on unconditional generation metrics.
MaskGAN fills in text blanks more flexibly and effectively.
problem Handling complex manipulations on given tokens in conditional text generation.
method Decomposed conditional text generation into make-a-blank and fill-in-the-blank tasks, and used hierarchical multi-agent reinforcement learning.
result The model produces realistic texts without compromising quality and diversity.
Study predicts success of crypto-tokens on Pump.fun platform.
problem Identify factors affecting the success of new crypto-tokens.
method Build predictive models using bonding curve mechanism and structural/behavioral variables.
result Conditional variables significantly improve the predictive power of token success.
Expands MLM by masking token positions, improving performance and convergence.
problem Improving language model performance and convergence.
method Masking token positions along with [MASK] tokens, using a fully connected classifier stage.
result Shows .3% improvement and 50% faster convergence for BERT Base with position masking.
Detects potential rug pulls in Uniswap tokens before they occur.
problem Rug pulls in Uniswap, a decentralized exchange, leading to token scams.
method Collects and analyzes 20K transactions, proposes machine learning algorithms with new features.
result Achieved an accuracy of 0.9936 in detecting potential scams before they happen.
This paper compares token and equity financing for startups.
problem Understanding differences in return rates between token and equity financing.
method Developed a three-period model to analyze liquidity and return differences.
result Entrepreneurs can achieve higher payoffs by issuing tokens, especially for risk-averse investors with liquidity needs.
QA-Token improves tokenization for noisy data, boosting model performance.
problem Tokenization ignores data quality, limiting model effectiveness on noisy corpora.
method QA-Token combines signal quality with vocabulary construction through bilevel optimization and reinforcement learning.
result QA-Token achieves state-of-the-art performance on genomic and financial datasets.
Minimal token perturbations reveal how Transformer models process information.
problem Understanding information propagation in Transformer models for interpretability.
method Study of minimal token perturbations on embedding space.
result Rare tokens cause larger shifts, and input information mixes deeper.
Generative model predicts NFT collection transactions based on early history.
problem Predict future transactions of newly minted NFT collections.
method Unsupervised learning to extract contexts, then generate future transactions.
result Projected market value of new NFT collections.
New insights show stochastic initialization prevents token clustering in deep Transformers.
problem Understanding token dynamics in deep stochastic Transformers.
method Analysis of deep Transformers with random initialization noise, proving convergence to an interacting-particle system on the sphere.
result Initialization noise prevents token clustering, leading to antipodal formations.
The paper analyzes risk spillovers between AI ETFs, AI tokens, and green markets.
problem Risk spillovers among AI ETFs, AI tokens, and green markets.
method R2 decomposition method
result AI ETFs and clean energy act as risk transmitters, while AI tokens and green assets act as receivers.
Paper shows SGD can learn RNNs efficiently for certain functions.
problem Understanding what concept class RNNs can learn and how efficiently.
method Vanilla stochastic gradient descent (SGD) approach.
result RNNs can learn some functions efficiently with polynomial complexity in input length.
Gradient descent biases linear models in next-token prediction towards data entropy.
problem Optimization bias in next-token prediction models.
method Analysis of gradient descent on linear models with sparse conditional distributions.
result Gradient descent selects parameters that equate token logits differences to log-odds in the data subspace.
Global watermark for diffusion language models decouples detection from local contexts.
problem Watermarking in diffusion language models is challenging due to joint sampling of distributions over many unresolved positions.
method Proposes a global vector-valued sketch representation to control watermarking in masked diffusion language models.
result The method decouples detection from local contexts, resulting in an order-agnostic statistic and robustness.
Study reveals risks of investing in new crypto-tokens in decentralized exchanges.
problem Risks associated with investing in newly created tokens in decentralized exchanges.
method Analysis of financial impact, market dynamics, profitability, and liquidity manipulations.
result Significant market liquidity trapped in honeypots, reducing market efficiency and misleading investors.
Tokenized RWAs face liquidity issues despite promising markets.
problem Low trading volumes and limited investor participation in tokenized assets.
method Empirical analysis of tokenized real estate, private credit, and treasury funds.
result Most tokenized assets exhibit low transfer activity and limited secondary trading.
This paper optimizes survey questions to reduce bias and select key tokens for QoE analysis.
problem Reducing bias in user surveys and selecting informative tokens for Quality of Experience analysis.
method Randomized question order and greedy submodular maximization for selecting tokens.
result Randomizing token order can significantly reduce bias, and a subset of 30% tokens captures 94% of the information.