Transformer models align words through attention weights, closely approximating Optimal Transport.
problem Understanding the internal mechanism of transformer models in language processing.
method Empirical evidence and theoretical analysis of attention weights and their relation to Optimal Transport.
result Transformer models can simulate gradient descent on the dual of entropy-regularized OT problem, providing a theoretical foundation for token alignment.
Study finds no significant alignment between whitepaper claims and market structure.
problem Correlation between cryptocurrency whitepaper narratives and market behavior.
method Developed a contamination-aware pipeline for measuring structural correspondence, combining NLP classification and market statistics.
result No significant claims-market alignment detected in the sample.
nGPT learns to transfer learning rates across model dimensions and token horizons.
problem nGPT does not transfer learning rates across model size and token horizon.
method Combining numerical experiments with alignment exponents, a novel nGPT parameterization νGPT is developed.
result νGPT exhibits learning rate transfer across width, depth, and token horizon.
The paper analyzes risk spillovers between AI ETFs, AI tokens, and green markets.
problem Risk spillovers among AI ETFs, AI tokens, and green markets.
method R2 decomposition method
result AI ETFs and clean energy act as risk transmitters, while AI tokens and green assets act as receivers.
Fan tokens surged before World Cup matches, but declined during them, revealing cognitive biases.
problem Analyzing the impact of FIFA World Cup matches on fan tokens.
method Event study and intraday analysis of blockchain-based fan tokens.
result Fan tokens experienced a surge in returns six months before the World Cup, followed by a decline during the matches, revealing asymmetries in performance.
RESTA defends LLMs against jailbreaking attacks by adding random noise to embeddings.
problem Vulnerability of LLMs to jailbreaking attacks that generate harmful outputs.
method Adds random noise to embedding vectors and aggregates during token generation.
result RESTA achieves superior robustness versus utility tradeoffs compared to baseline defenses.
Foundation models fail to preserve continuous geometry, identified as the Geometric Alignment Tax.
problem Continuous geometry is lost in foundation models due to discrete categorical bottlenecks.
method Controlled ablations on synthetic systems and evaluation of 14 biological models using rate-distortion theory and MINE.
result Replacing cross-entropy with a continuous head reduces geometric distortion by up to 8.5x.
SOM-VQ tokenizes discrete models with semantic structure and navigable topology.
problem Lack of semantic structure in vector quantized representations limits interpretable human control.
method Combines vector quantization with Self-Organizing Maps to learn discrete codebooks with explicit topology.
result SOM-VQ produces more learnable token sequences and provides an explicit navigable geometry in code space.
Attention mechanism learns to focus on sparse tokens efficiently.
problem Detecting weak, rare, and sparsely located features in long sequences.
method Theoretical analysis and training of a single-layer attention classifier in a sparse-token classification model.
result A single-layer attention classifier can achieve vanishing test error with logarithmic signal strength growth, unlike linear classifiers requiring linear growth.
Gradient-trained embeddings select important tokens in text.
problem Understanding the theoretical structure of token embeddings.
method One-layer softmax attention model with gradient descent.
result Gradient-trained embeddings align with important tokens based on frequency.
Unified model integrates text and time series for financial forecasting.
problem Challenges in integrating complementary modalities for improved forecasting.
method Modality-specific experts and cross-modal alignment framework.
result State-of-the-art performance on financial forecasting task.
Develops STC for sequential data with missing labels.
problem Learning from partially labeled and unsegmented sequential data.
method Introduces Star Temporal Classification (STC) using a star token and GTN framework.
result Recover most of supervised baseline performance with up to 70% missing labels.
Study reveals AI's spontaneous topic changes in text prediction.
problem AI's inability to spontaneously switch topics like humans.
method Defined topic as Token Priority Graphs (TPGs) and analyzed self-attention models.
result AI can only switch topics if lower-priority tokens outnumber higher-priority ones.
Sparse text alignments learned via optimal transport improve model explainability.
problem Building self-explaining models by selecting relevant text pieces.
method Employing optimal transport to find minimal cost alignments, introducing constrained variants for sparsity.
result Sparse and interpretable alignments achieved, preserving prediction accuracy.
Study finds whitepaper narratives do not predict market factor structure.
problem Predicting market behavior from cryptocurrency whitepaper claims.
method Zero-shot NLP classification combined with CP tensor decomposition of market data.
result Weak alignment between whitepaper claims and market statistics and latent factors.
New approach predicts tokens in context, explaining how ICL emerges.
problem Limited understanding of in-context learning emergence.
method Auto-regressive next-token prediction (AR-NTP) with prompt token-dependency and a two-level expectation.
result ICL emerges from the generalization of sequences and topics.
A new algorithm RTO integrates DPO and PPO for RLHF, improving performance.
problem Learning from sparse, sentence-level rewards in RLHF.
method Modeling RLHF as MDP, introducing RTO that learns token-wise reward function.
result RTO outperforms PPO on benchmarks by 7.5 points on AlpacaEval 2 and 4.1 points on Arena-Hard.
CARDS improves decoding efficiency and alignment quality for LLMs.
problem Efficiency bottlenecks in decoding-time alignment for LLMs.
method Cascade Reward Sampling (CARDS) with segment-level rejection sampling and uncertainty-based segmentation.
result Significant improvement in decoding efficiency and alignment quality.
This paper examines how different decoding algorithms for LLMs align with various goals.
problem Consistency of decoding algorithms with different goals in LLMs.
method Analysis of greedy, lookahead, random sampling, and temperature-scaled random sampling algorithms.
result Random sampling is consistent with the true probability distribution, but other goals require optimal algorithms for specific probability distributions.
Randomly initialized transformers show extreme token preferences.
problem Structural biases in randomly initialized transformers.
method Dissection of transformer architecture at initialization.
result Initialization-induced biases persist throughout training.
Paper proposes AXE loss for non-autoregressive machine translation, improving performance.
problem Challenges in training non-autoregressive models due to lack of autoregressive factors and cross entropy loss penalties.
method Proposes aligned cross entropy (AXE) loss function using a differentiable dynamic program for better word order alignment.
result AXE-based training improves performance on major WMT benchmarks and sets a new state of the art for non-autoregressive models.
Simple adaptive attacks can jailbreak state-of-the-art safety-aligned language models.
problem Vulnerability of safety-aligned language models to simple adaptive attacks.
method Designing adversarial prompt templates, random search on suffixes, and transfer/prefilling attacks.
result 100% attack success rate on various models including GPT-4, Vicuna, Mistral, and Claude.
This paper shows ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.
problem Understanding the lookahead capabilities of next-token prediction models.
method Unified view of ARMs and EBMs, establishing a bijection and deriving equivalence.
result ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.
Framework controls uncertainty in LLMs without labels or probabilities.
problem Managing uncertainty in black-box LLMs without token-level probability or true labels.
method Integrates generative models, UCP, and conformal alignment to control uncertainty.
result Achieves close-to-nominal coverage and tighter thresholds than split UCP.
Transformers forecast time series in-context, improving efficiency and performance.
problem Overfitting and limited performance in time series forecasting.
method Reformulate time series forecasting as input tokens, aligning with in-context learning mechanisms.
result Consistently better performance across various settings (full-data, few-shot, zero-shot).
Language evolves over time in many ways relevant to natural language processing tasks. For example, recent occurrences of tokens 'BERT' and 'ELMO' in publications refer to neural network architectures rather than persons. This type of temporal signal is typically overlooked, but is important if one aims to deploy a mac…
Thinking LLMs struggle with stock prediction, especially as data complexity increases.
problem Evaluating the performance of 'thinking' LLMs in stock prediction, especially under varying levels of cross-sectional complexity.
method Rolling 48m/1m walk-forward evaluation, comparing direct LLMs, TLLMs, and classical learners on cross-sectional ranking loss, MSE, and backtests with transaction costs.
result TLLMs' ranking quality deteriorates as cross-sectional complexity grows, while direct LLMs remain stable.
Training autoregressive models to better predict under the test metric, instead of maximizing the likelihood, has been reported to be beneficial in several use cases but brings additional complications, which prevent wider adoption. In this paper, we follow the learning-to-search approach (Daumé III et al., 2009; Leblo…
PAC-Bayesian theory improves text-to-image models by enforcing alignment and generalization.
problem Text-to-image models struggle with complex prompts, misaligning modifiers and neglecting certain elements.
method Proposes a Bayesian approach with custom priors over attention distributions to enforce desirable properties.
result Achieves state-of-the-art results across multiple metrics on standard benchmarks.
RIVCoin stabilizes cryptocurrency portfolios through a DAO and redistributes income.
problem Stabilizing cryptocurrency value and aligning incentives for all users.
method Decentralized DAO, diversified reserves, and income redistribution.
result Aligns incentives for wealthier users to stabilize smaller users' risk.
Framework calculates positional influence in causal residual Transformers.
problem Understanding positional influence in causal residual Transformers.
method Adjoint-sensitivity framework for positional influence in causal residual Transformers.
result Exact evolution of adjoint-energy influence density and decomposition into residual transmission, nonlocal Volterra, and local channels.
Proposes a proportional masking strategy for better tabular data imputation.
problem Heterogeneity of tabular data disrupts uniform random masking in MAEs.
method Computes missingness statistics, generates proportional masks, uses MLP token mixing.
result Proportional masking preserves missingness distribution, improves imputation performance.
GMC benchmark isolates retrieval in Transformers, revealing max-margin alignment.
problem Understanding how Transformers develop match-and-copy behavior on natural data.
method Introducing Gaussian Match-and-Copy (GMC) as a minimalist benchmark.
result Gradient descent drives parameters to diverge while aligning with max-margin separator.
Discrete Flow Maps bypass sequential prediction limits for parallel text generation.
problem Sequential autoregressive prediction limits large language model speed.
method Flow Maps compress generative trajectories into single-step mappings.
result Discrete Flow Maps surpass previous state-of-the-art results in discrete flow modeling.
A new topology design improves zero-shot classification performance in contrastive learning.
problem Improving zero-shot classification performance in contrastive visual-textual alignment.
method Proposed an alternative topology design using multiple class tokens and an oblique manifold with negative inner product.
result Improves zero-shot classification performance by an average of 6.1%.
Neural sequence generation is typically performed token-by-token and left-to-right. Whenever a token is generated only previously produced tokens are taken into consideration. In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into consideration, …
Due to the unparallelizable nature of the autoregressive factorization, AutoRegressive Translation (ART) models have to generate tokens sequentially during decoding and thus suffer from high inference latency. Non-AutoRegressive Translation (NART) models were proposed to reduce the inference time, but could only achiev…
Spectral measurements reveal hidden representation geometry in language model training.
problem Hidden internal representation in language model training is hard to examine.
method Empirical protocol using activation covariance and per-sample gradient SVD spectra.
result Batch size affects representation geometry, and activation spectra predict token efficiency.
Training-free source selection for LLM families with shared vocabularies
problem Source selection for LLM families with shared vocabularies
method Fisher alignment at vocabulary scale
result Fisher alignment is a cosine between kernel mean embeddings in the joint activation-error space
Paper reduces vocabulary losslessly for language model cooperation.
problem Language models struggle to cooperate with different tokenizations.
method Established a theoretical framework for lossless vocabulary reduction.
result Efficiently converts models with different tokenizations to cooperate with maximal common vocabulary.
This research improves capital efficiency and impermanent loss in cryptocurrency markets using multi-token trading pools.
problem Poor impermanent loss and capital efficiency in automated market makers.
method Analysis and construction of a multi-token token proactive market maker (MPMM).
result MPMM shows better impermanent loss and capital efficiency than comparable market makers.
iREPA shows spatial structure, not global semantic, drives generation performance in REPA.
problem Understanding what aspect of the target representation matters for generation.
method Empirical analysis of 27 vision encoders, two modifications to REPA.
result Spatial structure, not global semantic, drives generation performance.
This study examines whether tokenized assets improve liquidity and finds significant differences across categories.
problem Improving liquidity for real-world assets through tokenization.
method Examined tokenized real-world assets using Ethereum-based data, measuring liquidity through turnover, active addresses, and active-month indicator.
result Gold-backed tokens show more persistent on-chain activity than Treasury and private-credit-related products, but asset value alone does not reliably predict liquidity.
Study examines revenue from scam tokens on Ethereum, revealing key characteristics and market factors.
problem Revenue from scam tokens on Ethereum blockchain.
method Empirical analysis of Uniswap, examining characteristics and market factors.
result Revenue from scam tokens is influenced by market economic factors and community engagement.
MAESTRO improves multimodal learning for dynamic time series with adaptive attention and robustness.
problem Challenges in multimodal learning, especially in healthcare and daily living.
method Dynamic intra- and cross-modal interactions, symbolic tokenization, adaptive attention budgeting, sparse cross-modal attention, MoE mechanism.
result Average relative improvements of 4% and 8% over existing multimodal and multivariate approaches, respectively, under complete observations.
Blockchain-based exchanges adopt based on token pair volatility and personal use.
problem Token value loss and arbitrage issues in decentralized exchanges.
method Investigation of Automated Market Makers (AMMs) using transaction-level data.
result AMMs are adopted for high personal use or highly correlated token price movements.
Blockchain fan tokens boost sports fan engagement by 50%.
problem Low fan engagement in sports decisions.
method Analyzed 3,576 fan token polls to identify determinants of participation.
result Fan tokens engage 4,003 participants per poll, 50% of token holders.
Study SGD dynamics in sequence models, revealing training phases and influence of sequence length.
problem Understanding SGD in sequence models like attention networks.
method Derived closed-form population loss and analyzed SGD dynamics for SSI models.
result Two distinct training phases: escape from uninformative initialization and alignment with target subspace.