Bayesian nonparametric CMS improves frequency estimation for power-law data.
problem Estimating frequencies of low-frequency tokens in power-law data streams.
method Developed a learning-augmented count-min sketch using a normalized inverse Gaussian process prior.
result The approach achieves remarkable performance in estimating low-frequency tokens.
Enhances CMS with Bayesian nonparametrics for better low-frequency token estimation.
problem Improving frequency estimation of low-frequency tokens in data streams.
method Integrates Bayesian nonparametrics (Pitman-Yor process) into CMS for more accurate frequency estimation.
result CMS-PYP outperforms CMS and CMS-DP in estimating low-frequency tokens.
STRATA generates code adversarial examples efficiently without gradients.
problem Generating adversarial examples for code that retains functional meaning.
method Uses token frequency statistics to construct gradient-free adversarial examples.
result Empirically outperforms gradient-based methods with less information and effort.
Gradient-trained embeddings select important tokens in text.
problem Understanding the theoretical structure of token embeddings.
method One-layer softmax attention model with gradient descent.
result Gradient-trained embeddings align with important tokens based on frequency.
Minimal token perturbations reveal how Transformer models process information.
problem Understanding information propagation in Transformer models for interpretability.
method Study of minimal token perturbations on embedding space.
result Rare tokens cause larger shifts, and input information mixes deeper.
Generative model predicts financial market order flow with high accuracy.
problem Creating realistic order flow models for financial markets.
method Token-level autoregressive generative model using deep state space layers.
result Model generates high-quality order flow data with low perplexity.
AaSP improves audio self-supervised learning by addressing aliasing issues.
problem Alias issues in audio spectrogram transformers.
method AaSP combines aliasing-aware patch representation, teacher-student masked modeling, cross-attention predictor, and contrastive regularization.
result AaSP learns more stable representations that integrate high-frequency cues.
Decoding strategies often exclude human-like tokens, creating a detectable gap in generated text.
problem Decoding strategies exclude contextually appropriate but statistically rare tokens, creating a detectable gap in generated text.
method Analysis of 1.8 million texts across 8 language models, 5 decoding strategies, and 53 hyperparameter configurations.
result 8-18% of human-selected tokens fall outside typical truncation boundaries, indicating a detectable gap.
ByteGen models LOB dynamics without tokenization, achieving realistic market metrics.
problem Modeling high-frequency LOB dynamics in finance.
method Autoregressive next-byte prediction on packed binary data, using H-Net architecture.
result Successfully reproduces stylized facts of financial markets.
Paper develops a risk scoring framework for tokenized RWA markets.
problem Tokenized assets may not reflect true risk due to illiquidity and concentration.
method Develops a risk scoring framework based on observable indicators.
result Assets with limited transfer activity and concentrated ownership have high empirical risk.
Moirai-MoE improves time series forecasting by automatically specializing tokens without human-defined frequency.
problem Unified training on time series data remains challenging due to heterogeneity and non-stationarity.
method Uses sparse mixture of experts (MoE) within Transformers to automatically specialize tokens for diverse time series patterns.
result Moirai-MoE outperforms existing foundation models in both in-distribution and zero-shot scenarios.
The study shows how to accurately estimate embedding vectors in high dimensions.
problem How to accurately estimate embedding vectors in high-dimensional spaces.
method A simple probability model and a variant of low-rank approximate message passing (AMP) method.
result The AMP approach enables precise predictions of the accuracy of the estimation in certain high-dimensional limits.
Analyzes how BPE tokenisation affects corpus statistics and model entropy in transformer models.
problem Understanding how natural language properties relate to tokenisation schemes in transformer models.
method Analyzes Shannon entropy of corpora under Zipfian distribution, investigates BPE transformations, trains language models, and uses attention diagnostics.
result Transformer models trained on BPE-tokenised corpora increasingly agree with Zipfian predictions as BPE depth increases, indicating reduced local token dependencies.
The paper explores how gradient descent trains associative memories, revealing oscillations and convergence issues.
problem Training dynamics of associative memories in overparameterized and underparameterized settings.
method Reduction to particle system dynamics, theory, and experiments.
result Oscillatory transitory regimes and benign loss spikes in overparameterized settings, suboptimal memorization in underparameterized settings.
Enhances non-life insurance pricing models using transformer models.
problem Improving predictive power of non-life insurance pricing models.
method Enhances actuarial non-life models with transformer models for tabular data.
result Transformer models outperform benchmark models in claim frequency prediction.
We empirically verify that the market capitalisations of coins and tokens in the cryptocurrency universe follow power-law distributions with significantly different values, with the tail exponent falling between 0.5 and 0.7 for coins, and between 1.0 and 1.3 for tokens. We provide a rationale for this, based on a simpl…
Detects corruption in agentic models during execution.
problem Inconsistent context, retrieval errors, or adversarial inputs corrupt intermediate steps of reasoning chains.
method Analyzes token graphs induced by attention and computes spectral statistics to emit accept/reject signals.
result A single threshold on the high frequency energy ratio optimally detects context inconsistency in agentic models.
Study analyzes AI's impact on firms, markets, and workers using large language model data.
problem Understanding AI's effect on firms, markets, and workers.
method Used 380 trillion tokens from 400+ large language models to analyze AI's impact.
result Firms with higher AI exposure earn higher returns, creating an AI premium.
Dynamic Vocabulary Pruning stabilizes LLM training by removing low-probability tokens.
problem Training Large Language Models (LLMs) with Reinforcement Learning (RL) causes numerical divergence between inference and training.
method Dynamic Vocabulary Pruning (DVP) constrains the RL objective to a safe vocabulary that excludes low-probability tokens.
result DVP stabilizes training by reducing systematic bias introduced by the extreme tail of the token distribution.
This research tackles uncertainty estimation in autoregressive structured prediction tasks.
problem Ensuring safety and robustness of AI systems through accurate uncertainty estimation.
method Develops a unified probabilistic ensemble-based framework for token-level and sequence-level uncertainty estimation.
result Provides baselines for error and out-of-domain detection on translation and speech recognition datasets.
Improved language models learn complex distributions using Fourier series.
problem Capturing continuous structure in discrete token distributions.
method Introducing a Fourier head layer to model continuous structures.
result Significant improvements in performance across various tasks.
Researchers analyze the structure of token spaces in large language models.
problem Understanding the geometric and topological structure of token spaces in large language models.
method Developed estimators for the dimension and Ricci scalar curvature of the token subspace and applied them to three large language models.
result The token subspace is not a manifold but a stratified manifold with significantly negative Ricci curvature.
User surveys for Quality of Experience (QoE) are a critical source of information. In addition to the common "star rating" used to estimate Mean Opinion Score (MOS), more detailed survey questions (problem tokens) about specific areas provide valuable insight into the factors impacting QoE. This paper explores two aspe…
Proposes a conservative LR estimator for infrequent data near a frequency threshold.
problem Overestimation of likelihood ratios for infrequent data near a frequency threshold.
method Conservative likelihood ratio estimator for frequencies slightly above a threshold.
result Improves prediction accuracy in named entity context prediction.
Two new estimators reduce costs and improve accuracy for EHR outcome prediction.
problem Sparse estimate distributions, high computational cost, and high sampling variance in EHR outcome prediction.
method Proposed SCOPE and REACH estimators that leverage next-token probability distributions.
result SCOPE and REACH match Monte Carlo accuracy with token reductions of 2.5-3.4 times and variance guarantees.
NFM models time-series data directly in the Fourier domain, achieving state-of-the-art performance.
problem Traditional time-series analysis focuses on the time domain, limiting flexibility.
method NFM models time-series data in the Fourier domain, using frequency extrapolation and interpolation.
result NFM achieves state-of-the-art performance on various time-series tasks.
CATR rationalizes text data to stabilize causal effect estimation.
problem Observational positivity violation in high-dimensional text data.
method Confounding-Aware Token Rationalization (CATR) selects necessary subset of tokens.
result CATR yields more accurate and stable causal effect estimates.
Model compression is essential for serving large deep neural nets on devices with limited resources or applications that require real-time responses. As a case study, a state-of-the-art neural language model usually consists of one or more recurrent layers sandwiched between an embedding layer used for representing inp…
Frequency estimation is a fundamental problem in signal processing, with applications in radar imaging, underwater acoustics, seismic imaging, and spectroscopy. The goal is to estimate the frequency of each component in a multisinusoidal signal from a finite number of noisy samples. A recent machine-learning approach u…
Enhances uncertainty estimation in medical image segmentation.
problem Frequency-related noise in medical imaging leads to biased uncertainty estimates.
method Extends MC-Dropout to the frequency domain for better uncertainty estimation.
result MC-Frequency Dropout improves calibration and uncertainty in semantic segmentation.
Neural sequence generation is typically performed token-by-token and left-to-right. Whenever a token is generated only previously produced tokens are taken into consideration. In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into consideration, …
Stochastic methods improve data assimilation with high-frequency sensor data.
problem Computational challenges in data assimilation with high-frequency sensor data.
method Adapted stochastic approximation methods to handle high-frequency observations.
result Produces high-quality estimates using all observations without compromising statistical accuracy.
EPSTE: A geometric token and deep learning approach to estimating transfer entropy in neuroimaging time series
problem Inferring directed interactions between neural systems from EEG and MEG
method Reframing TE estimation as a learnable problem operating on structured symbolic representations
result EPSTE achieves near-perfect recovery of ground-truth directed structure and significantly lower absolute error than the baseline
Paper proves estimates for heat and conjugate heat equations under Ricci flow, leading to monotonicity of parabolic frequencies.
problem Establishing estimates for heat and conjugate heat equations under Ricci flow.
method Proving matrix Li-Yau-Hamilton estimates for positive solutions to the heat and conjugate heat equations coupled with Ricci flow.
result Monotonicity of parabolic frequencies established up to correction factors.
In target tracking, the estimation of an unknown weaving target frequency is crucial for improving the miss distance. The estimation process is commonly carried out in a Kalman framework. The objective of this paper is to examine the potential of using neural networks in target tracking applications. To that end, we pr…
Model analyzes Proof-of-Stake network dynamics and speculative capital effects on token prices.
problem Understanding and managing price dynamics in Proof-of-Stake networks.
method Developed an open-economy macroeconomic model to analyze Proof-of-Stake dynamics and speculative capital effects.
result Speculative capital can shift staked-token ownership, potentially improving consensus decentralization.
The paper introduces a frequency-domain estimator for low-order systems from noisy data.
problem Estimating frequency responses of low-order systems from noisy measurements.
method Uses a quadratic data-fitting term regularized by the nuclear norm of a Loewner matrix, subject to a convex stability constraint.
result Proves a finite-sample error bound and extends it to all frequencies through rational interpolation.
Study examines NFT market dynamics using correlation and noise analysis.
problem Understanding correlations and noise in NFT market.
method Used detrended correlation coefficient and correlation matrix analysis.
result Correlation strength in NFT market is lower than in cryptocurrency markets.
The study explores how Transformers predict the next token in a sequence.
problem Understanding the mechanism behind Transformers' autoregressive learning ability.
method Exploring the approximation ability of Transformers for next-token prediction through specific instances and a causal kernel descent method.
result Transformer models can learn context-dependent functions f for next-token prediction based on past and current observations. The paper distinguishes between conditional and marginal processes in language models and discusses conditions for usefulness.
problem The conditional nature of language models trained on observed sequences and the need for marginal text-only processes.
method Distinguishing between full conditional language process, marginal text-only process, and model-induced distribution; analyzing assumptions of stationarity and ergodicity.
result The marginal text-only law is useful only when the observed prefix is an approximately sufficient statistic for the latent circumstances relevant to continuation.
Paper introduces a new IV regression method for mixed-frequency data.
problem Estimating high-dimensional slope parameters in mixed-frequency data.
method Tikhonov-regularized estimator for high-dimensional linear IV regression.
result High-dimensional slope parameter can be accurately estimated using a low-frequency instrumental variable.
Machine learning approaches to multi-label document classification have to date largely relied on discriminative modeling techniques such as support vector machines. A drawback of these approaches is that performance rapidly drops off as the total number of labels and the number of labels per document increase. This pr…
Paper reduces vocabulary losslessly for language model cooperation.
problem Language models struggle to cooperate with different tokenizations.
method Established a theoretical framework for lossless vocabulary reduction.
result Efficiently converts models with different tokenizations to cooperate with maximal common vocabulary.
This research improves capital efficiency and impermanent loss in cryptocurrency markets using multi-token trading pools.
problem Poor impermanent loss and capital efficiency in automated market makers.
method Analysis and construction of a multi-token token proactive market maker (MPMM).
result MPMM shows better impermanent loss and capital efficiency than comparable market makers.
This study examines whether tokenized assets improve liquidity and finds significant differences across categories.
problem Improving liquidity for real-world assets through tokenization.
method Examined tokenized real-world assets using Ethereum-based data, measuring liquidity through turnover, active addresses, and active-month indicator.
result Gold-backed tokens show more persistent on-chain activity than Treasury and private-credit-related products, but asset value alone does not reliably predict liquidity.
Study examines revenue from scam tokens on Ethereum, revealing key characteristics and market factors.
problem Revenue from scam tokens on Ethereum blockchain.
method Empirical analysis of Uniswap, examining characteristics and market factors.
result Revenue from scam tokens is influenced by market economic factors and community engagement.
The paper analyzes dynamics of momentum in high dimensions with sparse updates.
problem Theoretical analysis of momentum dynamics in high-dimensional sparse settings.
method Theoretical analysis of two models: least squares with sparse inputs and logistic regression with a rare class.
result Characterization of high-dimensional limits of momentum dynamics and phase structure.
New formula for instantaneous frequency in unbalanced systems.
problem Estimating frequency in unbalanced electrical systems.
method Utilizes affine differential geometry to link frequency and voltage derivatives.
result Proposes a new formula for instantaneous frequency estimation.