Research
On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

169,341 papers · 148 categories

Trend · papers per month

1223 · Aug 202419922001200920182026
17 results for token-based

Transformers can predict new tokens based on any number of context tokens, approximating continuous mappings with fixed resources.

problem Handling an arbitrarily large number of context tokens in transformers.
method Mathematical analysis of transformer's expressivity using Wasserstein distance and continuous mappings.
result Deep transformers are universal and can approximate continuous in-context mappings to arbitrary precision, uniformly over compact token domains.

We study the problem of building generative models of natural source code (NSC); that is, source code written and understood by humans. Our primary contribution is to describe a family of generative models for NSC that have three key properties: First, they incorporate both sequential and hierarchical structure. Second…

2014-01-02abs ↗pdf ↗

Study assesses how much security restaking protocols need to pay for.

problem Determining the optimal security level for restaking protocols using token incentives.
method Expanding a model by Durvasula and Roughgarden to include strategic attackers and node operators, constructing an approximation algorithm for token-based incentives.
result Restaking protocols can be secure with proper incentive management, even against strategic adversaries.

DDPD separates generation into planning and denoising for improved efficiency.

problem Efficiently denoise corrupted data during generation.
method Separates generation into a planner and denoiser, selecting denoising positions based on corruption severity.
result DDPD outperforms traditional methods on language and image generation benchmarks.

Improved SGD bounds for machine learning models with Markovian noise.

problem Uniform high-probability bounds for SGD under PL condition with Markovian noise.
method Combining Poisson equation for Markovian noise and probabilistic induction for almost-sure bounds.
result Matching 1/k1/k decay rate for expected suboptimality.

Decoding strategies often exclude human-like tokens, creating a detectable gap in generated text.

problem Decoding strategies exclude contextually appropriate but statistically rare tokens, creating a detectable gap in generated text.
method Analysis of 1.8 million texts across 8 language models, 5 decoding strategies, and 53 hyperparameter configurations.
result 8-18% of human-selected tokens fall outside typical truncation boundaries, indicating a detectable gap.

Context2Name predicts natural names from minified code, improving code readability.

problem Minified code makes it hard to understand natural names in JavaScript.
method Combines static analysis and neural networks to predict natural names.
result Successfully predicts 47.5% of all minified identifiers in real-world code.

Non-autoregressive transformer improves speech recognition speed and accuracy.

problem Reducing inference computation cost in speech recognition.
method Two non-autoregressive transformer structures (A-CMLM and A-FMLM) trained with masked tokens and iterative prediction during inference.
result Non-autoregressive transformer can match autoregressive transformer performance with 7x speedup.

This study improves sentence embeddings from BERT models.

problem Capturing the underlying meaning of sentences using BERT models.
method Comprehensive review and testing of various sentence embedding extraction and refinement methods.
result Representation-shaping techniques significantly improve sentence embeddings from BERT-based and simple baseline models.

Enhances neural processes to learn from multiple related datasets.

problem Improving predictions from datasets with shared similarities.
method Developed the in-context in-context learning pseudo-token TNP (ICICL-TNP) to condition on both sets of datapoints and sets of datasets.
result Demonstrated the importance and effectiveness of in-context in-context learning.

New approach predicts tokens in context, explaining how ICL emerges.

problem Limited understanding of in-context learning emergence.
method Auto-regressive next-token prediction (AR-NTP) with prompt token-dependency and a two-level expectation.
result ICL emerges from the generalization of sequences and topics.

Transformers preserve support and can approximate any continuous map.

problem Understanding the mathematical properties of transformers.
method Characterizing maps between measures that can be represented as transformers and proving their properties.
result Transformers preserve support and have uniformly continuous Fréchet derivatives.