Transformers learn chain-of-thought reasoning for longer problems, proving length generalization.
problem Challenging problems require deeper reasoning, but how do models generalize this to longer tasks?
method Theoretical analysis of transformers on synthetic state-tracking tasks, proving length generalization through attention concentration.
result Transformers can learn chain-of-thought reasoning for longer problems, proving length generalization.
A method for learning with autoregressive chain-of-thoughts.
problem Learning prompt-to-answer mappings from sequence-to-next-token generators.
method Iterating a fixed, time-invariant generator for multiple steps to generate a chain-of-thought, then taking the final token as the answer.
result Universal representability and computationally tractable chain-of-thought learning for a simple base class.
LaRT models LLMs' response accuracy and CoT length to evaluate reasoning ability and speed.
problem Valid evaluation of Large Language Models (LLMs) via response accuracy and chain-of-thought length.
method Introduces Latency-Response Theory (LaRT) to jointly model response accuracy and CoT length using latent ability and latent speed.
result LaRT yields higher estimation accuracy and shorter confidence intervals for latent traits compared to IRT.
Optimal sample complexity for autoregressive chain-of-thought learning proven.
problem Determining the minimum number of samples needed for accurate autoregressive chain-of-thought learning.
method Proved upper bound on sample complexity using Daniely-Shalev-Shwartz dimension and roll-out stable parity dimension.
result The sample complexity is bounded by the local next-token class rate, with no dependence on rollout length.
Reasoning models generate differently based on problem difficulty, not just length.
problem Understanding how reasoning models handle different problem difficulties.
method Examined hidden-state trajectories across competitive programming, mathematics, and Boolean satisfiability.
result Corrected trajectory geometry shows difficulty-dependent differences in reasoning models, with stronger effects in the code domain.
Study reveals how depth of reasoning affects generalization in models.
problem Understanding scaling behavior of generalization with CoT depth.
method Theoretical model of CoT in linear regression using random matrix theory.
result Sharp phase transition between exponential and polynomial improvement, saturation, and overthinking.
The paper explores how LLMs with CoT improve performance on complex tasks.
problem Understanding the mechanisms behind LLMs' improved performance with CoT.
method Using circuit complexity theory, the paper examines LLMs' expressivity in solving mathematical and decision-making problems.
result LLMs with CoT can generate correct solutions step-by-step, even for complex tasks.
CoT enhances transformer accuracy on serial tasks by enabling serial computation.
problem Improving accuracy of large language models on inherently serial problems.
method Integrating a chain of thought (CoT) into decoder-only transformers to enable serial computation.
result Constant-depth transformers with CoT can solve problems in AC^0, surpassing TC^0 without CoT.
This research proves guarantees on sequence models' generalization to longer and novel sequences.
problem Generalization to longer sequences and novel token combinations in sequence models.
method Provable guarantees on length and compositional generalization for various sequence models.
result Limited capacity models achieve both length and compositional generalization with diverse training distributions.
AI systems that explain their decisions can be monitored for harmful intentions.
problem Monitoring AI systems' decision-making processes for harmful intentions is imperfect and can miss some misbehavior.
method Monitoring the chain of thought (CoT) of AI systems that communicate in human language.
result CoT monitoring is a promising but fragile approach to AI safety.
Transformers struggle with time series forecasting, especially in context.
problem Transformers' limitations in time series forecasting, particularly in in-context settings.
method Theoretical analysis of Transformers' limitations through In-Context Learning theory.
result Linear Self-Attention models cannot outperform classical linear models in in-context time series forecasting.
Prefix consistency improves model reliability by weighting answers based on their reproducibility.
problem Improving the reliability of large language models' reasoning traces.
method Use prefix consistency to weight candidate answers based on their reproducibility during regeneration.
result Prefix consistency is the best correctness predictor, reaching Standard MV plateau accuracy with up to 21x fewer tokens.
New system resists meme coin copy trading bots.
problem Manipulative bots exploit copy trading in illiquid meme coins.
method Multi-agent architecture with LLM and CoT reasoning.
result System outperforms other methods in prediction and economic performance.
This paper improves learning complex functions with CoT supervision, reducing sample complexity.
problem Learning complex functions with multi-step reasoning.
method Develops a statistical theory linking CoT risk and end-to-end risk, using CoT information measure.
result CoT supervision can achieve significantly faster learning rates compared to standard E2E supervision.
Theoretical work shows integrating coherent reasoning improves LLM performance and error correction.
problem Improving reasoning and error correction in large language models (LLMs) with few-shot prompting.
method Theoretical analysis and sensitivity experiments on transformer behavior with coherent reasoning and corrupted demonstrations.
result The transformer gains better error correction ability and more accurate predictions when coherent reasoning is integrated.
Summarizes financial news for better investment decisions.
problem Information overload from financial news hinders timely investment decisions.
method Personalized Chain-of-Thought summarization framework integrating user-specified keywords.
result Personalized summaries highlight relevant market signals, improving investment narratives.
EORM boosts LLM accuracy with a lightweight, energy-based verifier.
problem Efficiently verifying mathematical reasoning in large language models.
method Energy-based framework to rank Chain-of-Thought solutions using simple outcome labels.
result EORM boosts LLM accuracy to 90.7% on GSM8k and 63.7% on MATH with only 55M parameters.
Transformers solve parity problems efficiently with step-by-step reasoning.
problem Training transformers to solve complex, recursive problems like parity.
method Training a one-layer transformer to solve k k k -parity, incorporating intermediate parities into the loss function, and using teacher forcing or augmented data. result Transformers can learn parity in one gradient update with intermediate supervision or self-consistency checks.
CoT-UQ improves LLM uncertainty quantification by integrating reasoning steps.
problem LLMs' overconfidence and lack of response-wise uncertainty quantification.
method Integrates LLMs' reasoning steps into uncertainty estimation.
result Significantly improves uncertainty quantification accuracy (5.9% AUROC improvement).
Fractured Sampling improves LLM reasoning efficiency by truncating CoT trajectories.
problem Efficiently scaling reasoning in large language models with limited tokens.
method Integrating truncated Chain-of-Thought (CoT) with Fractured Sampling across multiple dimensions.
result Fractured Sampling achieves superior accuracy-cost trade-offs compared to full CoT.
The paper studies how search and distillation improve reasoning in large language models.
problem Improving reasoning capabilities of large language models.
method Viewing chain-of-thought generation as a metastable Markov process, proving benefits of search and distillation.
result Search protocol rewards sparse edges, reducing the expected number of steps to reach different clusters.
A conformal procedure improves CoT reasoning by aggregating reasoning paths and calibrating abstention rules.
problem Aggregation uncertainty in chain-of-thought reasoning makes correct answers less reliable.
method Introduces a conformal procedure for CoT reasoning that uses weighted score aggregation and abstention rules.
result Achieves higher selective accuracy with abstention, reducing confident-error rate.
The study examines how extra compute during testing affects the performance of large language models.
problem Understanding the conditions under which test-time scaling improves model performance.
method An in-context weight prediction task for linear regression was used to train transformers. The performance was analyzed under varying levels of test-time compute.
result Training transformers on diverse, relevant, and hard tasks leads to the best performance for test-time scaling.
Transformers with CoT don't enhance reasoning power across all tasks.
problem Does CoT enhance the reasoning power of transformers?
method Examined the memorization capabilities of fixed-precision transformers with and without CoT.
result Transformers with CoT cannot memorize all reasoning tasks, leading to a negative answer.
This work analyzes CoT prompting methods from a statistical estimation perspective.
problem Improving the effectiveness of LLMs in solving multi-step reasoning problems.
method Introducing a multi-step latent variable model to characterize CoT prompting from a statistical estimation viewpoint.
result The CoT estimator is equivalent to a Bayesian estimator when the pretraining dataset is large.
CoT improves transformer sample efficiency by reducing input token dependencies and attention sparsity.
problem Transformer sample inefficiency in simple tasks.
method Demonstrated through parity-learning setup, showing CoT reduces required samples from exponential to polynomial.
result Transformer learns function within polynomial samples with CoT, requiring exponential samples without CoT.
Transformers can generalize to a large task family with only a few demonstrations.
problem Can learning from a small set of tasks generalize to a large task family?
method Investigating autoregressive compositional structure where each task is a composition of T T T operations, each from a finite family of D D D subtasks. result Transformers can generalize to D T D^T D T tasks with only O ~ ( D ) \widetilde{O}(D) O ( D ) demonstrations. Dynamic abstention improves LLM accuracy by selectively terminating unpromising reasoning.
problem LLMs waste compute on incorrect responses, leading to inefficiency.
method Formal reinforcement learning framework with abstention reward parameter.
result Dynamic abstention outperforms natural baselines in selective accuracy.
CHOOSE enhances shallow Transformers for wireless symbol detection.
problem Improving wireless symbol detection with shallow Transformers.
method Introducing autoregressive latent reasoning steps within hidden space.
result Lightweight Transformers achieve comparable performance to deep models.
RNNs struggle with in-context retrieval, while Transformers excel.
problem In-context retrieval capability of RNNs.
method Theoretical analysis and experimental techniques (CoT, RAG, Transformer layer).
result Enhancing RNNs with techniques improves their in-context retrieval capability, closing the representation gap with Transformers.
This work investigates how multi-round reasoning improves LLM performance.
problem Improving problem-solving abilities in complex tasks with LLMs.
method Investigates approximation, learnability, and generalization properties of multi-round auto-regressive models.
result Transformers with finite context windows are universal approximators for Turing-computable functions and can approximate any Turing-computable sequence-to-sequence function through multi-round reasoning.
New technique reduces language biases in large language models.
problem Gap between language and thought in large language models.
method Proposes Language-of-Thoughts (LoT) technique to adjust order and tokens.
result Significantly reduces language biases and improves reasoning tasks.
Transformers learn multi-step reasoning through gradient descent.
problem Understanding how transformers solve symbolic multi-step reasoning tasks.
method Theoretical analysis of gradient descent dynamics and multi-phase training.
result Trained one-layer transformers can solve both backward and forward reasoning tasks with generalization guarantees.
Reinforcement learning after next-token prediction aids in learning from diverse sequence lengths.
problem Learning from sequences of varying lengths and complexity.
method Introducing a framework to study reinforcement learning with autoregressive transformers, focusing on next-token prediction and mixture distributions of short and long sequences.
result Reinforcement learning after next-token prediction enables autoregressive transformers to generalize from long sequences, even when they are rare.
SelfReflect measures LLM uncertainty by summarizing belief distribution.
problem LLMs lack transparency in expressing uncertainty.
method Developed SelfReflect metric to quantify uncertainty.
result Modern LLMs cannot reflect their uncertainty.
LLMs struggle with financial reasoning but can outperform the market with human oversight.
problem Financial reasoning failures in LLM-generated stock market predictions.
method Evaluated four LLMs using three prompting strategies and compared to human oversight.
result LLMs require human oversight to fully realize their potential in financial markets.
Minimalist softmax attention learns constrained Boolean functions with supervision.
problem Learning k k k -bit Boolean functions, especially AND and OR. method Minimalist single-head softmax-attention mechanism with teacher forcing.
result Minimalist attention can solve Boolean tasks with supervision, but not without.
This paper proves a conjecture about trisections with a specific length.
problem Proving a conjecture about trisections with a specific length.
method Examining trisections with Kirby-Thompson length 2 and proving the conjecture.
result Proves the conjecture about length 2 trisection being a 4-manifold with length 0.
Study evaluates LLMs like ChatGPT and GPT-4 on financial analysis tasks.
problem Assessing financial reasoning capabilities of large language models.
method Mock CFA exam questions, Zero-Shot, Chain-of-Thought, Few-Shot scenarios.
result LLMs perform well on financial analysis tasks, but have limitations.
Filling length measures the length of the contracting closed loops in a null-homotopy. The filling length function of Gromov for a finitely presented group measures the filling length as a function of length of edge-loops in the Cayley 2-complex. We give a bound on the filling length function in terms of the log of an …
Constructs non-isometric iso-length-spectral surfaces.
problem Creating non-isometric surfaces with identical geodesic lengths.
method Combining Sunada's construction with amalgams of hyperbolic surfaces.
result Found non-isometric surfaces with the same geodesic lengths.
Study on stable torsion length in groups, showing it vanishes in crystallographic groups and providing algorithms for computation.
problem Understanding the stable torsion length in groups, especially in crystallographic and free products of groups.
method Developed linear programming and exact algorithms to compute stable torsion length in free products of groups and finite groups.
result Showed that stable torsion length vanishes in crystallographic groups and provided exact computations for nontrivial examples.
Study on stable translation lengths of surface homeomorphisms and their approximations.
problem Understanding stable translation lengths of homeomorphisms and their finite approximations.
method Comparing stable translation lengths of homeomorphisms and their finite approximations on curve graphs.
result Stable translation length of homeomorphisms with dense periodic points equals the supremum of their approximations.
We study the length, weak length and complex length spectrum of closed geodesics of a compact flat Riemannian manifold, comparing length-isospectrality with isospectrality of the Laplacian acting on p-forms. Using integral roots of the Krawtchouk polynomials, we give many pairs of p-isospectral flat manifolds having di…
This paper reviews LLMs for credit risk assessment, creating a taxonomy.
problem Assessing credit risk using financial text analysis.
method Systematic review of 60 papers, focusing on model architectures, data types, and explainability mechanisms.
result Developed a taxonomy of LLM-based credit risk models.
J.-B. Meilhan and the second author showed that any Milnor μ ˉ \barμ μ ˉ -invariant of length between 3 and 2 k + 1 2k+1 2 k + 1 can be represented as a combination of HOMFLYPT polynomial of knots obtained by certain band sum of the link components, if all μ ˉ \barμ μ ˉ -invariants of length ≤ k \leq k ≤ k vanish. They also showed that their formula do…
New proof shows surfaces can have identical length spectra but not simple ones.
problem Identifying when two covers of a surface have identical length spectra but not simple ones.
method Characterized isomorphism of covers and constructed surfaces with identical spectra but different simple length spectra.
result Found surfaces with identical length spectra but not simple length isospectral covers.
ReLU networks don't exponentially distort curve lengths as previously thought.
problem Understanding how neural networks distort curve lengths with depth.
method Analyzing expected length distortion of ReLU networks with random initialization.
result Expected length distortion does not grow with depth, and shrinks slightly.