Hydra boosts efficiency for long-context reasoning in resource-constrained settings.
problem Quadratic complexity of transformers limits long-context reasoning in resource-constrained systems.
method Hydra uses a modular architecture with adaptive routing between sparse global attention, mixture-of-experts, and dual memories.
result Hydra achieves significant throughput and accuracy improvements for long-context reasoning.
The paper tackles long-context linear system identification with improved sample complexity bounds.
problem Identifying dynamical systems with long dependencies over fixed context windows.
method Established sample complexity bounds for systems with linear dependencies over a context window of length p.
result The learning process is not hindered by slow mixing properties in extended context windows.
Scales attention for long contexts in LLMs.
problem Development of attention mechanisms for long context inference.
method Scale-invariant total attention and sparsity conditions, with a position-dependent transformation of logits.
result Scale-invariant attention scheme improves validation loss and long-context retrieval.
Sparse attention model reduces long-context inference time with exponential accuracy guarantees.
problem Efficiently processing long-context queries in large language models.
method Formalizes attention as a projection onto key vectors, analyzes entropic relaxation, and introduces Vashista Sparse Attention.
result Sparse attention concentrates on a constant-size active face, leading to exponential decay of inactive tokens' mass and linear scaling of active face error.
Transformers learn chain-of-thought reasoning for longer problems, proving length generalization.
problem Challenging problems require deeper reasoning, but how do models generalize this to longer tasks?
method Theoretical analysis of transformers on synthetic state-tracking tasks, proving length generalization through attention concentration.
result Transformers can learn chain-of-thought reasoning for longer problems, proving length generalization.
Transformer models waste resources on long-context tasks.
problem Redundant attention computations in Transformer models for long-context tasks.
method Reformulate sequence modeling as supervised learning, analyze attention sparsity, formulate attention optimization as linear coding problem, propose Dynamic Group Attention.
result DGA reduces computational costs while maintaining performance.
Improves many-shot learning by optimizing and generating influential examples.
problem Limited performance in many-shot in-context learning (ICL).
method Iterative optimization and generation of influential examples.
result Significant improvements across various tasks using BRIDGE.
Timer-XL predicts multidimensional time series using a unified Transformer approach.
problem Unified time series forecasting across various tasks and contexts.
method Decoder-only Transformers with a universal TimeAttention mechanism and deft position embedding.
result State-of-the-art performance across multiple forecasting benchmarks.
Enhanced TSFMs improve time series forecasting accuracy and reliability.
problem Variance, bias, and uncertainty in TSFMs' predictions on real data.
method Statistical and ensemble techniques including bagging, stacking, residual modeling, and prediction intervals.
result Hybrid models consistently outperform standalone TSFMs across multiple horizons.
Mamba struggles with long context lengths, but spectrum scaling improves performance.
problem Mamba's performance degrades with increasing context length.
method Spectrum scaling applied to pre-trained Mamba models to improve long-context generalization.
result Spectrum scaling significantly improves performance in long-context settings.
FibQuant improves KV-cache compression for long-context inference.
problem Memory traffic bottleneck in long-context inference due to KV cache growth.
method Introduces FibQuant, a universal vector quantizer that combines Beta-quantile radii, Fibonacci/Roberts-Kronecker directions, and Lloyd-Max refinement.
result FibQuant achieves high compression rates with minimal loss in attention cosine similarity.
New Performer model tackles long-sequence protein modeling.
problem Challenges of training complex Transformer models for long sequences.
method Linearly scalable long-context Transformer architecture, Performer.
result Performer provides strong theoretical guarantees and is effective for protein sequence modeling.
Existing language models such as n-grams for software code often fail to capture a long context where dependent code elements scatter far apart. In this paper, we propose a novel approach to build a language model for software code to address this particular issue. Our language model, partly inspired by human memory, i…
Jukebox generates high-fidelity songs with singing in raw audio.
problem Generating music with singing in raw audio.
method Multi-scale VQ-VAE for compression, autoregressive Transformers for modeling.
result Generates high-fidelity and diverse songs with coherence up to multiple minutes.
Express improves causal attention guarantees for language models.
problem Improving causal attention guarantees for language models.
method Introducing Express, a tool for converting non-causal attention into causal with matching guarantees.
result Express improves causal attention guarantees to log^(3/2)(n)/s with minimal memory and compression overhead.
Cisco introduces a new time series model for better forecasting.
problem Improving time series forecasting accuracy.
method Developed a new multiresolution decoder-only model trained on large datasets.
result The new model achieves superior performance on observability datasets.
ZeroS improves Transformers by adding negative weights, matching or beating softmax attention.
problem Limited performance of linear attention methods, especially in long context sequences.
method Proposes Zero-Sum Linear Attention (ZeroS) that removes the zero-order term and reweights zero-sum softmax residuals.
result ZeroS matches or exceeds standard softmax attention across various benchmarks, theoretically expanding representable functions.
Unified framework for critical scaling of inverse temperature in self-attention.
problem Conflicting inverse-temperature laws for long-context self-attention.
method Counting gaps and defining an upper-tail accumulation scale.
result Critical inverse-temperature scale determined by gap-counting function.
New insights into attention mechanisms reveal dramatic trade-offs between rank and heads.
problem Dramatic trade-offs between rank and number of heads in attention mechanisms.
method Presented a simple target function and proved theoretical limits.
result Full-rank attention is necessary for long contexts, while low-rank is sufficient for short ones.
Analysis of ridge regression under concept shift reveals nontrivial effects on generalization performance.
problem Understanding and mitigating the impact of distribution shift in machine learning models.
method Derivation of exact prediction risk expression in the thermodynamic limit for ridge regression under concept shift.
result Reveals a phase transition and nonmonotonic data dependence of test performance under concept shift.
A new framework for efficient sequence maps using Bayesian filtering and covariance.
problem Designing efficient recurrent sequence maps from explicit memory assumptions.
method Design-model framework, exact Bayesian filtering, query-dependent readout, linear-Gaussian instantiation.
result Improved robustness and retrieval performance across various benchmarks.
FinanceBench benchmarks LLMs on financial QA, revealing limitations.
problem Evaluating LLMs' performance on financial question answering.
method Developed a comprehensive test suite (FinanceBench) with 10,231 questions, tested 16 models, and manually reviewed answers.
result Existing LLMs have significant limitations for financial QA, especially GPT-4-Turbo.
LaTRO optimizes latent reasoning in LLMs without external reward.
problem Training LLMs to perform complex reasoning tasks.
method Formulates reasoning as latent distribution sampling and optimizes via variational approaches.
result LLMs improve reasoning and evaluation quality through self-improvement.
Auto-CEI improves LLM reasoning by balancing assertiveness and conservativeness.
problem Hallucinations and laziness in LLM reasoning tasks.
method Expert Iteration explores reasoning trajectories, guiding incorrect paths back on track and promoting appropriate 'I don't know' responses.
result Auto-CEI achieves superior alignment in logical reasoning, mathematics, and planning tasks.
A framework isolates VQA reasoning from perception for better model evaluation.
problem Improper separation of visual perception and reasoning in VQA models.
method Introducing a framework and a top-down calibration technique to decouple reasoning from perception.
result Improved evaluation of VQA models by separating reasoning from perception.
A new method for math reasoning that allows for iterative correction.
problem Standard reasoning models commit to each token and cannot recover from early errors.
method Generative framework with latent thought vectors for iterative self-correction.
result 30 rethinking iterations surpass baselines with 15 times more parameters.
Inferring new facts from existing knowledge graphs (KG) with explainable reasoning processes is a significant problem and has received much attention recently. However, few studies have focused on relation types unseen in the original KG, given only one or a few instances for training. To bridge this gap, we propose Co…
Transformers learn multi-step reasoning through gradient descent.
problem Understanding how transformers solve symbolic multi-step reasoning tasks.
method Theoretical analysis of gradient descent dynamics and multi-phase training.
result Trained one-layer transformers can solve both backward and forward reasoning tasks with generalization guarantees.
Transformers with CoT don't enhance reasoning power across all tasks.
problem Does CoT enhance the reasoning power of transformers?
method Examined the memorization capabilities of fixed-precision transformers with and without CoT.
result Transformers with CoT cannot memorize all reasoning tasks, leading to a negative answer.
Early stopping methods reduce unnecessary reasoning steps in LLMs by monitoring uncertainty signals.
problem LLMs sometimes generate unnecessary reasoning steps, especially under uncertainty.
method Statistically principled early stopping methods that monitor uncertainty signals during generation.
result Uncertainty-aware early stopping improves efficiency and reliability in LLM reasoning, especially in math reasoning.
Achieving artificial visual reasoning - the ability to answer image-related questions which require a multi-step, high-level process - is an important step towards artificial general intelligence. This multi-modal task requires learning a question-dependent, structured reasoning process over images from language. Stand…
FinTradeBench benchmarks LLMs for financial reasoning combining company fundamentals and market signals.
problem Challenges in evaluating financial reasoning models for LLMs.
method Developed a benchmark integrating company fundamentals and trading signals, using a calibration-then-scaling framework.
result Clear performance gap between LLMs, retrieval improves reasoning over textual fundamentals but not trading signals.
Hybrid framework injects TSLM insights into GRLM for robust time-series reasoning.
problem Lack of domain-specific knowledge in large language models for time-series reasoning.
method Hybrid knowledge-injection framework combining RLVR for efficient knowledge transfer.
result Consistently outperforms existing models by 7.9%-26.1% on multivariate time-series benchmarks.
Neural Logic Reasoning integrates deep learning and symbolic logic for better prediction tasks.
problem Lack of cognitive reasoning in deep neural networks limits their ability to solve complex prediction tasks.
method Proposes Logic-Integrated Neural Network (LINN) that learns logical operations and conducts propositional logical reasoning.
result LINN significantly outperforms state-of-the-art recommendation models in Top-K recommendation.
Forward-prediction models enhance physical reasoning, but only for specific tasks.
problem Improving physical reasoning in complex tasks involving many objects.
method Incorporated forward-prediction models into simple physical-reasoning agents and evaluated their performance on the PHYRE benchmark.
result Forward-prediction models improve physical-reasoning performance, especially on complex tasks, but generalization to new task templates is challenging.
Without relevant human priors, neural networks may learn uninterpretable features. We propose Dynamics of Attention for Focus Transition (DAFT) as a human prior for machine reasoning. DAFT is a novel method that regularizes attention-based reasoning by modelling it as a continuous dynamical system using neural ordinary…
RACER optimizes LLM-as-judge accuracy with dynamic reasoning selection.
problem Balancing reasoning accuracy with computational cost in LLM-as-judge settings.
method Formulates routing as a constrained distributionally robust optimization problem, accounting for distribution shift via KL-divergence uncertainty set.
result RACER achieves superior accuracy-cost trade-offs under distribution shift.
VTA combines verbal and latent reasoning for accurate stock time-series forecasts.
problem Challenges in combining textual analysis with time-series data for financial forecasting.
method Converts stock price data into textual annotations, optimizes reasoning trace using inverse MSE, conditions time-series model outputs on reasoning attributes.
result VTA achieves state-of-the-art forecasting accuracy and interpretable reasoning traces.
Fractured Sampling improves LLM reasoning efficiency by truncating CoT trajectories.
problem Efficiently scaling reasoning in large language models with limited tokens.
method Integrating truncated Chain-of-Thought (CoT) with Fractured Sampling across multiple dimensions.
result Fractured Sampling achieves superior accuracy-cost trade-offs compared to full CoT.
SRN improves set representations for relational reasoning.
problem Set permutational invariance limitations in existing approaches.
method Proposed a Set Refiner Network (SRN) to respect set invariance.
result Substantial gains in prediction performance and robustness on relational reasoning tasks.
Few-shot visual reasoning model learns analogical relationships from small data.
problem Training deep models on few samples for visual reasoning tasks.
method Meta-analogical contrastive learning to enforce structural similarity between training and test samples.
result Method outperforms state-of-the-art on RAVEN dataset with scarce training data.
Study reveals how depth of reasoning affects generalization in models.
problem Understanding scaling behavior of generalization with CoT depth.
method Theoretical model of CoT in linear regression using random matrix theory.
result Sharp phase transition between exponential and polynomial improvement, saturation, and overthinking.
Enhances large language models' reasoning through simpler off-policy reinforcement learning.
problem Improving large language models' ability to reason and solve problems.
method EM Policy Gradient, optimizing expected return over reasoning trajectories using Expectation-Maximization (EM) optimization.
result Achieves comparable or slightly superior performance to state-of-the-art methods on reasoning datasets, with additional cognitive behaviors.
Self-supervised skip-tree training improves mathematical reasoning in language models.
problem Improving logical reasoning in language models for formal mathematics.
method Self-supervised language modeling on mathematical formulas, skip-tree task.
result Models trained on skip-tree task outperform standard models in mathematical reasoning tasks.
Theoretical work shows integrating coherent reasoning improves LLM performance and error correction.
problem Improving reasoning and error correction in large language models (LLMs) with few-shot prompting.
method Theoretical analysis and sensitivity experiments on transformer behavior with coherent reasoning and corrupted demonstrations.
result The transformer gains better error correction ability and more accurate predictions when coherent reasoning is integrated.
R2-B2 optimizes game interactions with recursive reasoning.
problem Optimizing interactions between boundedly rational agents with unknown payoff functions.
method Recursive Reasoning-Based Bayesian Optimization (R2-B2) for repeated games.
result R2-B2 achieves faster asymptotic convergence to no regret than non-recursive methods.
This work investigates how multi-round reasoning improves LLM performance.
problem Improving problem-solving abilities in complex tasks with LLMs.
method Investigates approximation, learnability, and generalization properties of multi-round auto-regressive models.
result Transformers with finite context windows are universal approximators for Turing-computable functions and can approximate any Turing-computable sequence-to-sequence function through multi-round reasoning.
MXGNet tackles visual reasoning tasks using graph neural networks.
problem Abstract reasoning, especially in the visual domain, is challenging for AI.
method Combines object-level representations, graph neural networks, and multiplex graphs.
result Achieves state-of-the-art accuracy on Euler Diagram Syllogisms and outperforms state-of-the-art models on RPM datasets.