LTMs use latent vectors for efficient autoregressive generation.
problem Efficient autoregressive generation in language models.
method Dual-rate optimization in variational Bayes framework.
result LTMs achieve superior sample and parameter efficiency.
A new method for math reasoning that allows for iterative correction.
problem Standard reasoning models commit to each token and cannot recover from early errors.
method Generative framework with latent thought vectors for iterative self-correction.
result 30 rethinking iterations surpass baselines with 15 times more parameters.
LaRT models LLMs' response accuracy and CoT length to evaluate reasoning ability and speed.
problem Valid evaluation of Large Language Models (LLMs) via response accuracy and chain-of-thought length.
method Introduces Latency-Response Theory (LaRT) to jointly model response accuracy and CoT length using latent ability and latent speed.
result LaRT yields higher estimation accuracy and shorter confidence intervals for latent traits compared to IRT.
A method for learning with autoregressive chain-of-thoughts.
problem Learning prompt-to-answer mappings from sequence-to-next-token generators.
method Iterating a fixed, time-invariant generator for multiple steps to generate a chain-of-thought, then taking the final token as the answer.
result Universal representability and computationally tractable chain-of-thought learning for a simple base class.
This work analyzes CoT prompting methods from a statistical estimation perspective.
problem Improving the effectiveness of LLMs in solving multi-step reasoning problems.
method Introducing a multi-step latent variable model to characterize CoT prompting from a statistical estimation viewpoint.
result The CoT estimator is equivalent to a Bayesian estimator when the pretraining dataset is large.
Generative adversarial networks (GANs) transform latent vectors into visually plausible images. It is generally thought that the original GAN formulation gives no out-of-the-box method to reverse the mapping, projecting images back into latent space. We introduce a simple, gradient-based technique called stochastic cli…
This paper demonstrates how Dropout can be used in Generative Adversarial Networks to generate multiple different outputs to one input. This method is thought as an alternative to latent space exploration, especially if constraints in the input should be preserved, like in A-to-B translation tasks.
A new autoencoder method uses empirical beta copulas for generating data.
problem Creating a generative model from an autoencoder's latent space.
method Empirical Beta Copula Autoencoder method.
result The Empirical Beta Copula Autoencoder outperforms other methods in simplicity and effectiveness.
LaTRO optimizes latent reasoning in LLMs without external reward.
problem Training LLMs to perform complex reasoning tasks.
method Formulates reasoning as latent distribution sampling and optimizes via variational approaches.
result LLMs improve reasoning and evaluation quality through self-improvement.
Improved forecasting of suicide attempts using LSGPs for patients with little data.
problem Challenges in predicting suicide attempts due to their rarity and patient heterogeneity.
method Introduced Latent Similarity Gaussian Processes (LSGPs) to capture patient heterogeneity.
result LSGPs outperform baseline models, even without kernel-design, and offer new insights into patient similarity.
CHOOSE enhances shallow Transformers for wireless symbol detection.
problem Improving wireless symbol detection with shallow Transformers.
method Introducing autoregressive latent reasoning steps within hidden space.
result Lightweight Transformers achieve comparable performance to deep models.
New technique reduces language biases in large language models.
problem Gap between language and thought in large language models.
method Proposes Language-of-Thoughts (LoT) technique to adjust order and tokens.
result Significantly reduces language biases and improves reasoning tasks.
The paper explores indeterminacy in latent factor projections and its implications for data representation.
problem Indeterminacy in latent factor projections and its implications for data representation.
method Analyzes the fundamental problem of indeterminacy in latent factor projections and discusses its implications for data representation.
result Latent factor determinacy across all facets is achieved when the feature-dimension grows to infinity.
The stochastic variational inference (SVI) paradigm, which combines variational inference, natural gradients, and stochastic updates, was recently proposed for large-scale data analysis in conjugate Bayesian models and demonstrated to be effective in several problems. This paper studies a family of Bayesian latent vari…
Fast approximate inference for non-Gaussian data.
problem Efficient inference for non-Gaussian data.
method Laplace Matching for fast approximate inference in latent Gaussian models.
result Achieves high approximation quality with low computational cost.
Protein Thoughts interprets protein interactions with clear reasoning, improving prediction accuracy.
problem Lack of mechanistic justification in protein-protein interaction predictions.
method Interpretable search problem reformulation, hypothesis-guided entropy-regularized Tree-of-Thoughts search, embedding-space flow matching.
result Improves mean best-binder rank from 47.7 to 11.2 on SHS148k benchmark.
AI systems that explain their decisions can be monitored for harmful intentions.
problem Monitoring AI systems' decision-making processes for harmful intentions is imperfect and can miss some misbehavior.
method Monitoring the chain of thought (CoT) of AI systems that communicate in human language.
result CoT monitoring is a promising but fragile approach to AI safety.
We introduce a new class of context dependent, incomplete information games to serve as structured prediction models for settings with significant strategic interactions. Our games map the input context to outcomes by first condensing the input into private player types that specify the utilities, weighted interactions…
Transactional network data can be thought of as a list of one-to-many communications(e.g., email) between nodes in a social network. Most social network models convert this type of data into binary relations between pairs of nodes. We develop a latent mixed membership model capable of modeling richer forms of transacti…
Prefix consistency improves model reliability by weighting answers based on their reproducibility.
problem Improving the reliability of large language models' reasoning traces.
method Use prefix consistency to weight candidate answers based on their reproducibility during regeneration.
result Prefix consistency is the best correctness predictor, reaching Standard MV plateau accuracy with up to 21x fewer tokens.
Study reveals AI's spontaneous topic changes in text prediction.
problem AI's inability to spontaneously switch topics like humans.
method Defined topic as Token Priority Graphs (TPGs) and analyzed self-attention models.
result AI can only switch topics if lower-priority tokens outnumber higher-priority ones.
EORM boosts LLM accuracy with a lightweight, energy-based verifier.
problem Efficiently verifying mathematical reasoning in large language models.
method Energy-based framework to rank Chain-of-Thought solutions using simple outcome labels.
result EORM boosts LLM accuracy to 90.7% on GSM8k and 63.7% on MATH with only 55M parameters.
Neural SDEs model suicide risk with compact state space constraints.
problem Modeling suicide risk with irregular, noisy, and partially observed data.
method Developed neural SDEs confined to compact state spaces, addressing domain constraints and numerical stability.
result Improved forecasts and optimization dynamics over standard models on EMA datasets.
Transformers learn chain-of-thought reasoning for longer problems, proving length generalization.
problem Challenging problems require deeper reasoning, but how do models generalize this to longer tasks?
method Theoretical analysis of transformers on synthetic state-tracking tasks, proving length generalization through attention concentration.
result Transformers can learn chain-of-thought reasoning for longer problems, proving length generalization.
Summarizes financial news for better investment decisions.
problem Information overload from financial news hinders timely investment decisions.
method Personalized Chain-of-Thought summarization framework integrating user-specified keywords.
result Personalized summaries highlight relevant market signals, improving investment narratives.
Theoretical work shows integrating coherent reasoning improves LLM performance and error correction.
problem Improving reasoning and error correction in large language models (LLMs) with few-shot prompting.
method Theoretical analysis and sensitivity experiments on transformer behavior with coherent reasoning and corrupted demonstrations.
result The transformer gains better error correction ability and more accurate predictions when coherent reasoning is integrated.
Lasso performs poorly with correlated covariates, but a rescaled approach fixes this.
problem Lasso's performance degrades with correlated covariates, leading to inefficiency.
method Proposes a rescaling method for Lasso to handle correlated covariates effectively.
result Rescaled Lasso provides strong provable guarantees for estimation with quadratic sample complexity.
Study reveals how depth of reasoning affects generalization in models.
problem Understanding scaling behavior of generalization with CoT depth.
method Theoretical model of CoT in linear regression using random matrix theory.
result Sharp phase transition between exponential and polynomial improvement, saturation, and overthinking.
The paper studies how search and distillation improve reasoning in large language models.
problem Improving reasoning capabilities of large language models.
method Viewing chain-of-thought generation as a metastable Markov process, proving benefits of search and distillation.
result Search protocol rewards sparse edges, reducing the expected number of steps to reach different clusters.
The Machina thought experiments pose to major non-expected utility models challenges that are similar to those posed by the Ellsberg thought experiments to subjective expected utility theory (SEUT). We test human choices in the `Ellsberg three-color example', confirming typical ambiguity aversion patterns, and the `Mac…
New system resists meme coin copy trading bots.
problem Manipulative bots exploit copy trading in illiquid meme coins.
method Multi-agent architecture with LLM and CoT reasoning.
result System outperforms other methods in prediction and economic performance.
Optimal sample complexity for autoregressive chain-of-thought learning proven.
problem Determining the minimum number of samples needed for accurate autoregressive chain-of-thought learning.
method Proved upper bound on sample complexity using Daniely-Shalev-Shwartz dimension and roll-out stable parity dimension.
result The sample complexity is bounded by the local next-token class rate, with no dependence on rollout length.
Transformers solve parity problems efficiently with step-by-step reasoning.
problem Training transformers to solve complex, recursive problems like parity.
method Training a one-layer transformer to solve k-parity, incorporating intermediate parities into the loss function, and using teacher forcing or augmented data. result Transformers can learn parity in one gradient update with intermediate supervision or self-consistency checks.
Fractured Sampling improves LLM reasoning efficiency by truncating CoT trajectories.
problem Efficiently scaling reasoning in large language models with limited tokens.
method Integrating truncated Chain-of-Thought (CoT) with Fractured Sampling across multiple dimensions.
result Fractured Sampling achieves superior accuracy-cost trade-offs compared to full CoT.
CoT-UQ improves LLM uncertainty quantification by integrating reasoning steps.
problem LLMs' overconfidence and lack of response-wise uncertainty quantification.
method Integrates LLMs' reasoning steps into uncertainty estimation.
result Significantly improves uncertainty quantification accuracy (5.9% AUROC improvement).
This paper improves learning complex functions with CoT supervision, reducing sample complexity.
problem Learning complex functions with multi-step reasoning.
method Develops a statistical theory linking CoT risk and end-to-end risk, using CoT information measure.
result CoT supervision can achieve significantly faster learning rates compared to standard E2E supervision.
CoT enhances transformer accuracy on serial tasks by enabling serial computation.
problem Improving accuracy of large language models on inherently serial problems.
method Integrating a chain of thought (CoT) into decoder-only transformers to enable serial computation.
result Constant-depth transformers with CoT can solve problems in AC^0, surpassing TC^0 without CoT.
The computational properties of neural systems are often thought to be implemented in terms of their network dynamics. Hence, recovering the system dynamics from experimentally observed neuronal time series, like multiple single-unit (MSU) recordings or neuroimaging data, is an important step toward understanding its c…
The paper explores how LLMs with CoT improve performance on complex tasks.
problem Understanding the mechanisms behind LLMs' improved performance with CoT.
method Using circuit complexity theory, the paper examines LLMs' expressivity in solving mathematical and decision-making problems.
result LLMs with CoT can generate correct solutions step-by-step, even for complex tasks.
A conformal procedure improves CoT reasoning by aggregating reasoning paths and calibrating abstention rules.
problem Aggregation uncertainty in chain-of-thought reasoning makes correct answers less reliable.
method Introduces a conformal procedure for CoT reasoning that uses weighted score aggregation and abstention rules.
result Achieves higher selective accuracy with abstention, reducing confident-error rate.
We prove a smooth compactness theorem for the space of embedded self-shrinkers in $\RR^3$. Since self-shrinkers model singularities in mean curvature flow, this theorem can be thought of as a compactness result for the space of all singularities and it plays an important role in studying generic mean curvature flow.
A (complete) matching of the cells of a triangulated manifold can be thought as a combinatorial or discrete version of a nonsingular vector field. We give several methods for constructing such matchings.
Transformers with CoT don't enhance reasoning power across all tasks.
problem Does CoT enhance the reasoning power of transformers?
method Examined the memorization capabilities of fixed-precision transformers with and without CoT.
result Transformers with CoT cannot memorize all reasoning tasks, leading to a negative answer.
CoT improves transformer sample efficiency by reducing input token dependencies and attention sparsity.
problem Transformer sample inefficiency in simple tasks.
method Demonstrated through parity-learning setup, showing CoT reduces required samples from exponential to polynomial.
result Transformer learns function within polynomial samples with CoT, requiring exponential samples without CoT.
The paper shows FtU can reduce discrimination without sacrificing accuracy.
problem Discrimination in machine learning predictions.
method Theoretical and empirical analysis of FtU, connecting with Model Multiplicity.
result FtU can reduce discrimination without reducing accuracy.
We prove a compactness result for minimal hypersurfaces with bounded index and volume, which can be thought of as an extension of the compactness theorem of Choi-Schoen (Invent. Math. 1985) to higher dimensions.
Responding to discussions on missing data models.
problem Clarifying causal and counterfactual views in missing data models.
method Summarizing and responding to comments from discussants.
result Summarized main contributions and addressed each discussion.
Machine learning algorithms frequently require careful tuning of model hyperparameters, regularization terms, and optimization parameters. Unfortunately, this tuning is often a "black art" that requires expert experience, unwritten rules of thumb, or sometimes brute-force search. Much more appealing is the idea of deve…