Transformers learn sparse Boolean functions through RL and SFT, revealing distinct learning behaviors.
problem Learning sparse Boolean functions with Transformers.
method Reinforcement Learning (RL) with process rewards and Supervised Fine-Tuning (SFT).
result RL learns the whole CoT chain simultaneously, while SFT learns step by step.
Theoretical analysis of data quality and synergies in LLMs.
problem Understanding why different training methods require different amounts of data.
method Theoretical analysis of transformers trained on a weight prediction task for linear regression.
result SFT excels on smaller datasets challenging for the pretrained model, while RL benefits from large, not overly difficult data.
We sketch a construction of Legendrian Symplectic Field Theory (SFT) for conormal tori of knots and links. Using large N duality and Witten's connection between open Gromov-Witten invariants and Chern-Simons gauge theory, we relate the SFT of a link conormal to the colored HOMFLY-PT polynomials of the link. We presen…
Simplified computation of SFT invariants for Legendrian links.
problem Computing SFT invariants for Legendrian links is combinatorially intractable.
method Left-right-simplification of Legendrian links and analysis of holomorphic maps.
result SFT invariants of Legendrian links are combinatorially computable using disks with ≤ 2 positive punctures.
E2C separates planning and execution in LLMs, improving efficiency and performance.
problem Entangled planning and execution in LLMs waste tokens and limit flexibility.
method E2C splits exploration and execution phases, using SFT and RL for training.
result E2C achieves 53.3% accuracy on AIME'2024 with 12.4k tokens, outperforming alternatives.
InfoSFT improves LLMs by focusing on informative, medium-confidence tokens.
problem Overfitting to unlikely samples and degradation of prior capabilities in SFT.
method InfoSFT uses a principled weighting scheme to concentrate learning signals on medium-confidence tokens.
result InfoSFT improves generalization and preserves pre-existing capabilities over vanilla SFT and likelihood-weighted baselines.
We construct infinitely many non-diffeomorphic examples of 5-dimensional contact manifolds which are tight, admit no strong fillings, and do not have Giroux torsion. We obtain obstruction results for symplectic cobordisms, for which we give a proof not relying on the polyfold abstract perturbation scheme for SFT. The…
This is the first of two papers devoted to showing how the rich algebraic formalism of Eliashberg-Givental-Hofer's symplectic field theory (SFT) can be used to define higher algebraic structures on the symplectic cohomology of open symplectic manifolds. Using the SFT of Hamiltonian mapping tori we show how to define a …
New framework improves LLM performance by avoiding forgetting during sequential training stages.
problem Forgetting during sequential training stages of LLMs.
method Proposes a joint post-training framework with theoretical convergence guarantees.
result Empirically outperforms sequential post-training framework by up to 23%.
Self-improvement refines language models by verifying their own outputs.
problem Improving language models without external feedback.
method Formalizing self-improvement as sharpening, using the model itself as a verifier.
result RLHF-based self-improvement can outperform SFT-based methods.
Mitigates overoptimization in RLHF by reformulating SFT loss as a preference optimization loss.
problem Overoptimization in RLHF where reward model misguides generative model.
method Proposes a theoretical algorithm that minimizes maximum likelihood estimation and reward penalty, reformulates as simple objective combining preference optimization and supervised learning losses.
result Improved performance of RPO compared to DPO baselines in aligning LLMs.
In this paper we show how the rich algebraic formalism of Eliashberg-Givental-Hofer's symplectic field theory (SFT) can be used to define higher algebraic structures in Hamiltonian Floer theory. Using the SFT of Hamiltonian mapping tori we show how to define a homotopy extension of the well-known Lie bracket and discus…
Study shows multilingual LLM calibration effects improve model confidence but not accuracy.
problem Improving multilingual language model calibration in low-resource settings.
method Analysis of two multilingual benchmarks using instruction-tuning and label smoothing.
result Model confidence increases in low-resource languages after instruction-tuning but accuracy improvements are marginal.
Vanishing gradients hinder reinforcement finetuning of language models.
problem Vanishing gradients impede the optimization of language models using reinforcement finetuning.
method The study identifies vanishing gradients as a fundamental optimization obstacle in reinforcement finetuning and proposes an initial supervised finetuning phase to mitigate this issue.
result An initial supervised finetuning phase is crucial for successful reinforcement finetuning of language models, as it helps prevent vanishing gradients and maximizes rewards.
Fine-tunes LLMs to correct bias in predictions.
problem LLMs exhibit bias in predictions from data.
method Supervised fine-tuning with Low-Rank Adaptation (LoRA).
result Fine-tuning corrects bias in both controlled and real-world settings.
Study optimizes compute usage for LLM web agents, improving performance.
problem High compute costs and narrow focus on single-step tasks limit LLM web agents.
method Two-stage pipeline: SFT followed by on-policy RL, with hyperparameter optimization.
result Combining SFT and on-policy RL requires 55% less compute to match peak SFT performance.
L3Ms fine-tune LLMs with constraints for tailored applications.
problem Inadequate alignment of LLMs for diverse applications.
method Formulate SFT and alignment as constrained optimization, using logarithmic barriers.
result Versatile and effective in achieving tailored alignments for various applications.
New framework analyzes LLM personalization trade-offs under congestion.
problem Tension between personalization and resource sharing in LLMs.
method Developed a statistical-economic framework to model user incentives.
result Congestion can flip rankings of SFT and ICL, and offers both methods never hurt profits.
FisherSFT selects informative examples to fine-tune LLMs efficiently.
problem Adapting large language models to new domains efficiently.
method Selects examples maximizing information gain using Hessian of log-likelihood.
result Empirically demonstrates improved performance with reduced computational cost.
Develops gluing theory for contact instantons and pseudoholomorphic curves.
problem Constructing contact instanton Floer cohomology and Fukaya-type category.
method Gluing theory of contact instantons and pseudoholomorphic curves in symplectization context.
result Construction of Legendrian contact instanton homology and moduli spaces of holomorphic buildings.
Contact squeezing prevented in certain prequantized balls via generating functions.
problem Preventing contact squeezing in prequantized balls of different radii.
method Equivariant generating function homology with finite cyclic group action.
result Contact squeezing not possible in specified prequantized balls.
Curriculum learning improves model training efficiency and performance.
problem Expensive training of reasoning models using human or synthetic data.
method Autocurriculum, where models use their own performance to decide training problems.
result Autocurriculum reduces training costs and improves model performance.
Develops a method to control model behavior with synthetic data.
problem Controlling language model behavior via synthetic training data.
method Dataset Policy Gradient (DPG) to optimize synthetic data generators.
result Can cause target models to embed specific patterns or rephrase inputs.
New approach uses inverse reinforcement learning to improve language model training.
problem Training large language models using imitation learning methods.
method Developed a new method of inverse reinforcement learning to optimize sequences directly.
result IRL-based fine-tuning leads to better performance and diversity in language generation.
The study examines label smoothing to improve confidence calibration in fine-tuned LLMs.
problem Improving confidence calibration in fine-tuned large language models (LLMs) after instruction tuning.
method Examine various open-sourced LLMs, label smoothing, and custom kernel design.
result Label smoothing is effective in maintaining confidence calibration but faces challenges in large vocabulary LLMs.
FinDPO uses preference optimization to improve financial sentiment analysis models.
problem Financial sentiment analysis models often fail to generalize to unseen data.
method FinDPO uses Direct Preference Optimization (DPO) to align LLMs with human preferences.
result FinDPO achieves state-of-the-art performance and maintains positive returns under realistic trading conditions.
Painless Activation Steering automates post-training for LMs without manual intervention.
problem Manual post-training methods are time-consuming and labor-intensive.
method Painless Activation Steering (PAS) is a fully automated approach that requires no manual intervention.
result PAS reliably improves performance for behavior tasks but not for intelligence-oriented tasks.
SPIN converts weak LLMs to strong ones using self-play.
problem Growing strong LLMs without human-annotated data.
method Self-play fine-tuning (SPIN) starting from a supervised fine-tuned model.
result SPIN significantly improves LLM performance across benchmarks.
This paper classifies symplectic and Stein fillings of contact 3-manifolds with spinal open book decompositions.
problem Classifying symplectic and Stein fillings of contact 3-manifolds with spinal open book decompositions.
method Using holomorphic curves and Lefschetz fibrations to classify fillings.
result Symplectic and Stein fillings of contact 3-manifolds with spinal open book decompositions can be classified up to deformation equivalence.
NIFTY dataset for financial forecasting models.
problem Improving financial market forecasting with LLMs.
method Two datasets for supervised and reinforcement learning.
result Demonstrates applications in stock price prediction.
RL enhances LLM planning but introduces spurious solutions and diversity collapse.
problem Theoretical understanding of RL's benefits and limitations in LLM planning.
method Graph-based abstraction, policy gradient, Q-learning, supervised fine-tuning.
result RL's exploration is crucial for generalization, but PG suffers from diversity collapse.
RLVR maintains safety while improving reasoning capabilities in LLMs.
problem Safety-capability tradeoff in fine-tuning LLMs.
method Reinforcement Learning with Verifiable Rewards (RLVR) and theoretical analysis.
result RLVR can enhance reasoning while maintaining safety guardrails.
Success conditioning optimizes policies by imitating successful trajectories, solving a trust-region optimization problem.
problem Improving policies through random actions that lead to desired outcomes.
method Success conditioning, which involves collecting and updating policies based on successful trajectories.
result Success conditioning solves a trust-region optimization problem, maximizing policy improvement with a χ2 divergence constraint. This work proves win rate is key to understanding preference learning.
problem Understanding preference learning from generative models.
method Analyzing preference learning methods as win rate optimization or non-WRO.
result Proves win rate is the only evaluation respecting preferences and prevalences.
New loss functions based on f-divergences improve language model performance.
problem Improving multiclass classification and language modeling performance.
method Constructing new convex loss functions using f-divergences and deriving an operator for computation.
result The α-divergence loss function with α=1.5 performs well across various tasks. New method learns to answer questions from correct demonstrations without assuming bounded complexity of the demonstrator.
problem Learning to generate answers from correct demonstrations with multiple correct answers.
method Formalizes imitation learning in contextual bandits, relying on reward model complexity, not policy complexity.
result Achieves nearly optimal performance with sample complexity logarithmic in reward class cardinality.
Enhances math problem-solving models with multi-turn preference learning.
problem Improving mathematical problem-solving capabilities of large language models.
method Introduces a multi-turn direct preference learning framework for tool-integrated mathematical reasoning tasks.
result Significant performance improvements in model accuracy on math datasets.
We provide an online RLHF workflow for large language models.
problem Lack of online RLHF workflow for large language models.
method Construct preference models using open-source datasets, approximate human feedback, and implement online iterative RLHF.
result Our LLM achieves state-of-the-art performance on various benchmarks.
MM-DREX adapts LLM experts for financial trading via dynamic routing.
problem Challenges of non-stationary financial markets and static expert designs.
method MM-DREX uses a VLM-powered dynamic router to allocate expert weights and designs heterogeneous trading experts.
result Significantly outperforms 15 baselines across key metrics.
Active-GRPO improves molecular optimization by actively deciding when to imitate or self-improve.
problem Training robust and efficient molecular optimization models with large language models.
method Active-GRPO combines imitation and reinforcement learning, upgrading references and policies dynamically.
result Improves molecular optimization performance, achieving statistically significant gains.
New findings explain why online methods outperform offline methods in noisy expert feedback settings.
problem The challenge of learning from imperfect expert feedback in sequential decision-making systems.
method Introduced a noisy expert model and a novel variant of on-policy distillation (OPD) to address the gap between offline and online imitation learning.
result Online interaction with a noisy expert via OPD enables polynomial dependence on the horizon, unlike offline methods which require exponential growth in sample complexity.