New method reduces fine-tuning cost for reused models.
problem Repeating fine-tuning costs with outdated foundation models.
method Portable Reward Tuning (PRT) trains a reward model to maximize the same loss function as fine-tuning.
result PRT achieves comparable accuracy to inference-time tuning with less inference cost.
Adjoint Matching improves flow and diffusion models with reward fine-tuning.
problem Improving generative models with reward fine-tuning.
method Casting reward fine-tuning as stochastic optimal control (SOC) and enforcing a specific noise schedule.
result Adjoint Matching outperforms existing SOC algorithms.
Iterative tilting fine-tunes diffusion models for reward-tilted distributions.
problem Fine-tuning diffusion models for reward-tilted distributions.
method Decomposes large reward tilts into smaller, tractable tilts via first-order Taylor expansion, avoiding backpropagation.
result Validated on a two-dimensional Gaussian mixture, achieving exact closed-form solutions.
BRAID fine-tunes diffusion models to optimize reward models in offline scenarios.
problem Combining generative modeling and model-based optimization in offline scenarios.
method Conservative fine-tuning of diffusion models using RL to optimize reward models.
result BRAID outperforms existing methods in offline data, avoiding invalid designs.
Proposes a new RL method to fine-tune flow-based models with arbitrary rewards.
problem Challenges in fine-tuning continuous flow-based generative models with arbitrary reward functions.
method Online Reward-Weighted Conditional Flow Matching with Wasserstein-2 Regularization (ORW-CFM-W2)
result Achieves optimal policy convergence with controllable trade-offs between reward maximization and diversity preservation.
New method fine-tunes discrete diffusion models for RLHF tasks.
problem Fine-tuning discrete diffusion models with policy gradient methods is challenging.
method Proposed SEPO algorithm for efficient fine-tuning over non-differentiable rewards.
result Numerical experiments show scalability and efficiency of SEPO.
A scalable algorithm for sampling and fine-tuning models using Tilt Matching.
problem Efficient sampling and fine-tuning of generative models.
method Tilt Matching, arising from a dynamical equation, minimizes variance and inherits regularity from stochastic interpolants.
result Empirically verified to be efficient and highly scalable, providing state-of-the-art results.
RAFT fine-tunes models using high-quality samples to align them with human preferences.
problem Aligning generative models with human ethics and preferences.
method RAFT selects high-quality samples, discards undesired behavior, and fine-tunes the model on filtered samples.
result RAFT improves model performance in reward learning and automated metrics.
This work improves online fine-tuning of diffusion models for specific properties.
problem Efficiently fine-tuning diffusion models to maximize specific properties.
method A novel reinforcement learning procedure that efficiently explores feasible samples.
result The method provides a regret guarantee and empirical validation across multiple domains.
MFMs enable efficient reward alignment for generative models.
problem Computational bottleneck in controlling generative models.
method Meta Flow Maps (MFMs) extend consistency models and flow maps to stochastic regime for efficient value function estimation.
result MFMs enable inference-time steering and unbiased, off-policy fine-tuning to general rewards efficiently.
This paper shows RL with KL penalties is equivalent to Bayesian inference for fine-tuning LMs.
problem Fine-tuning large language models to avoid undesirable features.
method Analyzed KL-regularized RL and showed it's equivalent to variational inference.
result KL-regularized RL avoids distribution collapse and is more insightful as Bayesian inference.
Improved RL training for DMs reduces mode collapse and preserves diversity.
problem Mode collapse and training instability in RL fine-tuned diffusion models.
method Dynamic hierarchical RL training with sliding-window parameter regularisation.
result Models trained with HRF achieve better preservation of diversity in downstream tasks.
This work classifies SOC loss functions based on their gradient properties.
problem Optimizing noisy systems in stochastic optimal control.
method Grouping loss functions into classes with the same gradient expectation.
result Different loss functions have the same optimization landscape but differ in gradient variance.
New method optimizes diffusion models without fine-tuning, integrating soft value functions.
problem Optimizing natural design spaces of images, molecules, DNA, RNA, and protein sequences.
method Iterative sampling method integrating soft value functions into diffusion model inference.
result Directly utilizes non-differentiable features/reward feedback, applies to discrete diffusion models.
Fine-tunes diffusion models to generate diverse samples with high genuine rewards.
problem Reward collapse in finetuning diffusion models.
method Entropy-regularized control against pretrained diffusion models.
result Efficient generation of diverse samples with high genuine rewards.
Paper fine-tunes LLMs using user edits, unifying preference, supervision, and reward feedback.
problem Adapting LLMs to user preferences and feedback types.
method Derives bounds for learning algorithms from user edits, proposes an ensembling procedure.
result Ensembling procedure outperforms individual feedback methods and robustly adapts to different user-edit distributions.
Paper proposes a hybrid RL algorithm that combines offline and online data without needing reward info.
problem How to efficiently use online data to improve RL policies using only offline data.
method A three-stage hybrid RL algorithm that uses reward-agnostic exploration and model-based offline RL.
result The hybrid RL algorithm outperforms both pure offline and pure online RL in sample complexity.
The paper develops a theory for iterative self-improvement of models, proving conditions for better performance with easy-to-hard curricula.
problem Lack of theoretical foundation for iterative self-improvement in practical settings.
method Modeling self-improvement as maximum-likelihood fine-tuning on reward-filtered distributions and proving finite-sample guarantees.
result Explicit feedback loop and conditions for better performance with easy-to-hard curricula.
This tutorial reviews RL-based methods for optimizing diffusion models to maximize specific metrics.
problem Optimizing diffusion models to generate samples that maximize specific metrics in practical applications.
method Various RL algorithms including PPO, differentiable optimization, reward-weighted MLE, value-weighted sampling, and path consistency learning.
result Exploration of strengths and limitations of RL-based fine-tuning algorithms and their benefits compared to non-RL-based approaches.
New approach uses inverse reinforcement learning to improve language model training.
problem Training large language models using imitation learning methods.
method Developed a new method of inverse reinforcement learning to optimize sequences directly.
result IRL-based fine-tuning leads to better performance and diversity in language generation.
Transformers learn sparse Boolean functions through RL and SFT, revealing distinct learning behaviors.
problem Learning sparse Boolean functions with Transformers.
method Reinforcement Learning (RL) with process rewards and Supervised Fine-Tuning (SFT).
result RL learns the whole CoT chain simultaneously, while SFT learns step by step.
Behavior Transfer improves reinforcement learning by leveraging pre-trained policies.
problem Efficient transfer of knowledge in reinforcement learning.
method Behavior Transfer (BT) that uses pre-trained policies for exploration.
result BT combined with pre-training leads to better solutions than without pre-training.
Fine-tuning LLMs with observational data can lead to spurious correlations, but DeconfoundLM can mitigate this.
problem Aligning LLMs with human preferences and business objectives using observational data.
method DeconfoundLM, a method that removes confounders from reward signals.
result DeconfoundLM improves recovery of causal relationships and mitigates spurious correlations.
New method improves language model fine-tuning without forgetting.
problem Fine-tuning language models to match specific distributions without forgetting.
method Combines Distribution Matching and Reinforcement Learning techniques.
result Adding a baseline improves constraint satisfaction, stability, and efficiency.
DTM improves dLLM fine-tuning stability and performance.
problem Intractable sequence-level marginal likelihoods for masked diffusion models.
method Discrete Tilt Matching (DTM) recasts dLLM fine-tuning as state-level matching of local unmasking posteriors under reward tilting.
result DTM yields strong gains on Sudoku and Countdown while remaining competitive on MATH500 and GSM8K.
Unified framework for training diffusion and flow models to sample from target distributions.
problem Training diffusion and flow models to sample from target distributions defined by exponential tilting.
method Unified framework combining stochastic optimal control and non-equilibrium thermodynamics perspectives.
result Unified bias-variance decompositions and theoretical support for adjoint-based methods.
Generative models often misrepresent class frequencies; this paper calibrates them.
problem Miscalibration of class frequencies in generative models.
method Formulated as constrained optimization, using surrogate objectives to approximate constraints.
result Significant reduction in calibration error across various models and applications.
Two methods for model adaptation compared; fine-tuning outperforms Best-of-N in realizable settings.
problem Comparing methods for adapting large language models to new tasks.
method Supervised fine-tuning vs. Best-of-N approach.
result Supervised fine-tuning outperforms Best-of-N in realizable settings.
A new approach to fine-tuning LLMs with human feedback.
problem Inability of current reward models to fully represent human preferences.
method Introducing NLHF, a new pipeline for LLM fine-tuning using pairwise human feedback.
result NLHF produces a sequence of policies converging to the regularized Nash equilibrium.
EQD model improves domain-specific QA by 0.6% to 10.5%.
problem Challenges in domain-specific quantitative reasoning for LLMs.
method Two-step fine-tuning framework guided by a reward function.
result EQD outperforms state-of-the-art models and prompting strategies.
Paper resolves bias in ALFT training using generalized alignment games.
problem Systematic bias in estimating logarithmic rewards from small batches.
method Generalized Distributional Alignment Games, U-statistics, minimax polynomial estimators, Variance-Optimal Augmented Polynomial Optimization Program (AQP) Estimator.
result Proves optimal bias and accelerated convergence in ALFT training.
Unified LLM-agent with RL improves financial trading performance.
problem LLMs struggle with complex, multi-step financial tasks.
method Fusion of LLMs and gradient-based RL for policy optimization.
result Improves LLM performance in trading and other financial tasks.
FinRLlama wins FinRL Challenge 2024 by fine-tuning LLMs with market data.
problem Lack of contextual alignment for financial market applications in traditional LLMs.
method Fine-tuning LLaMA-3.2-3B-Instruct model with custom RLMF prompt design integrating historical data and reward feedback.
result RLMF-tuned FinRLlama framework outperforms baseline methods in signal consistency and trading outcomes.
Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and…
New method learns to answer questions from correct demonstrations without assuming bounded complexity of the demonstrator.
problem Learning to generate answers from correct demonstrations with multiple correct answers.
method Formalizes imitation learning in contextual bandits, relying on reward model complexity, not policy complexity.
result Achieves nearly optimal performance with sample complexity logarithmic in reward class cardinality.
Unified framework improves diffusion model rewards without full trajectories.
problem Limited theoretical understanding of guided diffusion samplers.
method Developed a unified algorithmic and theoretical framework for diffusion guidance and reward-guided diffusion.
result Framework shows CFG decreases expected reciprocal of classifier probability.
Study shows how a strong model can learn a task's feature while retaining other capabilities.
problem How to align superhuman AI systems using weak-to-strong generalization.
method Two-layer neural networks, reward-model learning, multi-step SGD, feature learning.
result The strong model efficiently learns task features while retaining general capabilities.
Tutorial on optimizing diffusion model samples for specific metrics.
problem Optimizing diffusion model samples for specific downstream metrics.
method Review and exploration of inference-time guidance and alignment methods.
result Unified perspective on inference-time algorithms and novel methods.
Improves text-to-image diffusion models using GFlowNets.
problem Aligning diffusion models with text descriptions.
method Post-training diffusion models with GFlowNets to generate high-reward images.
result Effective alignment of large-scale text-to-image diffusion models with reward information.
Proposes a new theoretical framework for PbRL that requires less human feedback.
problem Lack of theoretical work capturing practical PbRL frameworks.
method Introduces a reward-agnostic PbRL framework that acquires exploratory trajectories before human feedback.
result Demonstrates improved sample complexity for learning optimal policies in linear and low-rank MDPs.
CARDS improves decoding efficiency and alignment quality for LLMs.
problem Efficiency bottlenecks in decoding-time alignment for LLMs.
method Cascade Reward Sampling (CARDS) with segment-level rejection sampling and uncertainty-based segmentation.
result Significant improvement in decoding efficiency and alignment quality.
Paper proposes a robust RLHF algorithm for LLMs, improving response preference over baselines.
problem Reward model misspecifications in RLHF for LLMs.
method Proposes a robust algorithm to reduce reward and policy estimator variance, theoretically and empirically validated.
result Consistently outperforms existing methods on LLM benchmark datasets, favoring 77-81% of responses over baselines.
DPA aligns LLMs with multi-objective rewards for diverse user preferences.
problem Fine-grained control over LLMs for diverse user needs.
method Integrates multi-objective reward modeling and directional preference control.
result DPA offers better performance trade-offs and intuitive user control over LLM generation.
Efficiently adapting to new environments and changes in dynamics is critical for agents to successfully operate in the real world. Reinforcement learning (RL) based approaches typically rely on external reward feedback for adaptation. However, in many scenarios this reward signal might not be readily available for the …
The paper enhances preference learning by incorporating response time data.
problem Lack of temporal information in user decision-making for reward model learning.
method Integrates response time alongside binary choice data using the EZ model and Neyman-orthogonal loss functions.
result Response time-augmented approach reduces error rates from exponential to polynomial scaling, improving sample efficiency.
Proposes DAM for optimizing discrete generative models.
problem Challenges in optimizing discrete generative models.
method Discrete Adjoint Matching (DAM) for discrete state spaces.
result Demonstrates effectiveness on synthetic and mathematical reasoning tasks.
Paper develops methods to optimize policies directly from human feedback without reward inference.
problem Challenges in RLHF, including reward model overfitting and distribution shift.
method Develops two algorithms for RLHF without reward inference, using zeroth-order gradient approximators.
result Establishes polynomial convergence rates and outperforms existing methods in numerical experiments.
Generative Adversarial Networks (GANs) enjoy great success at image generation, but have proven difficult to train in the domain of natural language. Challenges with gradient estimation, optimization instability, and mode collapse have lead practitioners to resort to maximum likelihood pre-training, followed by small a…