Sharp analysis improves RLHF sample complexity with KL-regularization.
problem Improving RLHF sample complexity with KL-regularization.
method Sharp analysis of KL-regularized contextual bandits and RLHF.
result Achieved an O(1/ε) sample complexity when ε is sufficiently small.
Improved fast rates for decision making with forward-KL regularization in contextual bandits.
problem Improving fast rates for decision making with forward-KL regularization in contextual bandits.
method Streamlined analysis of forward-KL-regularized offline CBs, exploiting the pessimism principle and convex-analytical pipeline.
result First ildeO(ε−1) upper bounds in tabular and general function approximation settings. Paper analyzes and improves KL-regularized RL for LLMs with logarithmic regret.
problem Improving efficiency of RL fine-tuning for large language models.
method Optimism-based KL-regularized online contextual bandit algorithm with novel regret analysis.
result Achieves an O(ηlog(NRT)⋅dR) logarithmic regret bound. KL-regularized RL from expert demos can lead to slow, unstable learning.
problem Pathological training dynamics in KL-regularized RL from expert demonstrations.
method Empirical analysis and non-parametric behavioral reference policies.
result KL-regularized RL can be significantly improved by using non-parametric behavioral policies.
KL regularization helps RL algorithms by implicitly averaging q-values.
problem Understanding why KL regularization improves RL performance.
method An approximate value iteration scheme, studying KL and entropy regularization.
result Strong performance bound combining linear horizon dependency and averaging effect of estimation errors.
Paper analyzes sample complexity of offline MABs with KL regularization.
problem Optimizing sample complexity for offline decision-making with KL-regularized metrics.
method Sharp analysis of KL-PCB, providing upper and lower bounds.
result Characterizes sample complexity for offline MABs with KL regularization.
New algorithms achieve logarithmic regret in KL-regularized Markov games.
problem Improving sample efficiency in game-theoretic settings with KL regularization.
method Developed OMG and SOMG algorithms for matrix and Markov games, using best response sampling and superoptimistic bonuses.
result Logarithmic regret in T that scales inversely with KL regularization strength β. New method reduces multi-armed bandit regret to near-optimal levels.
problem Improving regret bounds for KL-regularized multi-armed bandits.
method Sharp analysis of KL-UCB with peeling argument.
result First high-probability regret bound with linear dependence on K.
This paper solves the multiple reference model problem in RLHF with exact solutions and sample complexity guarantees.
problem Limitations of single reference models in aligning LLMs with human feedback.
method Integrates multiple reference models into RLHF frameworks, addressing theoretical challenges with exact solutions and sample complexity guarantees.
result First exact solution to the multiple reference model problem in reverse KL-regularized RLHF.
POETS optimizes LLMs by combining policy ensembles and KL regularization.
problem Balancing exploration and exploitation in decision-making and optimization.
method POETS uses policy ensembles and KL regularization to optimize LLMs efficiently.
result POETS achieves state-of-the-art sample efficiency across various domains.
New RLHF framework handles general preference oracles without reward functions.
problem Handling general preference oracles without assuming a reward function.
method Developed a minimax game between two LLMs for RLHF under a general preference oracle, focusing on KL-regularized preference.
result Proposed algorithms for efficient offline and online RLHF learning.
This work improves RLHF performance guarantees using greedy sampling.
problem Learning the KL-regularized target with preference feedback.
method General preference model, greedy sampling.
result Order-wise improvements over existing RLHF performance guarantees.
Study shows how neural networks generalize with minimal training data.
problem Understanding how neural networks generalize with limited data.
method Mean-field analysis of KL-regularized empirical risk minimization.
result Generalization error rate is O(1/n) for large n. Many real world tasks exhibit rich structure that is repeated across different parts of the state space or in time. In this work we study the possibility of leveraging such repeated structure to speed up and regularize learning. We start from the KL regularized expected reward objective which introduces an additional c…
Demonstration-regularized RL reduces sample complexity for policy identification.
problem Improving reinforcement learning's sample efficiency with expert demonstrations.
method KL-regularization of behavior cloning using expert demonstrations.
result Demonstration-regularized RL achieves optimal policy identification with reduced sample complexity.
DDO-RM improves reward-based policies by converting reward scores into a target distribution.
problem Improving reward-based policies when the reward function is simpler than the policy.
method Converts reward scores into a target distribution and uses KL-regularized mirror-descent updates.
result DDO-RM outperforms DPO in pair accuracy and mean margin.
As reinforcement learning agents are tasked with solving more challenging and diverse tasks, the ability to incorporate prior knowledge into the learning system and to exploit reusable structure in solution space is likely to become increasingly important. The KL-regularized expected reward objective constitutes one po…
Hierarchical decoupling improves sample efficiency for complex robots.
problem Learning long-range behaviors on complex robots.
method Two-part policy: low-level imitation and high-level transfer, with KL regularization.
result Hierarchical transfer significantly improves zero-shot high-level transfer and stabilizes learning.
Efficiently addresses federated learning challenges with reduced communication and sample complexity.
problem Heterogeneity in data volumes and distributions at different clients compromises model generalization ability.
method Introduces algorithms for communication-efficient Federated Group Distributionally Robust Optimization (FGDRO).
result Communication complexity reduced to O(1/ε4) for FGDRO-CVaR and O(1/ε3) for FGDRO-KL. In this paper, we introduce Deep Probabilistic Ensembles (DPEs), a scalable technique that uses a regularized ensemble to approximate a deep Bayesian Neural Network (BNN). We do so by incorporating a KL divergence penalty term into the training objective of an ensemble, derived from the evidence lower bound used in var…
New method automates asymmetric choice for better skill transfer in reinforcement learning.
problem Improving sample efficiency and transferability of reinforcement learning agents.
method Attentive Priors for Expressive and Transferable Skills (APES) using hierarchical KL-regularization.
result APES automates asymmetric choice, leading to better skill transfer across sequential tasks.
Paper tackles offline RL from mixed datasets with adaptive KL regularizer.
problem Challenges in optimizing RL and BC signals with varying action coverage and multiple action modes.
method Adaptively weighted reverse KL divergence regularizer based on TD3 algorithm.
result Empirically outperforms existing offline RL algorithms in MuJoCo locomotion tasks.
Generating paraphrases that are lexically similar but semantically different is a challenging task. Paraphrases of this form can be used to augment data sets for various NLP tasks such as machine reading comprehension and question answering with non-trivial negative examples. In this article, we propose a deep variatio…
PDGMM-VAE uses adaptive priors for better ICA recovery.
problem Nonlinear ICA recovery of latent source signals.
method Adaptive per-dimension Gaussian mixture model priors in a variational autoencoder.
result PDGMM-VAE effectively recovers source-specific non-Gaussian marginals.
Identifies optimal base features for zero-shot adaptation in reinforcement learning.
problem Unclear what constitutes a good set of base features for a wide range of downstream tasks.
method Identifies optimal base features based on downstream performance, without assuming downstream tasks are linear.
result Optimal base features are the same across three task families, differing from Laplacian eigenfunctions.
Study shows benefits of transfer learning with neural networks.
problem Understanding generalization errors in transfer learning.
method Mean-field analysis applied to α-ERM and fine-tuning. result Established conditions for generalization error and convergence rates.
The paper studies convergence rates of Tsallis entropic regularization in optimal transport.
problem Optimal transport with regularization.
method Γ-convergence and quantization/shadow arguments.
result Derives convergence rate of Tsallis entropic regularization.
Variational autoencoders (VAEs) with an auto-regressive decoder have been applied for many natural language processing (NLP) tasks. The VAE objective consists of two terms, (i) reconstruction and (ii) KL regularization, balanced by a weighting hyper-parameter β. One notorious training difficulty is that the KL term ten…
TFB simplifies Bayesian LLM uncertainty estimation without extra training.
problem Estimating uncertainty in LLM responses remains challenging.
method Training-Free Bayesianization (TFB) that transforms low-rank adapters into Bayesian ones without additional training.
result TFB achieves superior uncertainty estimation and generalization compared to existing methods.
The paper examines how updates to probabilistic models influence behavior based on evidence.
problem Understanding how updates to probabilistic models influence behavior based on evidence.
method Study of KL-regularized soft updates as Bayesian posterior updates within a single probabilistic model.
result Posterior updates determine relative incentives but not absolute rewards, which are ambiguous up to context-specific baselines.
ADPO optimizes relative advantage in reinforcement learning from human feedback.
problem Optimizing policy alignment in reinforcement learning from human preferences.
method ADPO explicitly parameterizes the optimal structure through anchored logits, decoupling response quality from prior popularity.
result Empirically, ADPO achieves state-of-the-art performance on reasoning tasks, outperforming GRPO by 30.9 percent.
Variational inference struggles with weight symmetries in neural networks, leading to biased posteriors.
problem Weight space symmetries in neural networks cause multimodal posteriors, challenging variational inference.
method Developed a symmetrization mechanism to create permutation invariant variational posteriors.
result Symmetrized variational posteriors have a better fit to the true posterior and improved predictive performance.
Unified framework for policy improvement in RL with benefits in data efficiency and computation.
problem Improving data efficiency and computation in reinforcement learning for continuous control.
method Local, regularized policy improvement with tree search for continuous action spaces.
result Improves data efficiency and reduces wall-clock time in high-dimensional domains.
This paper explores how imperfect reward models can improve online RLHF.
problem Sample efficiency in online RLHF from imperfect reward models.
method Identifies policy coverability and proposes TPO algorithm for transfer learning.
result TPO algorithm improves sample efficiency compared to standard online learning.
LanguaShrink compresses prompts using psycholinguistic principles to reduce costs.
problem Reducing computational cost and efficiency in large language model inference.
method Leverages psycholinguistic principles and the Ebbinghaus memory curve to compress prompts.
result Achieves up to 26 times compression while maintaining semantic similarity.
This work develops a unified framework for RLHF with general f-divergence regularization.
problem Theoretical understanding of general f-divergence regularization in RLHF. method Holistic approach across f-divergence class, two algorithms based on distinct sampling principles. result Provably efficient algorithms with O(logT) regret and O(1/T) sub-optimality gap. Paper addresses RLHF alignment challenges with novel algorithms.
problem Challenges in RLHF alignment, especially in strategic exploration.
method Develops a reverse-KL regularized contextual bandit formulation and proposes efficient algorithms with theoretical guarantees.
result Proposed methods significantly outperform existing RLHF algorithms in real-world experiments.
RRPI improves offline RL by optimizing policies against worst-case dynamics.
problem Offline RL's performance degrades under distribution shift and transition uncertainty.
method Formulates offline RL as robust policy optimization, treating transition kernel as decision variable.
result RRPI achieves strong average performance on D4RL benchmarks, outperforming recent baselines.
BCPO optimizes offline RL policies by converting uncertainty into conservative bounds.
problem Offline RL's fragility under distribution shifts and model errors.
method Bayesian approach with credible lower bounds and KL regularization.
result BCPO yields an uncertainty-calibrated policy that avoids exploiting model errors.
The study analyzes how neural reward models learn features for policy optimization in a Gaussian single-index model.
problem Reward modeling in policy optimization and its impact on downstream value.
method Two-stage neural reward model: first learns hidden direction, then fits readout layer.
result For any feature-learning temperature above a dimension-free threshold, a constant fraction of neurons recover the hidden direction.
This paper introduces f-DPO, a generalized approach to Direct Preference Optimization using diverse divergence constraints.
problem Aligning large language models with human preferences while mitigating safety risks.
method Incorporates diverse divergence constraints to simplify the relationship between reward and optimal policy, eliminating the need for estimating the normalizing constant.
result Optimizes LLMs to align with human preferences more efficiently and under a broader set of divergence constraints.
Constructing VAE Latent Spaces with Prescribed Topology
problem Resolving topological mismatch in VAEs for non-Euclidean data
method A constructive framework for product covering spaces
result Topology-aware latent representations with closed-form KL divergences
Proposes an online method for solving non-convex DRO with KL regularization.
problem Solving distributionally robust optimization with non-convex objectives.
method Practical online stochastic methods for DRO with KL regularization, avoiding high-dimensional dual variables and online learning issues.
result Empirical studies show significant speedup and efficiency in training deep learning models.
XPO enhances RLHF by encouraging diverse responses, offering improved sample efficiency.
problem Limited exploration in RLHF leads to suboptimal models and bottlenecks.
method XPO is a simple one-line change to DPO, introducing a novel exploration bonus.
result XPO achieves strong theoretical guarantees and promising empirical performance.
Proposes a new RL method to fine-tune flow-based models with arbitrary rewards.
problem Challenges in fine-tuning continuous flow-based generative models with arbitrary reward functions.
method Online Reward-Weighted Conditional Flow Matching with Wasserstein-2 Regularization (ORW-CFM-W2)
result Achieves optimal policy convergence with controllable trade-offs between reward maximization and diversity preservation.
Tail-Safe hedging uses reinforcement learning with a safety layer to manage financial risks.
problem Managing financial risks in derivatives trading with robustness and explainability.
method Combines distributional reinforcement learning with a CBF-QP safety layer to enforce financial constraints.
result Improves risk management without degrading central performance and avoids hard constraint violations.
A Markov Chain approach for aligning generative models from pairwise human preferences.
problem Aligning generative models from pairwise human preferences.
method Markov Chain from Human Feedback (MCHF)
result MCHF converges geometrically fast to the stationary distribution.