TROLL improves RL for LLMs by replacing clipping with a trust region projection.
problem Clipping in RL for LLMs causes instability and suboptimal performance.
method TROLL uses a discrete differentiable trust region projection to replace clipping, balancing computational cost and effectiveness.
result TROLL consistently outperforms PPO-like clipping in training speed, stability, and final success rates.
Topic models are typically evaluated with respect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic assignments. Token-level assignments are important for downstream tasks such as classification. Even recent models, which aim to improv…
Proposes integrating global and local entropy for more reliable LLMs.
problem Uncertainty in large language models (LLMs) leads to unreliable predictions.
method Measures global uncertainty from hidden-state matrices and local uncertainty from tokens, combining them via a multiplicative gate.
result Global-Local Uncertainty (GLU) outperforms unsupervised baselines across multiple models and benchmarks.
TRM improves long-horizon LLM RL by masking divergent sequences.
problem Long-horizon reinforcement learning with LLMs suffers from off-policy mismatch and approximation errors.
method Derives and applies trust region bounds to control divergence, proposing Trust Region Masking.
result First non-vacuous monotonic improvement guarantees for long-horizon LLM-RL.
TRM improves long-horizon reinforcement learning for LLMs by masking divergent sequences.
problem Long-horizon reinforcement learning for LLMs suffers from off-policy mismatch and approximation errors.
method Derives and applies trust region bounds to control divergence, proposing Trust Region Masking.
result First non-vacuous monotonic improvement guarantees for long-horizon LLM-RL.
The task of Natural Language Inference (NLI) is widely modeled as supervised sentence pair classification. While there has been a lot of work recently on generating explanations of the predictions of classifiers on a single piece of text, there have been no attempts to generate explanations of classifiers operating on …
Uncertainty estimation is important for ensuring safety and robustness of AI systems. While most research in the area has focused on un-structured prediction tasks, limited work has investigated general uncertainty estimation approaches for structured prediction. Thus, this work aims to investigate uncertainty estimati…
Language models perform worse with implicit reward models than explicit ones.
problem Understanding why implicit reward models generalize worse than explicit ones.
method Investigated the root cause of the generalization gap between IM-RMs and EX-RMs.
result Implicit reward models rely more on superficial token-level cues, leading to worse generalization.
DOS improves language model generation by considering inter-token dependencies.
problem Lack of sequence-level information and inter-token dependencies in existing decoding strategies.
method Dependency-Oriented Sampler (DOS) that uses attention matrices to approximate inter-token dependencies.
result DOS consistently achieves superior performance on code generation and mathematical reasoning tasks.
Charmer improves character-level adversarial attacks for language models.
problem Efficiency and effectiveness of character-level adversarial attacks for language models.
method Query-based adversarial attack method that maintains semantic similarity.
result Charmer achieves high attack success rate and similar adversarial examples.
A new sampler for FLMs improves token-level decoding controls.
problem Sampling from FLMs using standard methods collapses marginals and produces invalid sequences.
method Samples clean one-hot endpoints from FLM token marginals and uses Ornstein-Uhlenbeck bridges conditioned on these endpoints.
result The method preserves token-wise posterior-predictive marginals and improves quality-diversity tradeoff.
New L2D framework allows deferring specific parts of a sequence prediction to experts.
problem Current L2D methods defer entire predictions, which is not ideal for long sequences.
method Proposes token-level and one-time rejectors to defer specific outputs of a model prediction to experts.
result Granular deferrals achieve better cost-accuracy tradeoffs than whole deferrals.
Efficiently estimates uncertainty for LLM-based entity linking in tabular data.
problem Accurate and reliable uncertainty estimates for LLM-based entity linking in tabular data.
method Self-supervised approach using token-level features for single-shot inference.
result Effective uncertainty estimates detected at a fraction of computational cost.
Paper analyzes inclusive KL inference using Wasserstein gradient flows.
problem Analyzing inclusive KL inference with mathematical tools.
method Gradient flows derived from PDE analysis.
result Unified view of existing sampling algorithms as inclusive-KL inference.
Funnel-Transformer reduces computation by compressing sequence data.
problem Redundant token-level representations in language processing.
method Gradually compresses sequence of hidden states to a shorter one.
result Funnel-Transformer outperforms standard Transformer with fewer FLOPs.
TSC uses HMC and adaptive transport maps to optimize forward KL for variational inference.
problem Variational inference underestimates uncertainty when minimizing reverse KL.
method TSC uses Hamiltonian Monte Carlo and adaptive transport maps to optimize KL(p||q).
result TSC achieves competitive performance in training variational autoencoders on large-scale data.
A classic setting of the stochastic K-armed bandit problem is considered in this note. In this problem it has been known that KL-UCB policy achieves the asymptotically optimal regret bound and KL-UCB+ policy empirically performs better than the KL-UCB policy although the regret bound for the original form of the KL-UCB…
Improved fast rates for decision making with forward-KL regularization in contextual bandits.
problem Improving fast rates for decision making with forward-KL regularization in contextual bandits.
method Streamlined analysis of forward-KL-regularized offline CBs, exploiting the pessimism principle and convex-analytical pipeline.
result First i l d e O ( ε − 1 ) ilde{O}(ε^{-1}) i l d e O ( ε − 1 ) upper bounds in tabular and general function approximation settings. Causal KL improves on existing metrics for evaluating causal models.
problem Insufficient discrimination between causal models using edit-distance and KL divergence.
method Introducing Causal KL, an augmented KL divergence that considers causal relationships.
result Causal KL variants effectively distinguish between observationally equivalent models.
New algorithm minimizes inclusive KL for VI, improving accuracy.
problem Improving variational inference accuracy with KL(p||q).
method Markovian score climbing (MSC) using stochastic gradients.
result MSC converges to local optimum of inclusive KL without bias.
Sharp analysis improves RLHF sample complexity with KL-regularization.
problem Improving RLHF sample complexity with KL-regularization.
method Sharp analysis of KL-regularized contextual bandits and RLHF.
result Achieved an O(1/ε) sample complexity when ε is sufficiently small.
Language models allocate information storage, not collapsing into uniform representations.
problem Incomplete neural collapse in language model representations.
method Analyzing variance and information sharing across 14 models, proving an information floor.
result Within-class variance is allocated information storage, not collapsed into uniform representations.
Kurdyka-Lojasiewicz (KL) exponent plays an important role in estimating the convergence rate of many contemporary first-order methods. In particular, a KL exponent of 1 2 \frac12 2 1 for a suitable potential function is related to local linear convergence. Nevertheless, KL exponent is in general extremely hard to estimate. I…
The paper develops new algorithms for KL-divergence NMF, proving convergence and performance.
problem Improving NMF for nonnegative data with KL divergence.
method Collect and analyze properties of KL objective function, propose and test new algorithms.
result Guaranteed non-increasing objective function for one proposed algorithm, global convergence.
In this paper we propose a deep neural network model with an encoder-decoder architecture that translates images of math formulas into their LaTeX markup sequences. The encoder is a convolutional neural network (CNN) that transforms images into a group of feature maps. To better capture the spatial relationships of mat…
This paper tightens the law of the iterated logarithm for empirical KL_inf, applicable to unbounded data.
problem Developing nonasymptotic concentration bounds for empirical KL_inf with optimal constants and rates.
method Presenting a tight law of the iterated logarithm for empirical KL_inf, applicable to unbounded data.
result A tight law of the iterated logarithm for empirical KL_inf, applicable to unbounded data.
Paper analyzes and improves KL-regularized RL for LLMs with logarithmic regret.
problem Improving efficiency of RL fine-tuning for large language models.
method Optimism-based KL-regularized online contextual bandit algorithm with novel regret analysis.
result Achieves an O ( η log ( N R T ) ⋅ d R ) \mathcal{O}\big(η\log (N_{\mathcal R} T)\cdot d_{\mathcal R}\big) O ( η log ( N R T ) ⋅ d R ) logarithmic regret bound. ECLIPSE detects AI hallucinations in finance with high accuracy.
problem Hallucinations in AI-generated answers limit safe deployment in finance.
method Combines entropy estimation and perplexity decomposition to measure model evidence use.
result ECLIPSE achieves ROC AUC of 0.89 and average precision of 0.90 on financial QA dataset.
Paper relaxes triangle inequality for KL divergence between Gaussian distributions.
problem KL divergence does not satisfy triangle inequality for Gaussian distributions.
method Investigates relaxed triangle inequality and finds supremum.
result Supremum of KL divergence is found and conditions for attaining it are determined.
This paper fine-tunes LLMs for stock return prediction using financial news.
problem Improving stock return forecasting accuracy using LLMs.
method Fine-tuning LLMs with text and forecasting modules, comparing encoder-only and decoder-only models, and integrating token-level representations.
result LLMs' aggregated token-level embeddings enhance return predictions for long-only and long-short portfolios.
This study explains RL training dynamics in LLMs, focusing on token-level optimization and reasoning pattern reshaping.
problem Understanding the training dynamics of RL in LLMs to improve their reasoning capabilities.
method Empirical analysis and theoretical modeling of RL training process, focusing on reasoning patterns and token optimization.
result RL primarily optimizes a sparse subset of critical tokens, reshaping reasoning pattern distributions and affecting model performance.
Theory for RLHF generalization under reward shift and clipped KL.
problem Theoretical understanding of RLHF generalization, especially with reward shift and clipped KL.
method Developed generalization theory for RLHF, accounting for reward shift and clipped KL.
result Presented generalization bounds for RLHF, suggesting generalization error from sampling, reward shift, and KL clipping.
KALE flow approximates KL divergence for distributions with disjoint support.
problem Approximating KL divergence for distributions with disjoint support.
method Relaxed KL gradient flow using RKHS, continuously interpolating between KL and MMD.
result Global convergence of KALE flow under sufficient smoothness assumptions.
The Dirichlet mechanism protects privacy while minimizing KL divergence.
problem Minimizing KL divergence while protecting sensitive data privacy.
method Using the exponential mechanism with the KL divergence loss function, resulting in the Dirichlet mechanism.
result Proved a probability tail bound on KL divergence and derived a lower bound for sample complexity.
Improved bounds for estimating discrete distributions in KL divergence.
problem Estimating discrete distributions in KL divergence with accuracy.
method Used Laplace estimator and established concentration bounds.
result Deviation from mean scales as k / n \sqrt{k}/n k / n for n ≥ k n \ge k n ≥ k . Paper analyzes sample complexity of offline MABs with KL regularization.
problem Optimizing sample complexity for offline decision-making with KL-regularized metrics.
method Sharp analysis of KL-PCB, providing upper and lower bounds.
result Characterizes sample complexity for offline MABs with KL regularization.
In this work, we study the credit assignment problem in reward augmented maximum likelihood (RAML) learning, and establish a theoretical equivalence between the token-level counterpart of RAML and the entropy regularized reinforcement learning. Inspired by the connection, we propose two sequence prediction algorithms, …
Paper proposes a method to stabilize estimation of KL divergence using a discriminator in RKHS.
problem High variance and instability in estimating KL divergence using neural network discriminators.
method Developed a novel construction of the discriminator in RKHS, controlled its complexity, and proved the consistency of the estimator.
result Reduced variance and stabilized training of KL divergence estimates.
Local mass perspective on Bayesian inference
problem Measuring distributional discrepancy in Bayesian inference
method Introducing Mass Index and Regularised Extended KL
result Proving inequalities for comparing local small-ball masses
Improved KL bounds and Wasserstein guarantees for diffusion flow matching under minimal conditions.
problem Theoretical convergence properties of Brownian motion based diffusion flow matching.
method Refined analysis under Kullback-Leibler and 2-Wasserstein distances.
result State-of-the-art scaling in KL convergence bounds under minimal conditions.
New method trains neural samplers to sample from multi-modal distributions efficiently.
problem Mode-seeking behavior of reverse KL divergence hinders effective sampling from multi-modal target distributions.
method Minimizing reverse diffusive KL divergence along diffusion trajectories of model and target densities.
result Demonstrated enhanced sampling performance across various multi-modal distributions.
KL-regularized RL from expert demos can lead to slow, unstable learning.
problem Pathological training dynamics in KL-regularized RL from expert demonstrations.
method Empirical analysis and non-parametric behavioral reference policies.
result KL-regularized RL can be significantly improved by using non-parametric behavioral policies.
New KL-divergence for Gaussian distributions based on Wasserstein geometry.
problem Computing KL-divergence for Gaussian distributions efficiently.
method Introducing WKL-divergence based on Wasserstein geometry.
result WKL-divergence evaluates to squared distance between points for Dirac measures.
KL regularization helps RL algorithms by implicitly averaging q-values.
problem Understanding why KL regularization improves RL performance.
method An approximate value iteration scheme, studying KL and entropy regularization.
result Strong performance bound combining linear horizon dependency and averaging effect of estimation errors.
New algorithms achieve logarithmic regret in KL-regularized Markov games.
problem Improving sample efficiency in game-theoretic settings with KL regularization.
method Developed OMG and SOMG algorithms for matrix and Markov games, using best response sampling and superoptimistic bonuses.
result Logarithmic regret in T T T that scales inversely with KL regularization strength β β β . SRFE clarifies KL divergences without unifying learning frameworks.
problem Inductive biases of KL divergences and their limitations.
method Introducing SRFE, a log-moment-based functional of the likelihood ratio.
result SRFE recovers KL divergences as limits and reveals a mean-variance tradeoff.
Unified view of KL-divergence and IPMs via DRE, with new DRM metrics.
problem Unified understanding of KL-divergence and IPMs.
method Unified representation via maximum likelihood density-ratio estimation (DRE).
result Unified form of IPMs and novel DRM metrics.
New method tackles nonconvex-nonconcave problems with local KL condition.
problem Nonconvex-nonconcave minimax problems under varying KL conditions.
method Inexact proximal gradient method for KL-structured subproblems.
result Complexity guarantees for approximate stationary points.