Paper improves reinforcement learning efficiency with deterministic value gradients.
problem High sample complexity in model-free DDPG algorithms for continuous control tasks.
method Proposes DVG and DVPG algorithms with infinite horizon value gradients to improve sample efficiency.
result DVPG algorithm substantially outperforms state-of-the-art methods on continuous control benchmarks.
Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…
New method improves deep policy gradient algorithms by learning relative state values.
problem High sample complexity and instability in policy gradient methods.
method Uses a new state-value function approximation based on residual variance.
result Empirical improvement across diverse continuous control tasks and algorithms.
Enhances SVGD with matrix-valued kernels for faster inference.
problem Efficient approximate inference in complex probability landscapes.
method Integrates geometric information through matrix-valued kernels in SVGD.
result Significant improvement in real-world Bayesian inference tasks.
Wasserstein gradient boosting predicts probability distributions for supervised learning.
problem Distribution-valued supervised learning where outputs are probability distributions.
method Fits a new weak learner to Wasserstein gradients of loss functionals of probability distributions.
result Superior performance in probabilistic prediction compared to existing methods.
We study how the behavior of deep policy gradient algorithms reflects the conceptual framework motivating their development. To this end, we propose a fine-grained analysis of state-of-the-art methods based on key elements of this framework: gradient estimation, value prediction, and optimization landscapes. Our result…
Under-parameterization hinders deep RL's efficiency.
problem Implicit under-parameterization impairs data-efficiency in deep RL.
method Characterized and mitigated the rank collapse of value network features.
result Controlling rank collapse improves deep RL performance.
Paper finds unique solutions for curved surfaces with specific gradient.
problem Existence of curved surfaces with specific gradient.
method Second boundary value problem of constant mean curvature equations.
result Unique convex solutions for constant mean curvature equations.
MAGE optimizes policies using action gradients from model-based learning.
problem Lack of direct gradient information from critics in actor-critic methods.
method Model-based actor-critic algorithm that learns action-value gradient.
result MAGE outperforms model-free and model-based baselines on continuous control tasks.
New methods estimate policy value and gradients for deterministic policies from off-policy data.
problem Estimating policy value and gradients for deterministic policies from off-policy data.
method Proposed new doubly robust estimators based on kernelization approaches.
result Demonstrated a rate independent of horizon length for policy value and gradient estimation.
Gradient flows of neural networks converge to optimal values or diverge, with thresholds and asymptotic behaviors.
problem Understanding the convergence and divergence of gradient flows in neural networks.
method Analysis of gradient flows on loss landscapes of neural networks using o-minimal structures.
result Gradient flows either converge to optimal values or diverge to infinity, with thresholds and asymptotic behaviors.
Paper presents a new policy gradient theorem using weak derivatives for reinforcement learning.
problem Continuous state-action reinforcement learning problems.
method Introduced an alternative policy gradient theorem using weak derivatives.
result The new approach yields algorithms that converge almost surely to stationary points of the value function.
This work establishes properties on diffeological structures for set-valued maps and measures.
problem Establish rigorous properties on diffeological structures for set-valued maps and measures.
method Using diffeologies, the authors link various structures including set-valued maps, relations, gradients, measures, and shape analysis.
result Established rigorous properties on sample diffeologies.
New method QMLE performs well in complex action spaces without policy gradients.
problem Why policy gradients outperform action-value methods in complex action spaces.
method QMLE framework for action-value methods based on three principles.
result QMLE performs comparably to policy gradient methods in complex action spaces.
The paper proposes methods to find a shared active subspace for multivariate vector-valued functions.
problem Minimizing the deviation between function evaluations in the original and reconstructed spaces.
method Manipulating gradients or SPD matrices to identify a shared structure.
result Summing SPD matrices often identifies the best shared active subspace.
Paper proves gradient estimates for Lagrangian mean curvature equation.
problem Proving gradient estimates for Lagrangian mean curvature equation.
method Interior gradient estimates for critical and supercritical Lagrangian mean curvature equation.
result Solves Dirichlet boundary value problem for critical and supercritical Lagrangian mean curvature equation.
The paper establishes inequalities and gradient estimates for harmonic functions on Finsler measure spaces.
problem Functional and geometric inequalities on Finsler measure spaces.
method Local uniform Poincaré and Sobolev inequalities, mean value inequality, Harnack inequalities, and gradient estimates.
result Global gradient estimates for positive harmonic functions on Finsler measure spaces.
PPG separates policy and value function training phases for better reinforcement learning efficiency.
problem Challenges in traditional reinforcement learning methods for policy and value function optimization.
method Integrates Phasic Policy Gradient framework that splits policy and value function training into distinct phases.
result Significantly improves sample efficiency on Procgen Benchmark compared to PPO.
Stochastic gradient descent regularizes least squares problems by smoothing large singular values.
problem Regularization of least squares problems using stochastic gradient descent.
method Analysis of stochastic gradient descent applied to least squares problems, showing a regularization effect.
result Stochastic gradient descent leads to a quick regularization effect, smoothing large singular values.
New convergence rates for shuffling gradient methods without strong convexity.
problem Theoretical gap between shuffling gradient methods' empirical success and established convergence rates.
method Proved last-iterate convergence rates for shuffling gradient methods using function value gap.
result First last-iterate convergence rates for shuffling gradient methods without strong convexity.
A new Shapley value approach for neural networks interpretable and stable.
problem Neural networks' interpretability and training stability issues.
method Shapley value approximation for ReLU activation, globally continuous Shapley gradient, Shapley Activation function.
result SA consistently outperforms ReLU in training convergence, accuracy, and stability.
This paper proposes GProp, a deep reinforcement learning algorithm for continuous policies with compatible function approximation. The algorithm is based on two innovations. Firstly, we present a temporal-difference based method for learning the gradient of the value-function. Secondly, we present the deviator-actor-cr…
New RL algorithm tackles complex discrete action spaces.
problem Challenges in applying on-policy RL in high-dimensional discrete action spaces.
method Action-value critic, correlated actions, gradient sparsification.
result Empirically outperforms related on-policy algorithms.
Gravilon improves gradient descent for neural networks.
problem Improving efficiency and accuracy of gradient descent methods.
method Uses geometric modification of gradient step lengths.
result Promising experimental results on MNIST classification.
Sobolev training helps neural nets fit function values and derivatives.
problem Training neural nets to match function values and derivatives accurately.
method Using Sobolev loss with gradient flow for overparameterized networks.
result Gradient flow from random initialization can fit any function and its derivatives.
Policy gradient methods with aggregated states can achieve better performance than approximate policy iteration.
problem Approximation errors in policy and value function approximations.
method State-aggregated representations and policy gradient methods.
result Policy gradient methods can achieve a per-period regret bounded by ε, while approximate policy iteration and value iteration have a higher regret.
Characterizes values at infinity for real polynomial maps with 2D fibers.
problem Understanding atypical values at infinity for real polynomial maps.
method Characterization using indices of gradient vector fields on spheres.
result Analogous to two-variable case, but for maps with 2D fibers.
Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting. However, vanilla online variants are on-policy only and not able to take advantage of off-policy data. In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experie…
DG improves policy gradient efficiency by selectively backpropagating only valuable samples.
problem Expensive backward passes in policy gradient methods reduce efficiency.
method Introduces 'delight' as a forward-pass signal of learning value and a Kondo gate to selectively backpropagate.
result Selective backpropagation reduces backward pass costs without sacrificing learning quality.
The paper tackles catastrophic risk in reinforcement learning using extreme value theory.
problem Mitigating catastrophic risk in sequential decision making with limited observations.
method Developed POTPG, a policy gradient algorithm based on extreme value theory.
result POTPG outperforms common benchmarks in numerical experiments.
This work analyzes centered binary Restricted Boltzmann Machines (RBMs) and binary Deep Boltzmann Machines (DBMs), where centering is done by subtracting offset values from visible and hidden variables. We show analytically that (i) centering results in a different but equivalent parameterization for artificial neural …
In reinforcement learning, temporal difference (TD) is the most direct algorithm to learn the value function of a policy. For large or infinite state spaces, exact representations of the value function are usually not available, and it must be approximated by a function in some parametric family. However, with \emph{no…
Paper approximates risk measures using SGD with Langevin dynamics.
problem Approximating arbitrary law invariant risk measures.
method Stochastic Gradient Langevin Dynamics (SGD-Langevin) for general risk measures.
result Non-asymptotic convergence rates of the approximation algorithm.
Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any p…
A new multi-agent learning method improves performance in complex games.
problem Performance gap between MAPG and value-based multi-agent approaches.
method Introduces value function decomposition into multi-agent actor-critic framework for off-policy learning.
result DOP significantly outperforms state-of-the-art multi-agent reinforcement learning algorithms.
Gradient boosting adapted for vector inputs.
problem Applying gradient boosting to multi-class classification problems.
method Extended gradient boosting framework to vector inputs using histogram-based decision trees.
result Efficient algorithm for vector-valued objectives.
This paper improves convergence guarantees for gradient clipping in deep learning.
problem Improving convergence guarantees for gradient clipping in deep learning models.
method Analyzes and provides precise convergence guarantees for arbitrary clipping thresholds.
result Shows tight convergence guarantees for clipped stochastic gradient descent.
Trust region and cubic regularization methods have demonstrated good performance in small scale non-convex optimization, showing the ability to escape from saddle points. Each iteration of these methods involves computation of gradient, Hessian and function value in order to obtain the search direction and adjust the r…
We derive a priori interior Hessian and gradient estimates for special Lagrangian equation of phase at least a critical value in dimension three.
Policy gradient method proves convergence in imperfect-information games.
problem Policy gradient methods in imperfect-information games (EFGs).
method Policy gradient approach with best-iterate convergence.
result Policy gradient leads to provable best-iterate convergence in self-play EFGs.
Study policy gradient and actor-critic methods for continuous-time reinforcement learning.
problem Continuous-time reinforcement learning with policy gradient and actor-critic approaches.
method Regularized exploratory formulation, martingale approach, simultaneous policy and value function updates.
result Proposed two types of actor-critic algorithms for online and offline learning.
Researchers derive a new equation for valuing American options.
problem Valuation and hedging of American options on dividend-paying assets.
method Derive a stochastic balance equation for the value function and its gradient.
result The derived equation uniquely solves the valuation problem.
Bayesian network approach for efficient cooperative MARL.
problem Leveraging inter-agent coupling information for scalable MARL algorithms.
method Modeling cooperative MARL via Bayesian networks, identifying value dependency sets, proposing P-DTDE paradigm.
result P-DTDE policy gradient estimator has lower total variance than CTDE.
Kernel smoothing improves LLM reasoning efficiency.
problem Efficiently estimate value functions with limited samples for reinforcement learning.
method Kernelized advantage estimation using classical nonparametric statistics.
result Improved policy optimization with accurate value and gradient estimation.
New analysis shows how cross-entropy training shapes attention in transformers.
problem Understanding how gradient-based learning creates the required internal geometry in transformers.
method Developed a first-order analysis of cross-entropy training effects on attention scores and values in a transformer attention head.
result Introduced an advantage-based routing law and responsibility-weighted update for attention scores and values, respectively.
Muon replaces matrix gradient with polar factor, optimizing flat spectrum updates
problem Optimization bias in matrix updates
method Using polar factor of gradient
result Muon update maximizes entropy among bounded updates
New method estimates minimizer and minimum value of a regression function.
problem Estimating minimizer and minimum value of a regression function from noisy data.
method Projected gradient descent with gradient estimated by regularized local polynomial algorithm, followed by a rate optimal nonparametric procedure.
result Achieves minimax optimal rates of convergence for smooth and strongly convex functions.
Developed policy gradient methods for stochastic control with exit time, outperforming traditional techniques in share repurchase pricing.
problem Optimal control with exit time in stochastic models.
method Two types of algorithms: direct policy learning and alternately learning value function and control.
result Policy gradient methods outperform PDE or neural networks in share repurchase pricing.