This paper investigates methods for estimating the optimal stochastic control policy for a Markov Decision Process with unknown transition dynamics and an unknown reward function. This form of model-free reinforcement learning comprises many real world systems such as playing video games, simulated control tasks, and r…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “stochastic policy optimization”
Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…
Designs a single policy for collecting data to train near-optimal policies.
problem Engineering overhead in deploying minimax procedures for stochastic linear contextual bandits.
method Designs a single stochastic policy to collect data from which a near-optimal policy can be extracted.
result The designed policy can collect data from which a near-optimal policy can be extracted.
A Simple and Optimal Policy Design with Safety against Heavy-Tailed Risk for Stochastic Banditsstat.ML
New policy optimizes risk and optimality in stochastic bandits.
problem Optimizing risk in stochastic bandits with heavy-tailed risk.
method Designing policies with worst-case optimality for expected regret and light-tailed risk distribution.
result Achieves worst-case optimality for expected regret and light-tailed risk distribution.
DNN policies improve stochastic AC OPF for power grid optimization.
problem Optimizing power grid operations under uncertainty.
method Deep neural network (DNN) policies for real-time generator dispatch decisions.
result DNN policies enforce feasibility constraints and produce near optimal solutions.
PS framework selects best policy from library for CSO problems.
problem Policy selection in CSO with heterogeneous performance across covariate space.
method PS framework constructs library of candidate policies and learns a meta-policy to select the best one.
result PS consistently outperforms best single policy in heterogeneous CSO problems.
This paper augments the reward received by a reinforcement learning agent with potential functions in order to help the agent learn (possibly stochastic) optimal policies. We show that a potential-based reward shaping scheme is able to preserve optimality of stochastic policies, and demonstrate that the ability of an a…
Develops RL for dynamic risk assessment in stochastic optimization.
problem Time-consistent risk assessment in stochastic optimization problems.
method Model-free reinforcement learning with dynamic convex risk measures, time-consistent dynamic programming, policy gradient updates, actor-critic neural network optimization.
result Demonstrates optimal policies for statistical arbitrage, financial hedging, and robot control.
Regret Distribution in Stochastic Bandits: Optimal Trade-off between Expectation and Tail Riskstat.ML
Optimizes regret distribution in stochastic bandits for risk balance.
problem Balancing regret expectation and tail risk in stochastic bandits.
method Characterizes optimal regret tail probability for any threshold, proposes new policies.
result Discovers an intrinsic gap in optimal tail rate based on time horizon uncertainty.
We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.
problem Some policy optimization algorithms do not have batch size-invariance, leading to inefficiencies.
method We decouple the proximal policy from the behavior policy to achieve batch size-invariance.
result Our approach makes policy optimization algorithms more efficient and allows them to use stale data more effectively.
A classic setting of the stochastic K-armed bandit problem is considered in this note. In this problem it has been known that KL-UCB policy achieves the asymptotically optimal regret bound and KL-UCB+ policy empirically performs better than the KL-UCB policy although the regret bound for the original form of the KL-UCB…
Memory-efficient algorithm reduces variance in off-policy RL.
problem High variance in off-policy policy optimization.
method Memory-efficient, stochastically variance-reduced algorithm using off-policy samples.
result Empirically validated effectiveness of the proposed algorithm.
The paper analyzes the intrinsic exploration terms in policy-gradient algorithms.
problem Exploration in policy-gradient algorithms and its impact on policy optimization.
method Numerical optimization criteria and stochastic gradient analysis.
result Exploration techniques improve policy optimization by smoothing the learning objective and modifying gradient estimates.
New approach to portfolio optimization shows entropy regularization is ineffective.
problem Entropy regularization in mean-variance portfolio optimization under drift uncertainty.
method Combining Bayesian filtering and stochastic policy optimization.
result Entropy regularization does not accelerate learning about unknown drift.
Developed policy gradient methods for stochastic control with exit time, outperforming traditional techniques in share repurchase pricing.
problem Optimal control with exit time in stochastic models.
method Two types of algorithms: direct policy learning and alternately learning value function and control.
result Policy gradient methods outperform PDE or neural networks in share repurchase pricing.
Optimizes control of noisy discrete systems without system matrix knowledge.
problem Optimal control of discrete-time systems with additive and multiplicative noises.
method Stochastic Lyapunov and Riccati equations, model-free reinforcement learning.
result Model-free reinforcement learning algorithm converges to optimal control policy.
Dynamic Weight Optimization for Double Linear Policy: A Stochastic Model Predictive Control Approacheess.SY
A new approach optimizes weights in DLP for better risk-adjusted performance.
problem Optimizing time-varying weights in Double Linear Policy (DLP) for better risk-adjusted performance.
method Stochastic Model Predictive Control (SMPC) framework to maximize risk-adjusted returns while enforcing constraints.
result Empirical results show improved risk-adjusted performance and drawdown control.
Improving sample efficiency has been a longstanding goal in reinforcement learning. This paper proposes algorithm: a sample efficient policy gradient method with stochastic mirror descent. In , a novel variance-reduced policy gradient estimator is presented to improve sample efficiency.…
Develops CLTs for Markov chain transition probabilities and policies.
problem Estimating transition probabilities and policies in controlled Markov chains.
method Non-parametric estimator for transition matrices; CLTs for value, Q-, and advantage functions; goodness-of-fit tests.
result Asymptotic normality of estimators under specific logging policies.
Develops first-order methods for average-reward MDPs with strong guarantees.
problem Lack of strong theoretical guarantees for first-order methods in AMDPs.
method Average-reward stochastic policy mirror descent (SPMD) and variance-reduced temporal difference (VRTD) methods.
result Establishes sample complexity results for solving AMDPs.
Reinforcement learning (RL) methods often rely on massive exploration data to search optimal policies, and suffer from poor sampling efficiency. This paper presents a mixed reinforcement learning (mixed RL) algorithm by simultaneously using dual representations of environmental dynamics to search the optimal policy wit…
Study uses SGD to find near-optimal execution cost policies in dynamic markets.
problem Finding optimal execution cost policies in complex markets.
method Stochastic Gradient Descent (SGD) approach to derive near-optimal policies.
result SGD-based policies offer valuable insights and are implementable in volatile markets.
Unified approach to Merton's portfolio problem using Pontryagin's principles.
problem Optimizing consumption and investment strategies in financial portfolios.
method PG-DPO framework combining neural networks with Pontryagin's maximum principle.
result Locally optimal policies closely tied to classical stochastic control.
Optimizes dividend policies in a Brownian model with controlled rates.
problem Realistic optimal dividend policies in a stochastic control problem.
method Delayed linear control strategies for refracted diffusion processes.
result Optimality of delayed linear control strategies for dividend payments.
Recent advances in policy gradient methods and deep learning have demonstrated their applicability for complex reinforcement learning problems. However, the variance of the performance gradient estimates obtained from the simulation is often excessive, leading to poor sample efficiency. In this paper, we apply the stoc…
Study shows sample complexity for learning optimal policies in SSP with generative model.
problem Learning optimal policies in Stochastic Shortest Path problems.
method Derive and prove lower and upper bounds on sample complexity.
result Lower bound of samples for general case, and up to logarithmic factors for bounded hitting time condition.
Study shows hard sample complexity for learning optimal policies in stochastic shortest path problems.
problem Learning optimal policies in stochastic shortest path problems.
method Analyzes sample complexity with and without generative models, derives lower and upper bounds.
result Proves sample complexity bounds and impossibility of horizon-free regret in SSPs.
Optimizes insurance pricing by accounting for policyholders' price sensitivity.
problem Traditional insurance pricing does not consider policyholders' price sensitivity.
method Formulates insurance pricing as a decision-making problem and uses off-policy evaluation and stochastic control.
result Neural networks outperform existing techniques for policy optimization.
Bayesian Markowitz portfolio problem shows entropy regularization is ineffective.
problem Entropy regularization in Bayesian Markowitz portfolio optimization.
method Combines continuous-time Bayesian filtering with stochastic policy optimization.
result Entropy regularization does not accelerate learning of unknown drift.
New framework for policy gradient methods in continuous time reinforcement learning.
problem Addressing policy gradient methods for continuous time reinforcement learning.
method Control randomisation technique to derive policy gradient representation for various Markovian control problems.
result Demonstrated application to optimal switching problems in the energy sector.
This paper optimizes sampling policies for Bayesian optimization to improve exploration and exploitation.
problem Improving the balance between exploration and exploitation in Bayesian optimization.
method Developed efficient methods to estimate and optimize non-myopic acquisition functions using rollout policies and stochastic gradient optimization.
result Efficient optimization of sampling policies leads to better performance in Bayesian optimization.
Study shows how to learn optimal policies quickly in stochastic control problems.
problem Learning optimal policies in large, continuous state and action spaces with limited data.
method Analyzes three geometric exponents to quantify fast policy regret convergence.
result Shows that fast policy regret convergence is induced by specific geometric structures.
Paper uses RL to optimize bid-ask spreads for diverse options.
problem Optimizing bid-ask spreads for options with various maturities and strikes.
method Combines stochastic policy with reinforcement learning.
result Proposes an effective approach for market making of options.
Under a Bayesian framework, we formulate the fully sequential sampling and selection decision in statistical ranking and selection as a stochastic control problem, and derive the associated Bellman equation. Using value function approximation, we derive an approximately optimal allocation policy. We show that this poli…
Paper proposes a reinforcement learning framework for efficient hyper-parameter tuning of stochastic optimization algorithms.
problem Efficient tuning of hyper-parameters for stochastic optimization algorithms.
method Modeling hyper-parameter tuning as a Markov decision process and using policy gradient algorithms.
result The proposed framework significantly reduces the time required for hyper-parameter tuning compared to Bayesian optimization.
Policy gradient methods are powerful reinforcement learning algorithms and have been demonstrated to solve many complex tasks. However, these methods are also data-inefficient, afflicted with high variance gradient estimates, and frequently get stuck in local optima. This work addresses these weaknesses by combining re…
The paper interprets policy-gradient algorithms using continuation theory.
problem Optimizing nonconvex functions in reinforcement learning.
method Formulates policy optimization as optimization by continuation, interprets policy-gradient algorithms as implicitly optimizing deterministic policies.
result Exploration in policy-gradient algorithms is seen as computing a continuation of the return of the policy.
ESOP uses Bayesian optimization to find optimal lock-down schedules.
problem Finding optimal lock-down schedules balancing health and economy.
method Bayesian optimization interacting with epidemiological models.
result ESOP schedules balance public health and economic impacts.
Policy gradient methods converge for LQR problems with noisy state dynamics.
problem Finding optimal policies in noisy LQR problems over finite time horizons.
method Policy gradient methods with convergence guarantees for finite time and stochastic state dynamics.
result Global linear convergence for policy gradient methods in LQR problems with weak assumptions.
This paper first describes a class of uncertain stochastic control systems with Markovian switching, and derives an Itô-Liu formula for Markov-modulated processes. And we characterize an optimal control law, which satisfies the generalized Hamilton-Jacobi-Bellman (HJB) equation with Markovian switching. Then, by using …
Paper tackles offline SSP with value iteration for policy evaluation and learning.
problem Goal-oriented RL with offline data and cost minimization.
method Simple value iteration algorithms for OPE and offline policy learning.
result Strong instance-dependent bounds implying near-minimax optimal worst-case bounds.
Improves policy optimization with polylog(T) regret bounds for stochastic losses.
problem Improves theoretical guarantees for policy optimization in stochastic settings.
method Leverages Tsallis and Shannon entropy regularizers for polylog(T) regret, and log-barrier regularizer for adversarial settings.
result Achieves a first-order polylog(T) regret bound for policy optimization in stochastic settings.
A method for accurate pricing of multidimensional derivatives under uncertain volatility.
problem High-dimensional stochastic control problem in uncertain volatility model.
method Backward actor-critic stochastic policy gradient scheme combining DP, PPO, and neural networks.
result Accurate and efficient pricing of multidimensional derivatives compared to benchmarks.
A new ML algorithm solves complex economic control problems.
problem Solving high-dimensional, finite-horizon stochastic control problems in economics.
method Deep neural network representation of optimal policy functions with three key features.
result Efficiently solves various economic control problems including recursive utility and growth models.
Deriving and applying Proximal Policy Optimization to GFlowNets for efficient training of discrete sampling policies
problem Training stochastic policies to sample from structured discrete probability distributions
method Deriving policy gradient algorithms for GFlowNets and applying Proximal Policy Optimization
result Improved convergence speed and data efficiency compared to standard GFlowNet training objectives
Entropy regularization is commonly used to improve policy optimization in reinforcement learning. It is believed to help with \emph{exploration} by encouraging the selection of more stochastic policies. In this work, we analyze this claim using new visualizations of the optimization landscape based on randomly perturbi…
Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policycs.LG
This work tackles risk-sensitive deep RL by optimizing policies with variance constraints.
problem Risk and aleatoric uncertainty in deep reinforcement learning.
method Lagrangian and Fenchel dualities to transform the problem into an unconstrained saddle-point policy optimization problem, and an actor-critic algorithm to iteratively update policy, Lagrange multiplier, and Fenchel dual variable.
result The proposed actor-critic algorithm finds a globally optimal policy at a sublinear rate.
The paper tackles counterfactual learning for stochastic policies with continuous actions.
problem Learning stochastic policies with continuous actions from logged data.
method Introduces a joint kernel embedding of contexts and actions to model continuous actions, and uses proximal point algorithms and smooth estimators for optimization.
result Demonstrates the benefits of using proximal point algorithms and smooth estimators for counterfactual learning.