Faster policy learning via continuous-time gradients.
problem Efficiently estimating policy gradients for continuous-time systems.
method Approximating continuous-time gradients directly, using adaptive discretization.
result More efficient policy gradient estimator leads to faster learning.
Develops DPG methods for continuous-time RL with deterministic policies.
problem High variance and slow convergence in stochastic policy RL methods.
method Derives continuous-time policy gradient formula and proposes CT-DDPG algorithm.
result CT-DDPG achieves superior stability and faster convergence in continuous-time RL.
Naive investors make riskier choices than optimal strategies in continuous-time finance.
problem Continuous-time Markowitz portfolio selection with naive reoptimization.
method Analytical derivation of naive policies from discretely naive policies.
result Naive policies are always riskier and less efficient than equilibrium policies.
Minimal assumptions analysis of Q-learning with time-varying policies.
problem Finite-time analysis of Q-learning with time-varying policies for discounted MDPs.
method Minimal assumptions, Poisson equation decomposition, sensitivity analysis.
result Established convergence rate and sample complexity for Q-learning.
Policy distillation in deep reinforcement learning provides an effective way to transfer control policies from a larger network to a smaller untrained network without a significant degradation in performance. However, policy distillation is underexplored in deep reinforcement learning, and existing approaches are compu…
Develops a reinforcement learning algorithm for learning deterministic equilibrium policies in time-inconsistent control problems.
problem Learning equilibrium policies in time-inconsistent control problems.
method Continuous-time model-free reinforcement learning algorithm using deterministic policy gradient approach.
result Learned equilibrium policies in general time-inconsistent control problems.
Develops an anytime-valid framework for optimal policy identification from logged contextual bandit data.
problem Selecting the optimal policy from a candidate policy class while monitoring evidence continuously.
method Constructs a time-indexed set that retains the true optimal policy set uniformly over time.
result The procedure allows the analyst to monitor policy values, eliminate clearly suboptimal policies, and stop at data-dependent times without invalidating inference.
Developed policy gradient methods for stochastic control with exit time, outperforming traditional techniques in share repurchase pricing.
problem Optimal control with exit time in stochastic models.
method Two types of algorithms: direct policy learning and alternately learning value function and control.
result Policy gradient methods outperform PDE or neural networks in share repurchase pricing.
Test-time data augmentation−averaging the predictions of a machine learning model across multiple augmented samples of data−is a widely used technique that improves the predictive performance. While many advanced learnable data augmentation techniques have emerged in recent years, they are focused on the training p…
Efficient deep policy gradient method for continuous-time control problems.
problem Optimal control in continuous time with fine time discretization.
method Multi-scale deep policy gradient method with varying time discretization.
result Targeted efficiency in computational resources achieved through multi-scale approach.
Paper analyzes convergence of dynamic policy gradient for MDPs, improving performance in finite-time problems.
problem Optimal policies in finite-time MDPs are not stationary and require epoch-specific training.
method Introduces dynamic policy gradient combining dynamic programming and policy gradient, analyzes convergence for softmax parametrisation.
result Dynamic policy gradient training exploits finite-time structure, leading to better convergence bounds.
Memory-efficient algorithm reduces variance in off-policy RL.
problem High variance in off-policy policy optimization.
method Memory-efficient, stochastically variance-reduced algorithm using off-policy samples.
result Empirically validated effectiveness of the proposed algorithm.
Study shows certainty equivalent policy minimizes regret in continuous-time systems.
problem Minimizing regret in continuous-time stochastic linear-quadratic systems.
method Theoretical analysis of randomized certainty equivalent policy.
result Establishes square-root of time regret bounds and linear scaling with parameters.
The paper provides a non-asymptotic error bound for linear system identification under nonlinear policies.
problem System identification for linear systems with nonlinear and/or time-varying policies under i.i.d. random excitation noises.
method Least square estimation with non-asymptotic error bound for bounded state and action trajectories.
result The error bound is consistent with linear policies and generalizes existing guarantees.
Extends double linear policy with time-varying weights and proves robust positive expectation.
problem Ensuring robustness in policy optimization with time-varying parameters.
method Employed a novel elementary symmetric polynomials characterization approach to prove robust positive expectation (RPE). Derived explicit expressions for expected cumulative gain-loss and variance.
result Proved the robust positive expectation property holds for the extended double linear policy.
Study policy gradient and actor-critic methods for continuous-time reinforcement learning.
problem Continuous-time reinforcement learning with policy gradient and actor-critic approaches.
method Regularized exploratory formulation, martingale approach, simultaneous policy and value function updates.
result Proposed two types of actor-critic algorithms for online and offline learning.
New algorithm reduces RL policy optimization gap.
problem Insufficient theoretical understanding of policy optimization methods.
method Reference-based Policy Optimization with Stable at Any Time guarantee (RPO-SAT)
result Achieves nearly minimax optimal policy-based algorithm for tabular RL.
Extends RL to random stopping times, improving optimization.
problem Real-world applications with random stopping times.
method Extended RL framework to random stopping times, derived new formulas.
result Improves optimization convergence with new formulas.
Early stopping method saves up to 75% computation time in policy search tasks.
problem Lengthy evaluation times in optimization problems, especially in robotics.
method A generalized early stopping criterion that only uses objective value at each time step.
result The method saves up to 75% computation time compared to no stopping.
RL solves discrete LQ control with Gaussian optimal policy.
problem Discrete-time linear-quadratic control problem.
method Entropy-based RL to find Gaussian optimal policy.
result RL algorithm solves mean-variance asset-liability management problem.
Considering a lead-time-and price-sensitive demand, we investigate whether a client rejection policy, modeled as M/M/1/K system, can be more profitable than an all-client acceptance policy, modeled as M/M/1 system. We provide analytical insights for the cases with and without holding and penalty costs by comparing M/M/…
Optimizes state monitoring in Markovian systems with cost constraints.
problem Balancing state queries with prediction costs in Markovian systems.
method Greedy policy and SGD-based learning variant for optimal predict-query tradeoff.
result Greedy policy is suboptimal but performs close to optimal under certain conditions.
Step-DAD improves BED by periodically updating a design policy during experiments.
problem Improving flexibility and robustness in Bayesian experimental design.
method Semi-amortized, policy-based approach that updates a design policy during data collection.
result Consistently superior decision-making and robustness compared to current BED methods.
Study policy gradient for large-agent mean-field control and game in continuous time.
problem Optimal policy learning for large number of agents in continuous-time mean-field systems.
method Policy gradient method applied to linear-quadratic mean-field control and game models.
result Policy gradient converges to optimal solution at a linear rate for both mean-field control and game.
New policy optimizes risk and optimality in stochastic bandits.
problem Optimizing risk in stochastic bandits with heavy-tailed risk.
method Designing policies with worst-case optimality for expected regret and light-tailed risk distribution.
result Achieves worst-case optimality for expected regret and light-tailed risk distribution.
New framework for policy gradient methods in continuous time reinforcement learning.
problem Addressing policy gradient methods for continuous time reinforcement learning.
method Control randomisation technique to derive policy gradient representation for various Markovian control problems.
result Demonstrated application to optimal switching problems in the energy sector.
New methods for anytime-valid off-policy inference in contextual bandits.
problem Estimating properties of hypothetical policies in adaptive experiments.
method Modern martingale techniques for comprehensive OPE inference.
result Valid anytime inference for off-policy mean reward values and entire reward distributions.
Second-order estimator improves continuous-time policy evaluation.
problem Estimating value surfaces from discrete data with time-inhomogeneous dynamics.
method Moment-matching coefficients for high-order generator regression.
result Second-order estimator consistently outperforms Bellman baseline.
New method learns adaptive exploration strategies for dynamic tasks.
problem Learning effective exploration strategies in changing environments.
method Informed policy regularization to reduce sample complexity of RNN-based policies.
result Method learns efficient exploration strategies balancing information gathering and reward maximization.
SDM Policy accelerates inference for robotic tasks while maintaining high action quality.
problem Prolonged inference times in diffusion-based policies for high-frequency control tasks.
method Two-stage optimization: score matching and distribution matching; dual-teacher mechanism.
result 6x inference speedup with state-of-the-art action quality.
Due to the recent advancements in wearables and sensing technology, health scientists are increasingly developing mobile health (mHealth) interventions. In mHealth interventions, mobile devices are used to deliver treatment to individuals as they go about their daily lives. These treatments are generally designed to im…
Paper addresses offline policy evaluation in RL, achieving near-optimal bounds for various policy classes.
problem Evaluate all policies in a class simultaneously for offline RL.
method Uniform convergence in OPE for various policy classes, achieving optimal episode complexity.
result Achieves optimal episode complexity of O(H^3/d_mε^2) for identifying ε-optimal policies.
Develops RL for dynamic risk assessment in stochastic optimization.
problem Time-consistent risk assessment in stochastic optimization problems.
method Model-free reinforcement learning with dynamic convex risk measures, time-consistent dynamic programming, policy gradient updates, actor-critic neural network optimization.
result Demonstrates optimal policies for statistical arbitrage, financial hedging, and robot control.
For training the sequence-to-sequence voice conversion model, we need to handle an issue of insufficient data about the number of speech pairs which consist of the same utterance. This study experimentally investigated the effects of Mel-spectrogram augmentation on training the sequence-to-sequence voice conversion (VC…
In multi-task reinforcement learning there are two main challenges: at training time, the ability to learn different policies with a single model; at test time, inferring which of those policies applying without an external signal. In the case of continual reinforcement learning a third challenge arises: learning tasks…
Algorithm safely learns from sub-optimal baseline policies while satisfying constraints.
problem Safe reinforcement learning with constraints when baseline policy is sub-optimal.
method Iterative policy optimization alternating between return maximization, baseline distance minimization, and constraint projection.
result Consistently outperforms baselines, achieving 10x fewer constraint violations and 40% higher reward.
Simulation framework assesses ROI of chronic disease adherence and policy timing.
problem Uncertainty in ROI of adherence-enhancing interventions under heterogeneous patient behavior and socioeconomic variation.
method Simulation-based framework integrating disease progression, time-varying adherence, and policy timing.
result Early and adaptive interventions yield highest ROI, exceeding 20% under certain conditions.
We consider assortment optimization over a continuous spectrum of products represented by the unit interval, where the seller's problem consists of determining the optimal subset of products to offer to potential customers. To describe the relation between assortment and customer choice, we propose a probabilistic choi…
Paper formulates mutual information optimal control for discrete-time systems.
problem Optimal control of discrete-time linear systems with mutual information.
method Formulates MIOCP as an extension of MEOCP, derives optimal policy and prior, proposes alternating minimization algorithm.
result Proposes an alternating minimization algorithm for MIOCP.
We address the problem of finding an optimal policy in a Markov decision process under a restricted policy class defined by the convex hull of a set of base policies. This problem is of great interest in applications in which a number of reasonably good (or safe) policies are already known and we are only interested in…
Opioid overdose rates have reached an epidemic level and state-level policy innovations have followed suit in an effort to prevent overdose deaths. State-level drug law is a set of policies that may reinforce or undermine each other, and analysts have a limited set of tools for handling the policy collinearity using st…
The paper examines how macroeconomic control tools lost effectiveness, leading to a 'dark ages' period.
problem Loss of effectiveness of control tools in macroeconomic stabilization policy.
method Historical analysis of macroeconomic stabilization policy from 1948 to 1993.
result The overstatement of the Lucas critique and Kydland and Prescott's time-inconsistency led to a period of ineffective stabilization policy.
This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves a target policy while following a distinct behavior policy. To this end, the paper develops a multi-…
Analyzes how economic policies affect wealth distribution in Bitcoin token economy.
problem Impact of economic policies on wealth distribution in token economies.
method Eliminated noise in wealth distribution data using macroeconomic and microeconomic time series. Causality analysis between BIPs and wealth distribution data.
result Proposed a structure for economic policy taxonomy in token economies.
Study online control of unknown time-varying systems with negative and positive results.
problem Online control of time-varying systems with unknown dynamics.
method Algorithmic upper bounds and lower bounds for different policy classes.
result Sublinear adaptive regret bounds for Disturbance Response policies.
We are witnessing an increasing use of data-driven predictive models to inform decisions. As decisions have implications for individuals and society, there is increasing pressure on decision makers to be transparent about their decision policies. At the same time, individuals may use knowledge, gained by transparency, …
Continuous-time Q-learning theory developed for reinforcement learning.
problem Continuous-time reinforcement learning challenges.
method Entropy-regularized, exploratory diffusion process formulation; first-order approximation of Q-function; martingale conditions.
result Developed a q-learning theory independent of time discretization.
LHIEM model predicts health, income, and employment over years.
problem Lack of path dependency in health policy simulations.
method Discrete-time microsimulation with Markov chain modules.
result Validates health care financing proposal through detailed modeling.