VPP learns joint policies for multi-agent RL through interactions.
problem Learning effective joint policies for multi-agent reinforcement learning.
method VPP integrates variational inference into policy layers for efficient sampling and differentiability.
result VPP outperforms previous methods on large-scale multi-agent tasks.
This work learns exploration policies for unknown distributions using samples and policy gradients.
problem Learning exploration policies for unknown distributions in Bayesian bandits.
method Meta-learning approach parameterizing policies in a differentiable way and optimizing them with policy gradients.
result Effective gradient estimators and variance reduction techniques are derived.
We present the first differentially private algorithms for reinforcement learning, which apply to the task of evaluating a fixed policy. We establish two approaches for achieving differential privacy, provide a theoretical analysis of the privacy and utility of the two algorithms, and show promising results on simple e…
Short note on soft-max and policy gradients in bandit problems using Lyapunov functions.
problem Analyzing soft-max and policy gradient methods in bandit problems.
method Lyapunov function argument for soft-max and differential equations for policy gradient algorithms.
result Regret bounds for soft-max and a different policy gradient algorithm in bandit problems.
This paper introduces a new method to evaluate multiple policies simultaneously.
problem Estimating the value of many policies for a single set of states.
method Developed a scalable, differentiable fingerprinting mechanism to represent complex policies.
result The method can produce policies that outperform those that generated the training data, in zero-shot manner.
SNAPO optimizes policies for complex sequential decisions using differentiable simulation.
problem Optimizing policies for high-dimensional, sequential decisions under uncertainty.
method Embeds neural policy in a differentiable simulator, computes gradients efficiently.
result Produces sensitivities at a cost proportional to one reverse pass, regardless of sensitivity count.
New method fine-tunes discrete diffusion models for RLHF tasks.
problem Fine-tuning discrete diffusion models with policy gradient methods is challenging.
method Proposed SEPO algorithm for efficient fine-tuning over non-differentiable rewards.
result Numerical experiments show scalability and efficiency of SEPO.
VaR-CPO optimizes VaR-constrained RL problems with conservative policy updates.
problem Optimizing VaR-constrained reinforcement learning problems.
method Combines Cantelli's inequality and trust-region framework for efficient and conservative optimization.
result Achieves zero constraint violations during training in feasible environments.
DiPS learns to optimize sketching policies for better recommendation quality.
problem Optimizing sketching policies for long-term user interest prediction in recommender systems.
method Differentiable policy for sketching that learns from training data.
result DiPS requires up to 50% fewer sketch items to achieve the same recommendation quality.
Study shows certainty equivalent policy minimizes regret in continuous-time systems.
problem Minimizing regret in continuous-time stochastic linear-quadratic systems.
method Theoretical analysis of randomized certainty equivalent policy.
result Establishes square-root of time regret bounds and linear scaling with parameters.
Most reinforcement learning algorithms are inefficient for learning multiple tasks in complex robotic systems, where different tasks share a set of actions. In such environments a compound policy may be learnt with shared neural network parameters, which performs multiple tasks concurrently. However such compound polic…
PriMORL trains private RL policies on offline data.
problem Private reinforcement learning on offline data.
method PriMORL learns DP models of the environment and optimizes a policy on the penalized private model.
result PriMORL enables training of private RL agents on complex tasks.
We introduce the value iteration network (VIN): a fully differentiable neural network with a `planning module' embedded within. VINs can learn to plan, and are suitable for predicting outcomes that involve planning-based reasoning, such as policies for reinforcement learning. Key to our approach is a novel differentiab…
Neural Index Policy for multi-action bandits with heterogeneous budgets.
problem Real-world settings often involve multiple interventions with heterogeneous costs and constraints, breaking classical assumptions.
method Introduces a Neural Index Policy (NIP) that learns to assign budget-aware indices to arm-action pairs using a neural network and differentiable knapsack layer.
result Empirically achieves near-optimal performance while strictly enforcing heterogeneous budgets and scaling to hundreds of arms.
We present foundations for using Model Predictive Control (MPC) as a differentiable policy class for reinforcement learning in continuous state and action spaces. This provides one way of leveraging and combining the advantages of model-free and model-based approaches. Specifically, we differentiate through MPC by usin…
This article examines arbitrage investment in a mispriced asset when the mispricing follows the Ornstein-Uhlenbeck process and a credit-constrained investor maximizes a generalization of the Kelly criterion. The optimal differentiable and threshold policies are derived. The optimal differentiable policy is linear with …
The goal of policy gradient approaches is to find a policy in a given class of policies which maximizes the expected return. Given a differentiable model of the policy, we want to apply a gradient-ascent technique to reach a local optimum. We mainly use gradient ascent, because it is theoretically well researched. The …
New meta-learning approach for bandit policies that achieve high average reward.
problem Designing bandit policies that balance between worst-case and Bayesian assumptions.
method Differentiable parameterized policies optimized using policy gradients.
result Proposed algorithm achieves low regret and is practical for various bandit problems.
We propose a new algorithm for solving parabolic partial differential equations (PDEs) and backward stochastic differential equations (BSDEs) in high dimension, by making an analogy between the BSDE and reinforcement learning with the gradient of the solution playing the role of the policy function, and the loss functi…
New method uses differential equations for better counterfactual analysis.
problem Estimating counterfactual outcomes for policy analysis.
method Continuous-time approach to synthetic controls using controlled differential equations.
result Improves counterfactual estimation for irregularly aligned multivariate time series.
We propose a lifelong learning architecture, the Neural Computer Agent (NCA), where a Reinforcement Learning agent is paired with a predictive model of the environment learned by a Differentiable Neural Computer (DNC). The agent and DNC model are trained in conjunction iteratively. The agent improves its policy in simu…
A key problem in reinforcement learning for control with general function approximators (such as deep neural networks and other nonlinear functions) is that, for many algorithms employed in practice, updates to the policy or Q-function may fail to improve performance---or worse, actually cause the policy performance …
In sequence generation task, many works use policy gradient for model optimization to tackle the intractable backpropagation issue when maximizing the non-differentiable evaluation metrics or fooling the discriminator in adversarial learning. In this paper, we replace policy gradient with proximal policy optimization (…
Paper develops a new probabilistic method for American options using entropy regularization.
problem Finding optimal stopping times for American options with entropy regularization.
method Entropy-regularized penalization scheme based on Doob-Meyer-Mertens decomposition and reflected backward stochastic differential equations.
result Explicit convergence rates and policy improvement algorithm for American options.
Derives FPDE for equity-linked insurance pricing.
problem Calculating prices for insurance policies with complex payment histories.
method Variational techniques in functional Itô calculus.
result Derives a functional partial differential equation.
New algorithms for set union in differential privacy improve efficiency and accuracy.
problem Efficiently discovering items from private user data in natural language processing.
method Developed algorithms that allow users to contribute items in a dependent fashion, guided by a policy with contractive properties.
result New algorithms significantly outperform existing mechanisms in terms of efficiency and accuracy.
This paper bounds min-entropy leakage for Blowfish privacy using graph symmetries.
problem Bounding min-entropy leakage for Blowfish privacy mechanisms.
method Organizing analysis over symmetrical partitions corresponding to orbits of graph automorphism groups.
result Demonstrates a construction meeting the bound with asymptotic equality, showing tightness.
Study improves sampling efficiency of diffusion models using RL and PDEs.
problem Training neural stochastic differential equations without access to target samples.
method Proves equivalences between RL methods and PDEs, uses coarse time discretization.
result Improves sample efficiency and reduces computational cost.
We analyze the problem of learning a single user's preferences in an active learning setting, sequentially and adaptively querying the user over a finite time horizon. Learning is conducted via choice-based queries, where the user selects her preferred option among a small subset of offered alternatives. These queries …
FQE with deep neural networks achieves asymptotic normality and finite-sample bounds.
problem Theoretical understanding of FQE with general differentiable function approximators.
method Z-estimation theory applied to FQE with deep neural networks.
result FQE estimation error is asymptotically normal with explicit variance.
Offline RL algorithms protect privacy while learning from sensitive data.
problem Learning sensitive data in financial, legal, and healthcare applications.
method Differential privacy guarantees for offline RL algorithms.
result Provable prevention of privacy risks with strong learning bounds.
AI models outperform simple rules in cross-asset futures timing, especially with lower transaction costs.
problem Optimizing cross-asset portfolio weights using traditional forecasting and optimization methods.
method End-to-end AI policies that map market states directly to portfolio weights, trained on CME futures using a differentiable Sharpe ratio loss function.
result Transformer-based AI policies outperform simple rules and equal weighting, trading less and matching or exceeding equal weighting through moderate transaction costs.
Paper introduces privacy-preserving inventory policy learning for feature-based newsvendor with unknown demand.
problem Privacy-preserving inventory policy learning for feature-based newsvendor with unknown demand distribution and nonsmooth loss function.
method Developed a clipped noisy gradient descent algorithm based on convolution smoothing for optimal inventory estimation within f-differential privacy framework.
result Achieved privacy-preserving optimal inventory policy with provable privacy guarantees and desirable statistical precision.
Novel methods generate diverse policies in reinforcement learning.
problem Generating diverse policies in reinforcement learning.
method Constrained optimization perspective, introducing new metrics, and novel policy generation methods.
result Improved novelty and performance of generated policies.
Private RL algorithm with privacy guarantees for personalized medicine decisions.
problem Privacy-preserving reinforcement learning for personalized medicine decisions.
method Developed a private optimism-based RL algorithm using joint differential privacy (JDP).
result Achieved strong PAC and regret bounds with a privacy guarantee.
New k-step policy gradient method avoids local optima in restricted policy classes.
problem Suboptimal local optima in policy gradient methods for restricted policy classes.
method Proposes a k-step policy gradient method to escape myopic local optima. result The method converges to near optimal solutions exponentially close to the optimal deterministic policy.
Many reinforcement learning applications involve the use of data that is sensitive, such as medical records of patients or financial information. However, most current reinforcement learning methods can leak information contained within the (possibly sensitive) data on which they are trained. To address this problem, w…
Algorithm optimizes system design and control for better rewards.
problem Optimizing system design and control for maximum rewards.
method Deep reinforcement learning combining policy gradient and model-based optimization.
result DEPS algorithm outperforms state-of-the-art methods in various environments.
Model-based reinforcement learning has the potential to be more sample efficient than model-free approaches. However, existing model-based methods are vulnerable to model bias, which leads to poor generalization and asymptotic performance compared to model-free counterparts. In addition, they are typically based on the…
Paper proposes interpretable RL policies from a mixture of experts.
problem Making RL policies transparent and understandable in real-world applications.
method Policy iteration scheme with interpretable experts and prototypical states.
result Proposed algorithm learns policies comparable to neural networks but more interpretable.
Learning optimal feedback control laws capable of executing optimal trajectories is essential for many robotic applications. Such policies can be learned using reinforcement learning or planned using optimal control. While reinforcement learning is sample inefficient, optimal control only plans an optimal trajectory fr…
DE-QT detects optimal Q-learning stopping points.
problem Information loss in Q-learning during prolonged training.
method Introducing DE-QT to detect entropy changes in Q-tables.
result DE-QT identifies the best stopping point for Q-learning.
This paper optimizes sampling policies for Bayesian optimization to improve exploration and exploitation.
problem Improving the balance between exploration and exploitation in Bayesian optimization.
method Developed efficient methods to estimate and optimize non-myopic acquisition functions using rollout policies and stochastic gradient optimization.
result Efficient optimization of sampling policies leads to better performance in Bayesian optimization.
We study a non-parametric multi-armed bandit problem with stochastic covariates, where a key complexity driver is the smoothness of payoff functions with respect to covariates. Previous studies have focused on deriving minimax-optimal algorithms in cases where it is a priori known how smooth the payoff functions are. I…
Paper derives Thiele's equation for unit-linked policies in a stochastic volatility model.
problem Deriving pricing formula for unit-linked policies in a stochastic volatility model.
method Derives Thiele's differential equation for a unit-linked policy in the Heston-Hawkes model.
result Established a method to compute reserves in life insurance via solving Thiele's equation.
New algorithm reduces variance in Monte Carlo simulations using deep neural networks and policy gradients.
problem Reducing variance in Monte Carlo simulations for estimating function values.
method Optimal correlation search using deep neural networks and policy gradients.
result Optimal correlation function reduces variance by approximating and calibrating policy.
Paper presents a privacy-preserving method for dynamic assortment selection.
problem Personalized assortment recommendations with data privacy concerns.
method Perturbed upper confidence bound method integrating calibrated noise.
result Policy satisfies Joint Differential Privacy (JDP) with near-optimal regret bound.
Decision trees are ubiquitous in machine learning for their ease of use and interpretability. Yet, these models are not typically employed in reinforcement learning as they cannot be updated online via stochastic gradient descent. We overcome this limitation by allowing for a gradient update over the entire tree that i…