Revisits PPO design choices, exposing failure modes and proposing alternatives.
problem Failure modes of standard PPO in new environments.
method Revisits standard PPO design choices, exposes failure modes, and proposes alternative approaches.
result Alternative design choices prevent failure modes in new environments.
AI framework for automated policy-making connects with econometrics and social choice.
problem Improving government and economic policy-making through AI.
method Social Environment Design framework integrating Reinforcement Learning, EconCS, and Computational Social Choice.
result Promotes ethical and responsible decision making by solving open problems.
Study investigates key design choices in on-policy RL algorithms.
problem Lack of transparency in RL algorithm implementations.
method Implemented >50 design choices in a unified RL framework.
result Insights and recommendations for on-policy RL training.
Paper tackles RLHF with DCPPO method, proving near-optimal suboptimality.
problem Challenges in offline RLHF with limited human feedback and bounded rationality.
method DCPPO method involving three stages: MLE, reward function recovery, and pessimistic value iteration.
result DCPPO's suboptimality almost matches classical pessimistic offline RL in terms of distribution shift and dimension.
Estimates personalized policies robust to shifts in target populations.
problem Estimating policies that perform well in diverse target populations.
method Develops methods for estimating robust policies considering shifts in outcomes and characteristics.
result Welfare-maximizing policies are robust to certain shifts in potential outcomes.
We consider assortment optimization over a continuous spectrum of products represented by the unit interval, where the seller's problem consists of determining the optimal subset of products to offer to potential customers. To describe the relation between assortment and customer choice, we propose a probabilistic choi…
The problem of retrosynthetic planning can be framed as one player game, in which the chemist (or a computer program) works backwards from a molecular target to simpler starting materials though a series of choices regarding which reactions to perform. This game is challenging as the combinatorial space of possible cho…
New framework optimizes multi-asset portfolio choice for high dimensions.
problem Optimizing high-dimensional continuous-time portfolio choice.
method Combines Pontryagin's Maximum Principle with BPTT for neural network policy learning.
result Achieves near-optimal policies with improved efficiency and precision.
We consider the problem of multi-product dynamic pricing, in a contextual setting, for a seller of differentiated products. In this environment, the customers arrive over time and products are described by high-dimensional feature vectors. Each customer chooses a product according to the widely used Multinomial Logit (…
The paper proposes a new policy for optimal treatment allocation based on quantile treatment effects.
problem Optimal treatment allocation policies that target distributional welfare, especially when individuals are heterogeneous.
method The approach involves allocating treatments based on the conditional quantile of individual treatment effects (QoTE), considering both prudent and negligent policymakers.
result The proposed minimax policies are robust to model uncertainty and can be generalized to various settings.
New method selects best offline RL policies from logged data.
problem Hyperparameter selection challenges offline RL.
method Offline hyperparameter selection for RL algorithms.
result Reliable ranking and selection of policies across hyperparameters.
AgensFlow learns multi-agent coordination policies from experience.
problem Difficult coordination choices in multi-agent systems built on LLMs.
method Online policy learning from repeated trajectories, treating decisions as learnable.
result Learned routing improves coordination-heavy workflows over static wiring.
We study the pricing problem faced by a firm that sells a large number of products, described via a wide range of features, to customers that arrive over time. Customers independently make purchasing decisions according to a general choice model that includes products features and customers' characteristics, encoded as…
Develops deep learning models for choice modeling.
problem Computational intractability and sample inefficiency in existing choice model learning methods.
method Deep learning-based choice models in two settings: feature-free and feature-based.
result Demonstrates improved recovery of existing choice models and reduced sample complexity.
Naive investors make riskier choices than optimal strategies in continuous-time finance.
problem Continuous-time Markowitz portfolio selection with naive reoptimization.
method Analytical derivation of naive policies from discretely naive policies.
result Naive policies are always riskier and less efficient than equilibrium policies.
Standard reinforcement learning methods aim to master one way of solving a task whereas there may exist multiple near-optimal policies. Being able to identify this collection of near-optimal policies can allow a domain expert to efficiently explore the space of reasonable solutions. Unfortunately, existing approaches t…
We analyze the problem of learning a single user's preferences in an active learning setting, sequentially and adaptively querying the user over a finite time horizon. Learning is conducted via choice-based queries, where the user selects her preferred option among a small subset of offered alternatives. These queries …
Approximate linear programs (ALPs) are well-known models based on value function approximations (VFAs) to obtain policies and lower bounds on the optimal policy cost of discounted-cost Markov decision processes (MDPs). Formulating an ALP requires (i) basis functions, the linear combination of which defines the VFA, and…
Study optimizes portfolio allocation policies using off-policy data and constraints.
problem Optimizing portfolio allocation policies under constraints using off-policy data.
method Solves a minimax objective with off-policy estimators and online learning to control constraint violations.
result Constructs near-optimal allocation policies for various regimes of operation and constraints.
An optimal feedback controller for a given Markov decision process (MDP) can in principle be synthesized by value or policy iteration. However, if the system dynamics and the reward function are unknown, a learning agent must discover an optimal controller via direct interaction with the environment. Such interactive d…
Recently, reinforcement learning (RL) algorithms have demonstrated remarkable success in learning complicated behaviors from minimally processed input. However, most of this success is limited to simulation. While there are promising successes in applying RL algorithms directly on real systems, their performance on mor…
Evolutionary Strategies optimize hyper-parameters for off-policy learning.
problem Hyper-parameter sensitivity in off-policy learning.
method Application of Evolutionary Strategies for online hyper-parameter tuning.
result Our method outperforms state-of-the-art baselines.
We study the dynamic assortment planning problem, where for each arriving customer, the seller offers an assortment of substitutable products and customer makes the purchase among offered products according to an uncapacitated multinomial logit (MNL) model. Since all the utility parameters of MNL are unknown, the selle…
Paper tackles optimal policy learning with observational data in multi-action scenarios.
problem Optimal policy learning in multi-action settings with observational data.
method Review of estimation approaches, analysis of risk preference, discussion of potential failures.
result Average regret of a policy with multi-valued treatment is contingent on the decision-maker's attitude towards risk.
New model combines neural networks and embeddings for better choice modeling interpretability.
problem Limited behavioral insights in embedding representations for categorical variables.
method Combines discrete choice models and neural networks using embeddings for interpretability.
result Proposed models deliver state-of-the-art predictive performance while preserving interpretability.
PS framework selects best policy from library for CSO problems.
problem Policy selection in CSO with heterogeneous performance across covariate space.
method PS framework constructs library of candidate policies and learns a meta-policy to select the best one.
result PS consistently outperforms best single policy in heterogeneous CSO problems.
Accelerates policy optimization in RL with optimistic and adaptive updates.
problem Improving policy optimization methods in reinforcement learning.
method Integrates foresight into policy improvement step via optimistic and adaptive updates.
result Designs an optimistic policy gradient algorithm, adaptive via meta-gradient learning.
HO2 learns options from data efficiently, improving robot manipulation tasks.
problem Learning options from raw pixel inputs in 3D robot manipulation tasks.
method HO2 infers likely option choices and trains all policy components off-policy.
result HO2 outperforms existing methods on 3D robot manipulation tasks.
Deep Reinforcement Learning (DRL) algorithms for continuous action spaces are known to be brittle toward hyperparameters as well as \cut{being}sample inefficient. Soft Actor Critic (SAC) proposes an off-policy deep actor critic algorithm within the maximum entropy RL framework which offers greater stability and empiric…
A method for dynamic portfolio choice with uncertain parameters using Pontryagin projection.
problem Continuous-time CRRA portfolio choice in markets with estimated and uncertain coefficients.
method Simulation-based two-stage solver (DPO + Pontryagin projection) to maximize ex-ante objective.
result Projection stabilizes learning and accurately recovers analytic decisions, improving over model-free PPO.
New mirror maps improve PMD performance in reinforcement learning.
problem Limited exploration of PMD's full potential due to focus on negative entropy.
method Evolutionary strategies to identify and learn more efficient mirror maps.
result Learned mirror maps outperform negative entropy in various environments.
Policy optimization is an effective reinforcement learning approach to solve continuous control tasks. Recent achievements have shown that alternating online and offline optimization is a successful choice for efficient trajectory reuse. However, deciding when to stop optimizing and collect new trajectories is non-triv…
The paper analyzes the sample complexities for policy evaluation with linear function approximation.
problem Policy evaluation with linear function approximation in discounted infinite horizon Markov decision processes.
method Investigates sample complexities for two policy evaluation algorithms: TD and TDC.
result Establishes high-probability sample complexity bounds for policy evaluation algorithms.
DBQPG improves policy gradient estimation with fewer samples.
problem Accurate policy gradient estimation with limited samples.
method Deep Bayesian Quadrature Policy Gradient (DBQPG).
result DBQPG provides more accurate and less variable gradient estimates.
This survey is an introduction to asymptotic methods for portfolio-choice problems with small transaction costs. We outline how to derive the corresponding dynamic programming equations and simplify them in the small-cost limit. This allows to obtain explicit solutions in a wide range of settings, which we illustrate f…
We accelerate PMD algorithms for reinforcement learning using functional methods.
problem Improving reinforcement learning algorithms for large-scale optimization.
method Proposed a momentum-based update for PMD algorithms leveraging duality.
result The proposed method accelerates reinforcement learning algorithms.
Optimizes biomolecular simulations by ranking adaptive sampling policies.
problem Efficiently sampling biomolecular systems to capture complex dynamical behaviors.
method Metric-driven ranking of adaptive sampling policies to identify the optimal policy for each round.
result Different adaptive sampling policies lead to faster convergence and improved sampling performance.
Investors benefit from long horizons in a market with mean-reverting equity returns.
problem Optimal portfolio choice in a market with mean-reverting risk-free rate and equity risk-premium.
method Mean-variance optimization, Euler-Lagrange equation, Calculus of Variations, spectral problem.
result Optimal policies are characterized by eigenvalues of the lambda-matrix, leading to better risk-return trade-offs for long-term investors.
New framework for learning policies that converge in out-of-sample regions.
problem Reliable out-of-sample recovery in imitation learning.
method Contractive dynamical systems and recurrent equilibrium networks.
result Policy rollouts converge regardless of perturbations, enabling efficient OOS recovery.
Paper provides convergence guarantees for off-policy NAC with finite sample complexity.
problem Convergence analysis of off-policy natural actor-critic algorithm.
method Finite-sample analysis with Importance Sampling and Q-trace algorithm.
result Converges to global optimal policy with sample complexity O(ε−3log2(1/ε)). We propose a trust region method for policy optimization that employs Quasi-Newton approximation for the Hessian, called Quasi-Newton Trust Region Policy Optimization QNTRPO. Gradient descent is the de facto algorithm for reinforcement learning tasks with continuous controls. The algorithm has achieved state-of-the-art…
Advocates focusing on utility functions to avoid unfair outcomes.
problem Unfair outcomes from fairness criteria in optimizing policies.
method Defines value of information fairness and proposes modifying utility functions.
result Value of information fairness leads to better answers than existing fairness notions.
New method optimizes policies without assuming known link functions between preferences and rewards.
problem Policy alignment with unknown and unrestricted link functions.
method Formulates an f-divergence-constrained reward maximization problem, learning policies directly. result Induces a semiparametric single-index binary choice model for policy alignment.
Automates infectious disease policy-making via inference in epidemiological models.
problem Improving policy-making for infectious diseases during pandemics.
method Performing inference in existing epidemiological models using a probabilistic programming language.
result Automated inference leads to better disease progression outcomes and policy prescriptions.
Novel framework for policy optimization with general parameterization and linear convergence.
problem Lack of theoretical guarantees for policy optimization with general parameterization schemes.
method Mirror descent approach for policy optimization with general parameterization.
result First result of linear convergence for policy-gradient-based method with general parameterization.
We model a network economy with three sectors: downstream firms, upstream firms, and banks. Agents are linked by productive and credit relationships so that the behavior of one agent influences the behavior of the others through network connections. Credit interlinkages among agents are a source of bankruptcy diffusion…
Study optimal portfolio choice with risk control for log-returns.
problem Optimal portfolio choice with risk management in continuous-time markets.
method Characterized optimal terminal wealth using concave envelope, derived analytical expressions for optimal wealth and policy, found efficient frontier.
result Efficient frontier is concave curve connecting minimum-risk to growth-optimal portfolios, not a vertical line.
A novel efficient method for computing the Knowledge-Gradient policy for Continuous Parameters (KGCP) for deterministic optimization is derived. The differences with Expected Improvement (EI), a popular choice for Bayesian optimization of deterministic engineering simulations, are explored. Both policies and the Upper …