Study non-stationary MDPs using worst-case RL, proposing RATS algorithm.
problem Robust zero-shot planning in non-stationary stochastic environments.
method Model-Based Reinforcement Learning, worst-case approach.
result RATS algorithm demonstrates benefits over reference methods.
Monte Carlo Tree Search improves financial derivative hedging efficiency.
problem Optimizing pricing and hedging of derivative contracts in incomplete markets.
method Integrates tree search techniques with Reinforcement Learning for optimal control problems.
result Monte Carlo Tree Search outperforms Q-learning in sample efficiency and learning speed. A new algorithm, Regular Tree Search, tackles non-convex simulation optimization problems.
problem Non-convex objective functions in simulation optimization.
method Integrates adaptive sampling with recursive partitioning of the search space.
result Proves global convergence and reliably identifies the global optimum.
We consider the classical sparse regression problem of recovering a sparse signal x0 given a measurement vector y=Φx0+w. We propose a tree search algorithm driven by the deep neural network for sparse regression (TSN). TSN improves the signal reconstruction performance of the deep neural network designed for sp…
Paper proposes MCTSPO for better reinforcement learning policy optimization.
problem Local optima and saddle points in gradient-based methods and poor initialization in gradient-free methods.
method Monte-Carlo tree search combined with gradient-free optimization.
result Improved performance on reinforcement learning tasks with deceptive or sparse reward functions.
Paper uses CMAB to improve NAS efficiency and accuracy.
problem Improving efficiency and accuracy of NAS for DNNs.
method Formulated NAS as CMAB, used Nested Monte-Carlo Search.
result Discovered cell structure achieves comparable accuracy to state-of-the-art, 20x faster.
Bayesian optimization improves Monte-Carlo tree search for better state value estimation.
problem Slow convergence in Monte-Carlo tree search due to averaging in backpropagation.
method Softmax MCTS and Monotone MCTS, using Bayesian optimization with Gaussian process prior.
result Our framework outperforms previous methods in computer Go.
This paper improves self-play learning in games by manipulating experience distributions.
problem Improving self-play learning in games through better experience sampling.
method Three approaches: weighted sampling, Prioritized Experience Replay, and diversifying trajectories.
result Major improvements in early training performance in some games, minor improvements overall.
Bayesian methods improve drug discovery experiment design.
problem Optimizing drug screening experiments in high-dimensional data.
method Bayesian inference and optimisation with upper confidence bound algorithms, Thompson sampling, and sparse tree search.
result Sparse tree search techniques outperform other methods in drug toxicity screening.
A new algorithm improves sample complexity for thresholding in Monte Carlo Tree Search.
problem Determining if the root node value of a tree is at least a given threshold.
method Developed a δ-correct sequential sampling algorithm based on the Track-and-Stop strategy.
result Ratio-based modification of D-Tracking strategy reduces sample complexity and computational cost.
DR-MCTS improves decision quality and sample efficiency in complex environments.
problem Improving decision quality and sample efficiency in complex environments.
method Integrates Doubly Robust off-policy estimation into Monte Carlo Tree Search (MCTS).
result DR-MCTS achieves superior performance in Tic-Tac-Toe and VirtualHome tasks.
Paper shows MCTS approximates policy optimization, proposing an improved variant.
problem Improving AI performance through better MCTS algorithms.
method Shows MCTS approximates policy optimization problem, proposes a new algorithm.
result Proposed algorithm reliably outperforms original AlphaZero in multiple domains.
Finite-horizon lookahead policies are abundantly used in Reinforcement Learning and demonstrate impressive empirical success. Usually, the lookahead policies are implemented with specific planning methods such as Monte Carlo Tree Search (e.g. in AlphaZero). Referring to the planning problem as tree search, a reasonable…
Deep reinforcement learning has been successfully applied to several visual-input tasks using model-free methods. In this paper, we propose a model-based approach that combines learning a DNN-based transition model with Monte Carlo tree search to solve a block-placing task in Minecraft. Our learned transition model pre…
Parallelizes MCTS for continuous domains using leaf and root parallelization.
problem Solving challenging tasks in continuous domains using MCTS.
method Extends existing parallelization strategies to continuous domains, focusing on leaf and root parallelization.
result Proposes two final selection strategies for continuous states in root parallelization.
AutoML uses MCTS to optimize machine learning algorithms and hyperparameters.
problem Optimizing machine learning algorithms and hyperparameters efficiently.
method MCTS-based approach for hybrid optimization of machine learning portfolios.
result Mosaic outperforms Auto-Sklearn on OpenML 100 benchmark and Scikit-learn portfolio.
Combines MCTS and neural networks for efficient multi-period financial planning.
problem Solving multi-period financial planning models with high transaction costs and regime switching.
method Integrates Monte Carlo Tree Search with deep neural networks, using UTC and lookup search.
result Combined approach outperforms individual methods, solving complex models.
A hybrid model for Bayesian optimization handles mixed variables using MCTS for categorical and GP for continuous.
problem Optimizing functions with mixed variable types (continuous, integer, categorical).
method Merges MCTS for categorical and GP for continuous variables, integrates UCTS search strategy, and dynamically selects kernels.
result Hybrid models outperform traditional methods in Bayesian optimization.
Improves RL planning by proposing sub-goals hierarchically.
problem Sequential planning assumption in RL.
method Divide-and-Conquer Monte Carlo Tree Search (DC-MCTS).
result Improves navigation and control tasks.
New algorithm identifies optimal subtrees in fixed-budget tree search.
problem Identifying optimal subtrees in fixed-budget Monte Carlo Tree Search.
method ε-agnostic algorithm for max-min action identification.
result Misidentification probability decays exponentially with sample size.
New approach handles stochastic and partially-observable environments using discrete autoencoders and Monte Carlo tree search.
problem Challenges in planning for stochastic and partially-observable environments.
method Uses discrete autoencoders and a stochastic variant of Monte Carlo tree search.
result Significantly outperforms MuZero on stochastic chess and scales to DeepMind Lab.
PCTS optimizes noisy, delayed, multi-fidelity feedbacks in black-box optimization.
problem Optimizing unknown functions with noisy, delayed, and multi-fidelity feedbacks.
method ProCrastinated Tree Search (PCTS) with DUCB1 and DUCBV algorithms.
result PCTS achieves better regret bounds for delayed, noisy, and multi-fidelity feedbacks.
Algorithm finds best mixture of training datasets for improved validation performance.
problem Learning from mixture distributions with covariate shift.
method Combines SGD with optimistic tree search and model re-use over mixture space.
result Proves simple regret guarantees for recovering optimal mixture.
We present an extension of Monte Carlo Tree Search (MCTS) that strongly increases its efficiency for trees with asymmetry and/or loops. Asymmetric termination of search trees introduces a type of uncertainty for which the standard upper confidence bound (UCB) formula does not account. Our first algorithm (MCTS-T), whic…
A reinforcement learning framework combining value function and tree search planner for strategic and tactical decisions.
problem Strategic and tactical decision-making in discrete environments.
method Combines value function and tree search planner, using uncertainty modeling and risk measurement.
result Improves performance and learning speed on hard exploration environments.
MVPI framework optimizes risk in reinforcement learning, improving performance in robot simulations.
problem Optimizing risk in reinforcement learning control problems.
method Mean-Variance Policy Iteration (MVPI) framework for risk-averse control in MDPs.
result Risk-averse TD3 outperforms previous methods in robot simulation tasks.
Different models of capital exchange among economic agents have been proposed recently trying to explain the emergence of Pareto's wealth power law distribution. One important factor to be considered is the existence of risk aversion. In this paper we study a model where agents posses different levels of risk aversion,…
A new algorithm avoids worst-case outcomes in risky contexts.
problem Risk-averse behavior in contextual bandits is challenging.
method Developed a first risk-averse contextual bandit algorithm with online regret guarantees.
result First algorithm with an online regret guarantee for risk-averse contextual bandits.
A new measure quantifies how risk-averse different risk measures are.
problem Measuring the degree of risk aversion among different risk measures.
method Two axioms: normalization and linearity. Two formulas for the functional.
result Quantifies the degree of risk aversion among spectral risk measures.
Robo-advisors estimate clients' risk aversion using interactive questionnaires.
problem Estimating risk aversion of non-expert clients using adaptive questionnaires.
method Model risk aversion with cost functions and spectral risk measures. Use inverse reinforcement learning to design questions maximizing distinguishing power.
result Designing questions by maximizing distinguishing power achieves satisfactory accuracy in learning risk aversion with fewer than 50 questions.
Risk aversion is a key element of utility maximizing hedge strategies; however, it has typically been assigned an arbitrary value in the literature. This paper instead applies a GARCH-in-Mean (GARCH-M) model to estimate a time-varying measure of risk aversion that is based on the observed risk preferences of energy hed…
Active Reinforcement Learning (ARL) is a twist on RL where the agent observes reward information only if it pays a cost. This subtle change makes exploration substantially more challenging. Powerful principles in RL like optimism, Thompson sampling, and random exploration do not help with ARL. We relate ARL in tabular …
RiskMiner discovers formulaic alphas using MCTS for better performance.
problem Mining formulaic alphas without considering structural information and alpha correlations.
method Formulates alpha mining as an MDP and solves it with a risk-seeking MCTS.
result Our method outperforms state-of-the-art benchmarks and achieves the most profitable results.
While many recent advances in deep reinforcement learning (RL) rely on model-free methods, model-based approaches remain an alluring prospect for their potential to exploit unsupervised data to learn environment model. In this work, we provide an extensive study on the design of deep generative models for RL environmen…
Modeling informed trading with risk-averse market makers.
problem Understanding informed trading and its impact on market liquidity and risk premia.
method Connections between optimal transport theory and Kyle's model, including new characterizations of profits and duality.
result Liquidity is lower, assets exhibit short-term reversals, and risk premia depend on market maker inventories, which are mean reverting.
Study risk-averse insider's behavior in dynamic signal asset pricing.
problem Analyzing risk-averse insider's dynamic signal in asset pricing.
method Employing a weak conditioning methodology to construct a Schrödinger bridge, deriving necessary conditions for equilibrium.
result Derive explicit closed-form solutions for important cases.
The standard asset pricing models (the CCAPM and the Epstein-Zin non-expected utility model) counterintuitively predict that equilibrium asset prices can rise if the representative agent's risk aversion increases. If the income effect, which implies enhanced saving as a result of an increase in risk aversion, dominates…
Investigates how diversification preferences relate to risk attitudes.
problem Connecting diversification preferences to risk attitudes.
method Analyzes diversification preferences for various pairs of risks under different conditions.
result Diversification preferences for certain pairs of risks imply specific levels of risk aversion.
Recent advances in bandit tools and techniques for sequential learning are steadily enabling new applications and are promising the resolution of a range of challenging related problems. We study the game tree search problem, where the goal is to quickly identify the optimal move in a given game tree by sequentially sa…
New insights into risk aversion for complex decision models.
problem Understanding risk aversion in non-monotone decision models.
method Characterization of probabilistic risk aversion for generalized rank-dependent functions.
result Probabilistic risk aversion is determined by the distortion function, which is convex or scaled quantile-spread mixtures.
Unified formula for optimal portfolio under piecewise hyperbolic risk aversion.
problem Optimizing portfolios with piecewise hyperbolic risk aversion utilities.
method Derive a unified closed-form formula for the optimal portfolio.
result Unified formula reflects risk aversion behaviors and risk-taking behaviors.
New algorithm reduces sample complexity for planning in MDPs.
problem Planning in MDPs with unknown transitions.
method MDP-GapE, a trajectory-based MCTS algorithm.
result Proves upper bound on sample complexity in terms of sub-optimality gaps.
Quantum algorithm speeds up MIP solving by a near-quadratic factor.
problem Solving Mixed Integer Programs (MIPs) efficiently.
method Incremental-Quantum-Branch-and-Bound algorithm combining quantum speedup with classical search heuristics.
result Universal near-quadratic speedup over classical Branch-and-Bound algorithms.
Investigates a Kyle model with imperfect information and risk aversion.
problem Tackles a Kyle model with imperfect information and risk-averse informed traders.
method Solves an optimal transport problem and a filtering problem under specific measures.
result Constructs an equilibrium for the Gaussian Kyle model with imperfect information and risk aversion.
Deep Hedging learns optimal strategies for various risk levels.
problem Finding optimal hedging policies for diverse risk aversions.
method Continuous Reinforcement Learning with actor-critic algorithm.
result Demonstrated effectiveness in a stochastic volatility model.
Risk-averse reinforcement learning optimizes option hedging.
problem Optimizing option hedging under risk aversion and realistic market conditions.
method Applied Trust Region Volatility Optimization (TRVO) to a vanilla option hedging environment.
result The derived hedging strategy outperforms Black & Scholes and is robust to market variations.
New methods reduce bias in estimating optimality gaps for risk-averse stochastic programs.
problem Optimality gap estimation bias in risk-averse stochastic programs.
method Two independent samples, each estimating a different component of the optimality gap.
result Our method reduces bias in estimating optimality gaps for risk-averse problems.
Paper develops NPG for risk-averse RL with ECRMs, proving global convergence.
problem Ensuring reliable performance in stochastic RL problems with risk-averse policies.
method Developed natural policy gradient updates for ECRMs-based RL problems, proving global optimality and iteration complexity.
result Global convergence of risk-averse NPG algorithm with ECRMs.