Paper introduces a new method for risk-sensitive investment management using RL.
problem Risk-sensitive portfolio management with unknown model parameters.
method Combines RL and risk-sensitive stochastic control with Gaussian perturbations for exploration.
result Endogenous relative-entropy regularization and optimal investment strategy derived.
Separates estimation and control in risk-sensitive investment problems with partial observation.
problem Risk-sensitive investment problems with incomplete observation.
method Investigates separability of a general class of risk-sensitive investment management problems using a finite-dimensional filter.
result The separated problem is strictly equivalent to the original control problem.
Solves risk-sensitive investment via duality, entropic regularization, and RL.
problem Risk-sensitive portfolio management in a factor-based setting.
method Free energy-entropy duality, Kuroda-Nagai change-of-measure, RL algorithm.
result Direct analytical solution, explicit controls, two interpretations of optimal allocation.
We study a risk sensitive control version of the lifetime ruin probability problem. We consider a sequence of investments problems in Black-Scholes market that includes a risky asset and a riskless asset. We present a differential game that governs the limit behavior. We solve it explicitly and use it in order to find …
This paper introduces a method to incorporate risk sensitivity in RL using quadratic variation penalties.
problem Risk-sensitive reinforcement learning under entropy regularization.
method Equivalent martingale property and quadratic variation penalty for value process.
result The proposed method improves finite-sample performance in linear-quadratic control problems.
A new jump diffusion regime-switching model is introduced, which allows for linking jumps in asset prices with regime changes. We prove the existence and uniqueness of the solution to the risk-sensitive asset management criterion maximisation problem in this setting. We provide an ODE for the optimal value function, wh…
In this paper, we extend the jump-diffusion model proposed by Davis and Lleo to include jumps in asset prices as well as valuation factors. The criterion, following earlier work by Bielecki, Pliska, Nagai and others, is risk-sensitive optimization (equivalent to maximizing the expected growth rate subject to a constrai…
In this article we extend earlier work on the jump-diffusion risk-sensitive asset management problem [SIAM J. Fin. Math. (2011) 22-54] by allowing jumps in both the factor process and the asset prices, as well as stochastic volatility and investment constraints. In this case, the HJB equation is a partial integro-diffe…
Reinforcement learning for continuous-time risk-sensitive asset allocation
problem Continuous-time risk-sensitive asset allocation
method Free energy-entropy duality reformulation and q-learning actor-critic method result Optimal policy learning with high accuracy
Optimizes consumption under regime-switching economic states with risk-sensitive preferences.
problem Optimizing consumption in an economy with uncertain states and random shocks.
method Risk-sensitive optimization of consumption-utility with a Markov chain model of economic states and i.i.d. random shocks.
result Existence of unique optimal policy and value function in stationary policies.
The paper solves investment problems with uncertain factors using game theory.
problem Optimal forward investment in an incomplete market with model uncertainty.
method Combining stochastic differential games and ergodic BSDE approach.
result Representation of robust forward performance processes in factor form.
This survey reviews portfolio selection problem for long-term horizon. We consider two objectives: (i) maximize the probability for outperforming a target growth rate of wealth process (ii) minimize the probability of falling below a target growth rate. We study the asymptotic behavior of these criteria formulated as l…
For a stochastic factor model we maximize the long-term growth rate of robust expected power utility with parameter λ∈(0,1). Using duality methods the problem is reformulated as an infinite time horizon, risk-sensitive control problem. Our results characterize the optimal growth rate, an optimal long-term trading s…
We consider a long-term optimal investment problem where an investor tries to minimize the probability of falling below a target growth rate. From a mathematical viewpoint, this is a large deviation control problem. This problem will be shown to relate to a risk-sensitive stochastic control problem for a sufficiently l…
In this paper, we investigate dynamic optimization problems featuring both stochastic control and optimal stopping in a finite time horizon. The paper aims to develop new methodologies, which are significantly different from those of mixed dynamic optimal control and stopping problems in the existing literature, to stu…
Paper introduces RCaI, a risk-sensitive control method using Rényi divergence.
problem Risk-sensitive control in reinforcement learning.
method RCaI extends CaI using Rényi divergence variational inference.
result Risk-sensitive optimal policy can be obtained by solving a soft Bellman equation.
Study risk-sensitive RL in offline settings, improving efficiency and accuracy.
problem Efficiently derive near-optimal policies for risk-sensitive RL using offline data.
method Introduced two provably sample-efficient algorithms for risk-sensitive offline RL in linear MDPs.
result First provably efficient risk-sensitive offline RL algorithms.
Paper introduces risk-sensitive bandits with optimal arm mixtures.
problem Designing algorithms for risk-sensitive multi-armed bandits.
method Formalizes risk-sensitive bandits framework, identifies optimal arm mixtures, designs regret-efficient algorithms.
result Regret-efficient algorithms track optimal arm mixtures or solitary arms.
Improved risk-sensitive RL with exponential Bellman equation and better regret bounds.
problem Exponential gap between upper and lower bounds in risk-sensitive RL.
method Identified and addressed deficiencies in existing algorithms and analysis; developed novel analysis and exploration mechanism.
result Improved regret upper bounds over existing ones.
Study risk-sensitive reinforcement learning with Lipschitz dynamic risk measures, establishing regret bounds.
problem Risk-sensitive reinforcement learning in Markov decision processes.
method Two model-based algorithms for Lipschitz dynamic risk measures, focusing on regret bounds.
result Upper bounds demonstrate optimal dependencies on actions and episodes, reflecting risk sensitivity vs. sample complexity trade-off.
Develops an actor-critic algorithm for risk-sensitive Markov decision processes.
problem Risk-sensitive cost criterion in Markov decision processes.
method Actor-critic algorithm with function approximation.
result Asymptotic convergence of the actor-critic algorithm.
Overview of risk-sensitive Markov decision processes with Optimized Certainty Equivalent.
problem Optimizing decision-making under risk in Markov processes.
method Analyzes risk-sensitive criteria using Optimized Certainty Equivalent, including entropic risk and Conditional Value-at-Risk.
result Conditions for the existence of optimal policies and solution procedures are provided.
We study risk-sensitive imitation learning where the agent's goal is to perform at least as well as the expert in terms of a risk profile. We first formulate our risk-sensitive imitation learning setting. We consider the generative adversarial approach to imitation learning (GAIL) and derive an optimization problem for…
Kuroda and Nagai \cite{KN} state that the factor process in the Risk Sensitive control Asset Management (RSCAM) is stable under the Föllmer-Schweizer minimal martingale measure . Fleming and Sheu \cite{FS} and more recently Föllmer and Schweizer \cite{FoS} have observed that the role of the minimal martingale measure i…
We establish a stochastic maximum principle (SMP) for control problems of partially observed diffusions of mean-field type with risk-sensitive performance functionals.
Paper tackles risk-sensitive decision-making under uncertainty.
problem Risk-sensitive decision-making problem under uncertainty.
method Formulated as a stochastic control problem, delineated necessary optimality conditions.
result Illustrative examples from optimal betting and inventory management support the theory.
This paper analyzes risk-sensitive reinforcement learning with Conditional Value-at-Risk (CVaR) for robust Markov Decision Processes.
problem Risk-sensitive reinforcement learning for robust Markov Decision Processes (RMDPs) with state-action-dependent ambiguity sets.
method The paper establishes a connection between robustness and risk sensitivity, defining a new risk measure NCVaR and proposing value iteration algorithms.
result The proposed approach using NCVaR optimization and value iteration algorithms can solve problems with state-action-dependent ambiguity sets.
New algorithms optimize risk in reinforcement learning with exponential utility.
problem Optimizing rewards under risk in reinforcement learning with unknown transition kernels.
method Two model-free algorithms: Risk-Sensitive Value Iteration (RSVI) and Risk-Sensitive Q-learning (RSQ).
result Proved near-optimal regret bounds for RSVI and RSQ.
Study gap-dependent regret bounds for risk-sensitive RL.
problem Risk-sensitive reinforcement learning with entropic risk measure.
method Propose cascaded gaps to adapt to problem structures, derive regret bounds.
result Exponential improvement over existing bounds in appropriate settings.
Novel framework for risk-sensitive reinforcement learning using martingale decomposition.
problem Risk sensitivity in sequential decision-making with uncertain rewards.
method Martingale decomposition and chaotic variation for reward uncertainty, integrated into model-free reinforcement learning algorithms.
result Demonstrated relevance of risk-sensitive reinforcement learning in grid world and portfolio optimization problems.
Novel framework for risk-sensitive reinforcement learning with robustness against uncertainty.
problem Risk-sensitive reinforcement learning with uncertainty in transition dynamics.
method Developed a risk-sensitive robust Markov decision process (RSRMDP), derived its Bellman equation, and proposed a Bayesian Dynamic Programming (Bayesian DP) algorithm.
result Demonstrated convergence to near-optimal policies and analyzed sample and computational complexities.
The paper studies risk-sensitive learning schemes and provides learning bounds for empirical OCE minimizers.
problem Risk-sensitive learning aims to minimize risk-averse measures of loss.
method Proposes learning bounds for empirical OCE minimizers based on Rademacher average and variance.
result Provides two learning bounds on the performance of empirical OCE minimizers.
Framework improves ETF volatility forecasting by adapting to market conditions.
problem Challenges in volatility forecasting due to shifting market conditions and varying model performance.
method Risk-sensitive specialist routing using online risk-sensitive evaluation and state-dependent gating.
result Reduces forecast loss by 24% and underprediction loss by 22% compared to rolling-best baseline.
Formulates Markov property for risk-sensitive dynamic optimisation.
problem Risk-sensitive dynamic optimisation problems in discrete time.
method Formulates probabilistic Markov property under dynamic risk framework.
result Property holds for standard risk measures and has multiple equivalent versions.
The paper addresses human-like decision-making in multi-agent systems using bounded risk-sensitive Markov Games.
problem Modeling human-like decision-making in multi-agent systems with risk-seeking and loss-aversion behaviors.
method Forward policy design and inverse reward learning with iterative reasoning and cumulative prospect theory.
result The proposed algorithms demonstrate both risk-averse and risk-seeking behaviors in multi-agent systems.
New algorithm for risk-sensitive reinforcement learning with natural policy gradients.
problem Risk-sensitive reinforcement learning with downside risk constraints.
method Introduce a new Bellman equation to estimate the lower partial moment of returns, use natural policy gradients, and extend Reward Constrained Policy Optimization.
result Sample-efficient estimation of partial moments and effective risk-sensitive control.
In this paper we study mean-field type control problems with risk-sensitive performance functionals. We establish a stochastic maximum principle (SMP) for optimal control of stochastic differential equations (SDEs) of mean-field type, in which the drift and the diffusion coefficients as well as the performance function…
Develops variational framework for LQG risk-sensitive MFGs with major-minor interactions.
problem Risk-sensitive optimal control in LQG systems with major-minor interactions.
method Variational approach, nonlinear necessary and sufficient condition of optimality, equivalent risk-neutral measure, Markovian closed-loop best-response strategies.
result Derives optimal control strategies for LQG risk-sensitive MFGs with major-minor interactions, establishing Nash and ε-Nash equilibria. A new method for risk-sensitive reinforcement learning using Spectral Risk Measures.
problem Incorporating risk sensitivity into reinforcement learning algorithms.
method Proposes a novel framework for optimizing Spectral Risk Measures in both online and offline RL algorithms.
result Demonstrates consistent outperformance over existing risk-sensitive methods in various domains.
Autonomous systems can substantially enhance a human's efficiency and effectiveness in complex environments. Machines, however, are often unable to observe the preferences of the humans that they serve. Despite the fact that the human's and machine's objectives are aligned, asymmetric information, along with heterogene…
Optimizes portfolios with costs, showing existence of optimal strategies.
problem Risk-sensitive portfolio optimization with transaction costs.
method Log-return i.i.d. framework, Bellman equation analysis.
result Existence of optimal strategies for risk-averse and risk-seeking cases.
We study the continuous time portfolio optimization model on the market where the mean returns of individual securities or asset categories are linearly dependent on underlying economic factors. We introduce the functional Qγ featuring the expected earnings yield of portfolio minus a penalty term proportional with a…
Efficient RL in partially observable risk-sensitive environments with hindsight observations.
problem Risk-sensitive reinforcement learning in partially observable environments.
method Integrates hindsight observations into POMDP framework, develops novel RL algorithm.
result Achieves polynomial regret with provable efficiency, outperforming existing methods.
We discuss a class of risk-sensitive portfolio optimization problems. We consider the portfolio optimization model investigated by Nagai in 2003. The model by its nature can include fixed income securities as well in the portfolio. Under fairly general conditions, we prove the existence of optimal portfolio in both fin…
We demonstrate an application of risk-sensitive reinforcement learning to optimizing execution in limit order book markets. We represent taking order execution decisions based on limit order book knowledge by a Markov Decision Process; and train a trading agent in a market simulator, which emulates multi-agent interact…
We explore a new method for discrete-time control problems using randomization and entropy.
problem Discrete-time linear-exponential quadratic Gaussian (LEQG) control problem.
method Introduce exploration through randomization and apply duality between free energy and relative entropy.
result Reduced LEQG problem to equivalent risk-neutral LQG control problem with entropy regularization.
PRUDEX-Compass evaluates FinRL methods on 6 axes for financial market investments.
problem Insufficient evaluation of FinRL methods in financial markets.
method Introduces PRUDEX-Compass with 6 axes and 17 measures for evaluation.
result Demonstrates the effectiveness of PRUDEX-Compass on 4 real-world datasets.
In this paper we consider long-run risk sensitive average cost impulse control applied to a continuous-time Feller-Markov process. Using the probabilistic approach, we show how to get a solution to a suitable continuous-time Bellman equation and link it with the impulse control problem. The optimal strategy for the und…