Paper introduces RCaI, a risk-sensitive control method using Rényi divergence.
problem Risk-sensitive control in reinforcement learning.
method RCaI extends CaI using Rényi divergence variational inference.
result Risk-sensitive optimal policy can be obtained by solving a soft Bellman equation.
New algorithm for risk-sensitive reinforcement learning with natural policy gradients.
problem Risk-sensitive reinforcement learning with downside risk constraints.
method Introduce a new Bellman equation to estimate the lower partial moment of returns, use natural policy gradients, and extend Reward Constrained Policy Optimization.
result Sample-efficient estimation of partial moments and effective risk-sensitive control.
Paper introduces a new method for risk-sensitive investment management using RL.
problem Risk-sensitive portfolio management with unknown model parameters.
method Combines RL and risk-sensitive stochastic control with Gaussian perturbations for exploration.
result Endogenous relative-entropy regularization and optimal investment strategy derived.
The paper addresses human-like decision-making in multi-agent systems using bounded risk-sensitive Markov Games.
problem Modeling human-like decision-making in multi-agent systems with risk-seeking and loss-aversion behaviors.
method Forward policy design and inverse reward learning with iterative reasoning and cumulative prospect theory.
result The proposed algorithms demonstrate both risk-averse and risk-seeking behaviors in multi-agent systems.
This work tackles risk-sensitive deep RL by optimizing policies with variance constraints.
problem Risk and aleatoric uncertainty in deep reinforcement learning.
method Lagrangian and Fenchel dualities to transform the problem into an unconstrained saddle-point policy optimization problem, and an actor-critic algorithm to iteratively update policy, Lagrange multiplier, and Fenchel dual variable.
result The proposed actor-critic algorithm finds a globally optimal policy at a sublinear rate.
Study risk-sensitive market making with entropy regularization for better quote control.
problem Risk-sensitive market making with exponential utility and penalties.
method Entropy-regularized certainty-equivalent Bellman policies for discrete-time market dynamics.
result Proves convergence and performance bounds for entropy-regularized policies.
This paper introduces a method to incorporate risk sensitivity in RL using quadratic variation penalties.
problem Risk-sensitive reinforcement learning under entropy regularization.
method Equivalent martingale property and quadratic variation penalty for value process.
result The proposed method improves finite-sample performance in linear-quadratic control problems.
The objective in a traditional reinforcement learning (RL) problem is to find a policy that optimizes the expected value of a performance metric such as the infinite-horizon cumulative discounted or long-run average cost/reward. In practice, optimizing the expected value alone may not be satisfactory, in that it may be…
Efficiently computes optimal policies for Entropic Risk Measures.
problem Optimizing risk-sensitive metrics in MDPs is computationally expensive.
method Uses Entropic Risk Measures and novel structural analysis for efficient computation.
result Achieves strong performance in various decision-making scenarios.
Novel framework for risk-sensitive reinforcement learning using martingale decomposition.
problem Risk sensitivity in sequential decision-making with uncertain rewards.
method Martingale decomposition and chaotic variation for reward uncertainty, integrated into model-free reinforcement learning algorithms.
result Demonstrated relevance of risk-sensitive reinforcement learning in grid world and portfolio optimization problems.
Enhances GFlowNets with distributional approach for risk-sensitive policies.
problem Limited applicability of current GFlowNet framework in handling stochastic reward functions.
method Adopting a distributional paradigm, parameterizing each edge flow through quantile functions, and introducing a risk-sensitive learning algorithm.
result Significant improvement on benchmarks due to enhanced training algorithm, even in deterministic reward settings.
Develops an actor-critic algorithm for risk-sensitive Markov decision processes.
problem Risk-sensitive cost criterion in Markov decision processes.
method Actor-critic algorithm with function approximation.
result Asymptotic convergence of the actor-critic algorithm.
Optimizes consumption under regime-switching economic states with risk-sensitive preferences.
problem Optimizing consumption in an economy with uncertain states and random shocks.
method Risk-sensitive optimization of consumption-utility with a Markov chain model of economic states and i.i.d. random shocks.
result Existence of unique optimal policy and value function in stationary policies.
Novel framework for risk-sensitive reinforcement learning with robustness against uncertainty.
problem Risk-sensitive reinforcement learning with uncertainty in transition dynamics.
method Developed a risk-sensitive robust Markov decision process (RSRMDP), derived its Bellman equation, and proposed a Bayesian Dynamic Programming (Bayesian DP) algorithm.
result Demonstrated convergence to near-optimal policies and analyzed sample and computational complexities.
Overview of risk-sensitive Markov decision processes with Optimized Certainty Equivalent.
problem Optimizing decision-making under risk in Markov processes.
method Analyzes risk-sensitive criteria using Optimized Certainty Equivalent, including entropic risk and Conditional Value-at-Risk.
result Conditions for the existence of optimal policies and solution procedures are provided.
Study risk-sensitive RL in offline settings, improving efficiency and accuracy.
problem Efficiently derive near-optimal policies for risk-sensitive RL using offline data.
method Introduced two provably sample-efficient algorithms for risk-sensitive offline RL in linear MDPs.
result First provably efficient risk-sensitive offline RL algorithms.
A new method for risk-sensitive reinforcement learning using Spectral Risk Measures.
problem Incorporating risk sensitivity into reinforcement learning algorithms.
method Proposes a novel framework for optimizing Spectral Risk Measures in both online and offline RL algorithms.
result Demonstrates consistent outperformance over existing risk-sensitive methods in various domains.
Paper introduces risk-sensitive bandits with optimal arm mixtures.
problem Designing algorithms for risk-sensitive multi-armed bandits.
method Formalizes risk-sensitive bandits framework, identifies optimal arm mixtures, designs regret-efficient algorithms.
result Regret-efficient algorithms track optimal arm mixtures or solitary arms.
We study a risk sensitive control version of the lifetime ruin probability problem. We consider a sequence of investments problems in Black-Scholes market that includes a risky asset and a riskless asset. We present a differential game that governs the limit behavior. We solve it explicitly and use it in order to find …
A new approach to risk-sensitive reinforcement learning tackles computational challenges.
problem Computational challenges in estimating risk-sensitive policies for MDPs with finite state and action spaces.
method Proposes a new risk measure called 'caution' and uses a stochastic primal-dual method with KL divergence.
result Demonstrates improved reliability in reward accumulation without additional computational costs.
Improved sample complexity for identifying best policies in risk-sensitive reinforcement learning.
problem Identifying approximately optimal policies in risk-sensitive reinforcement learning with exponential horizon dependence.
method Forward-model based algorithm with KL-based exploration bonuses adapted for entropic criterion, leveraging smoothness properties of exponential utility and a new stopping rule.
result Achieved sample complexity matching the lower bound, closing the gap between upper and lower bounds.
Study risk-sensitive reinforcement learning with entropic risk measures and generative models.
problem Risk-sensitive reinforcement learning in discounted MDPs with recursive entropic risk measures.
method Introduced Model-Based ERM Q-Value Iteration (MB-RS-QVI) and derived PAC bounds on sample complexity for value and policy learning. result PAC bounds show exponential dependence on ∣β∣/(1−γ), with tight bounds in S and A. Several authors have recently developed risk-sensitive policy gradient methods that augment the standard expected cost minimization problem with a measure of variability in cost. These studies have focused on specific risk-measures, such as the variance or conditional value at risk (CVaR). In this work, we extend the p…
In this work, we build on recent advances in distributional reinforcement learning to give a generally applicable, flexible, and state-of-the-art distributional variant of DQN. We achieve this by using quantile regression to approximate the full quantile function for the state-action return distribution. By reparameter…
The paper studies risk-sensitive MDPs with recursive risk measures.
problem Risk-sensitive decision-making in MDPs with unbounded costs.
method Recursive application of static risk measures, Bellman equation derivation, existence of optimal policies.
result Existence of Markovian optimal policies for infinite planning horizons, contractive model for stationary optimal policy.
We demonstrate an application of risk-sensitive reinforcement learning to optimizing execution in limit order book markets. We represent taking order execution decisions based on limit order book knowledge by a Markov Decision Process; and train a trading agent in a market simulator, which emulates multi-agent interact…
Paper introduces a trading agent using LLMs for risk assessment and trading recommendations.
problem Developing a trading agent that can handle financial risks effectively.
method Extending CPPO algorithm with LLM-generated risk assessment and trading signals from financial news.
result Backtesting shows improved performance of the trading agent compared to benchmarks.
In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such c…
Dynamic reinsurance minimizes insurer's cost of capital over time.
problem Minimizing insurer's cost of capital in a dynamic reinsurance setting.
method Dynamic extension of the static optimal reinsurance problem, viewed as a risk-sensitive Markov Decision Process.
result Existence of a stationary Markovian optimal reinsurance policy under an infinite planning horizon.
Paper introduces a new distributional successor measure for reinforcement learning.
problem Learning the distributional consequences of behavior in reinforcement learning.
method Formulates distributional successor measure as a distribution over distributions, proposes algorithm to learn it from data.
result Demonstrates zero-shot risk-sensitive policy evaluation.
In this article we extend earlier work on the jump-diffusion risk-sensitive asset management problem [SIAM J. Fin. Math. (2011) 22-54] by allowing jumps in both the factor process and the asset prices, as well as stochastic volatility and investment constraints. In this case, the HJB equation is a partial integro-diffe…
Study risk-sensitive reinforcement learning with optimized certainty equivalents.
problem Risk-sensitive reinforcement learning in finite discounted MDPs.
method Analyzed a simple model-based approach and derived PAC sample complexity bounds.
result Established tight sample complexity bounds for value and policy learning.
Reinforcement learning for continuous-time risk-sensitive asset allocation
problem Continuous-time risk-sensitive asset allocation
method Free energy-entropy duality reformulation and q-learning actor-critic method result Optimal policy learning with high accuracy
Bayesian neural networks with latent variables are scalable and flexible probabilistic models: They account for uncertainty in the estimation of the network weights and, by making use of latent variables, can capture complex noise patterns in the data. We show how to extract and decompose uncertainty into epistemic and…
Paper proposes a new DRL algorithm optimizing Spectral Risk Measures for better risk management.
problem Inconsistencies and conservatism in existing risk measures in DRL.
method Optimizes a broader class of static Spectral Risk Measures (SRM) in DRL.
result Demonstrates improved performance over existing risk-neutral and risk-sensitive DRL models.
Deep reinforcement learning boosts commodities trading performance.
problem Improving algorithmic trading performance in commodities markets.
method Formulated as a stochastic dynamical system, employed actor-based and actor-critic-based policy gradient algorithms with CNN and LSTM function approximators.
result DRL models increase Sharpe ratio by 83% compared to buy-and-hold.
Unified market making controls risk, arbitrage, and volatility surfaces.
problem Market making risk, arbitrage, and volatility surface consistency.
method Constrained RL and stochastic control for risk-sensitive execution and hedging.
result Agent achieves positive P&L with zero calendar and butterfly violations.
Develops RL for dynamic risk assessment in stochastic optimization.
problem Time-consistent risk assessment in stochastic optimization problems.
method Model-free reinforcement learning with dynamic convex risk measures, time-consistent dynamic programming, policy gradient updates, actor-critic neural network optimization.
result Demonstrates optimal policies for statistical arbitrage, financial hedging, and robot control.
This paper considers a portfolio optimization problem in which asset prices are represented by SDEs driven by Brownian motion and a Poisson random measure, with drifts that are functions of an auxiliary diffusion 'factor' process. The criterion, following earlier work by Bielecki, Pliska, Nagai and others, is risk-sens…
Improved risk-sensitive RL with exponential Bellman equation and better regret bounds.
problem Exponential gap between upper and lower bounds in risk-sensitive RL.
method Identified and addressed deficiencies in existing algorithms and analysis; developed novel analysis and exploration mechanism.
result Improved regret upper bounds over existing ones.
Study risk-sensitive reinforcement learning with Lipschitz dynamic risk measures, establishing regret bounds.
problem Risk-sensitive reinforcement learning in Markov decision processes.
method Two model-based algorithms for Lipschitz dynamic risk measures, focusing on regret bounds.
result Upper bounds demonstrate optimal dependencies on actions and episodes, reflecting risk sensitivity vs. sample complexity trade-off.
Paper develops NPG for risk-averse RL with ECRMs, proving global convergence.
problem Ensuring reliable performance in stochastic RL problems with risk-averse policies.
method Developed natural policy gradient updates for ECRMs-based RL problems, proving global optimality and iteration complexity.
result Global convergence of risk-averse NPG algorithm with ECRMs.
We develop a framework for interacting with uncertain environments in reinforcement learning (RL) by leveraging preferences in the form of utility functions. We claim that there is value in considering different risk measures during learning. In this framework, the preference for risk can be tuned by variation of the p…
We study risk-sensitive imitation learning where the agent's goal is to perform at least as well as the expert in terms of a risk profile. We first formulate our risk-sensitive imitation learning setting. We consider the generative adversarial approach to imitation learning (GAIL) and derive an optimization problem for…
We establish a stochastic maximum principle (SMP) for control problems of partially observed diffusions of mean-field type with risk-sensitive performance functionals.
Separates estimation and control in risk-sensitive investment problems with partial observation.
problem Risk-sensitive investment problems with incomplete observation.
method Investigates separability of a general class of risk-sensitive investment management problems using a finite-dimensional filter.
result The separated problem is strictly equivalent to the original control problem.
Novel approach to Bayesian experimental design for non-exchangeable data.
problem Optimal experimental design for non-exchangeable data.
method Inside-Out SMC2 algorithm embedded in particle Markov chain Monte Carlo framework. result Efficacy demonstrated on a set of dynamical systems.
Paper tackles risk-sensitive decision-making under uncertainty.
problem Risk-sensitive decision-making problem under uncertainty.
method Formulated as a stochastic control problem, delineated necessary optimality conditions.
result Illustrative examples from optimal betting and inventory management support the theory.