Paper proposes a new risk measure (reward volatility) for optimizing financial decisions.
problem Managing uncertainty and volatility in financial decision-making.
method Defines reward volatility, derives policy gradient theorem, develops actor-only algorithm.
result Risk-averse optimization improves both reward volatility and return variance.
New method identifies uncertainty shocks in financial markets using revised VIX.
problem Traditional VIX fails to capture non-Gaussian, heavy-tailed asset returns.
method Fit a double-subordinated Normal Inverse Gaussian Levy process to S&P 500 option prices to construct a revised VIX.
result Revised VIX provides a more comprehensive measure of volatility reflecting extreme movements and heavy tails.
We implement momentum strategies using reward-risk measures as ranking criteria based on classical tempered stable distribution. Performances and risk characteristics for the alternative portfolios are obtained in various asset classes and markets. The reward-risk momentum strategies with lower volatility levels outper…
Machine learning improves portfolio allocation between index and risk-free assets.
problem Finding optimal portfolio rules for time-varying returns and volatility.
method Two Random Forest models: one for sign probabilities of excess return, the other for optimized volatility.
result Substantial improvements in utility, risk-adjusted returns, and maximum drawdowns over buy-and-hold.
We here adopt Bayesian nonparametric mixture models to extend multi-armed bandits in general, and Thompson sampling in particular, to scenarios where there is reward model uncertainty. In the stochastic multi-armed bandit, the reward for the played arm is generated from an unknown distribution. Reward uncertainty, i.e.…
This study examines deep hedging for S&P 500 options, revealing systematic delta corrections and fragility.
problem Understanding and validating deep hedging strategies for financial options.
method Compared TD3 agents with a Black-Scholes delta hedge, using walk-forward tests and symbolic regression.
result Deep hedging agents learn systematic delta corrections, which can improve performance but are regime-fragile.
Paper uses DRL to optimize portfolios, balancing risk and return.
problem Optimizing portfolios under market uncertainty and risk constraints.
method Integrates Sharpe ratio-based reward with risk control mechanisms, uses PPO for adaptive asset allocation.
result DRL agent stabilizes volatility but sacrifices risk-adjusted returns.
Adversarial RL recovers agent rewards from financial market data simulations.
problem Recovering agent rewards in volatile financial markets with unknown dynamics.
method Adversarial inverse reinforcement learning in latent space simulations.
result Adversarial RL can robustly recover agent rewards from latent space representations of real market data.
Deep RL strategies outperform classical models in trading.
problem Designing profitable trading strategies for futures contracts.
method Deep Reinforcement Learning with volatility scaling.
result Deep RL algorithms outperformed classical models with positive profits.
A new framework for risk-aware multi-armed bandits tackles volatile environments.
problem Volatility in healthcare and finance makes naive reward maximization unreliable.
method Risk-aware strategies with adaptive risk measures and change-point detection.
result Finite-time theoretical guarantees and asymptotic regret bound of order i l d e O ( K T T ) ilde O(\sqrt{K_T T}) i l d e O ( K T T ) . Quantum model outperforms classical in training but underperforms in real-world metrics.
problem Mismatch between proxy reward signals and true investment objectives in financial domains.
method Hybrid quantum-classical reinforcement learning framework with automated feature engineering.
result Quantum models achieve higher training rewards but underperform in real-world metrics.
Deep Q-Learning system for straddle options in volatile markets.
problem High computational costs and unstable performance in high-volatility markets.
method Attention mechanisms in Transformer-DDQN, novel reward function, and resistance level identification.
result Transformer-DDQN model exhibits lowest maximum drawdown and highest average return.
We reveal an interesting convex duality relationship between two problems: (a) minimizing the probability of lifetime ruin when the rate of consumption is stochastic and when the individual can invest in a Black-Scholes financial market; (b) a controller-and-stopper problem, in which the controller controls the drift a…
RL enhances cryptocurrency trading profits.
problem Enhancing cryptocurrency trading profits through dynamic scaling.
method Combining RL with pair trading, using new reward shaping and observation/action spaces.
result RL-based trading achieved 9.94% to 31.53% annualized profits, vs. 8.33% for traditional methods.
Study proposes adaptive RL for dynamic portfolio optimization.
problem Traditional portfolio optimization models fail to adapt to regime shifts.
method Regime-aware reinforcement learning framework with hybrid observations and constrained reward functions.
result Transformer PPO achieves highest risk-adjusted returns, while LSTM variants offer a good balance.
Algorithm learns diffusion processes with high-dimensional state spaces.
problem Stochastic control of unbounded diffusion processes with high-dimensional state spaces.
method Adaptive partitioning and learning algorithm that refines discretization based on estimation bias and statistical confidence.
result Established regret bounds that depend on problem parameters, extending to unbounded diffusion processes.
Study on RL on volatility surfaces, proving no free lunch for law-seeking methods.
problem Aligning RL agents with no-arbitrage laws in volatile markets.
method Built a law manifold, defined penalties, and used a Goodhart decomposition.
result No free lunch theorem: Law-seeking RL cannot outperform baselines.
How effective are the most common trading models? The answer may help investors realize upsides to using each model, act as a segue for investors into more complex financial analysis and machine learning, and to increase financial literacy amongst students. Creating original versions of popular models, like linear regr…
Study uses RL to optimize crypto portfolios with two-sided transactions and lending.
problem Managing downside risk and capital optimization in high-risk crypto markets.
method Integrates RL with a new environmental formulation and PnL-based reward function, using SAC agent with CNN-MHA.
result Significantly outperforms benchmarks, especially in high-volatility scenarios.
Although Bitcoin has long been dominant in the crypto scene, it is certainly not alone. Ether is another cryptocurrency related project that has attracted an intensive attention because of its additional features. This study seeks to test whether these cryptocurrencies differ in terms of their volatile and speculative …
We propose a probabilistic framework for pricing derivatives, which acknowledges that information and beliefs are subjective. Market prices can be translated into implied probabilities. In particular, futures imply returns for these implied probability distributions. We argue that volatility is not risk, but uncertaint…
New risk metric for RL in finance considers time splits of returns.
problem Optimizing financial decisions with a balance between return and risk.
method Developed a new risk metric for reinforcement learning that allows for flexible target levels of rewards over time.
result Proposed risk metric optimizes for arbitrary time splits of returns, improving upon classical risk measures.
The paper examines stability of shares in Proof of Stake protocol, identifying different investor behaviors and phase transitions.
problem Stability of shares in Proof of Stake protocol.
method Identification of large, medium, and small investors under various rewarding schemes; dynamical population model analysis.
result Phase transitions and thresholds for stability are characterized; chaotic centralization leads to concentration of shares.
Optimizes forecast distributions for financial risk management.
problem Improving risk management through better forecast distributions.
method Optimizes forecast distributions using scoring rules relevant to financial risk management.
result Tail-focused predictive distributions yield better outcomes in hedging strategies involving VIX futures.
Study uses RL to optimize risky vs. risk-free asset allocation.
problem Optimal asset allocation in volatile financial markets.
method Formulated as MDP, uses DDPG with TiDE for dynamic policy learning.
result DDPG-TiDE outperforms Q-learning and buy-and-hold strategies.
A RL framework for hedging equity index options with realistic costs.
problem Dynamic hedging of equity index option exposures under transaction costs.
method Reinforcement Learning (RL) with a leak-free environment, cost-aware reward function, and stochastic actor-critic agent.
result The RL policy improves risk-adjusted performance compared to no-hedge, momentum, and volatility-targeting baselines.
Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.
problem Reward hacking in RL models exploiting reward misspecifications.
method Constructed four RL environments with misspecified rewards; analyzed agent capabilities and behavior.
result More capable agents exploit reward misspecifications, achieving higher proxy reward but lower true reward.
Paper introduces PRMs to learn non-Markovian stochastic rewards for reinforcement learning.
problem Lack of structured representation for non-Markovian stochastic rewards in reinforcement learning.
method Introduces probabilistic reward machines (PRMs) and presents an algorithm to learn them from decision processes.
result Algorithm proves correct and convergent for learning PRMs from decision processes.
Paper addresses reward learning issues in RL, improving both under- and over-estimation.
problem Reward learning from data can lead to reward delusions or underestimation, causing unintended behaviors.
method Connects reward learning to positive-unlabeled (PU) learning and applies a large-scale PU learning algorithm.
result Improves both GAIL and supervised reward learning without additional assumptions.
This work analyzes the value of future reward information in RL.
problem Analyzing the impact of knowing future rewards in reinforcement learning.
method Competitive analysis and worst-case reward distribution.
result Exact ratios between standard RL agents and those with future-reward lookahead.
Enhanced portfolio optimization for a financial goal with G-Learning and GIRL algorithms.
problem Maximize portfolio value while minimizing periodic contributions by a target date in volatile markets.
method Combines G-Learning and GIRL algorithms for dynamic portfolio optimization.
result Improved Sharpe Ratio from 0.42 to 0.483, demonstrating robust optimization in volatile markets.
Self-supervised reward prediction improves RL in sparse reward settings.
problem Data efficiency and sparse reward signals in reinforcement learning.
method Learning a state representation for reward prediction and using it to shape rewards.
result Self-supervised reward prediction enhances RL algorithms in single-goal environments.
The study categorizes reward errors in reinforcement learning, finding some can be beneficial.
problem Training language models with imperfect proxy rewards.
method Theoretical analysis of policy gradient optimization and categorization of reward errors.
result Reward errors can be benign or even beneficial, preventing policy from stalling.
Reward collapse occurs when ranking-based reward models yield uniform rewards for different prompts.
problem Reward collapse in aligning large language models with human preferences.
method Introduced a prompt-aware optimization scheme to derive closed-form expressions for reward distributions.
result Our prompt-aware utility functions significantly alleviate reward collapse during training.
Reward models need more than just accuracy for effective RLHF.
problem The effectiveness of reward models in RLHF is not fully understood.
method An optimization perspective to evaluate reward models.
result Reward models with low reward variance can lead to a flat optimization landscape, hindering performance.
Proposes a method to boost deep reinforcement learning with sparse rewards.
problem Challenges in learning complex behaviors with long horizons and sparse rewards.
method Predictive coding for reward shaping.
result Achieves better learning by providing reward signals that understand environment dynamics and emphasize useful features.
Action guidance helps agents learn true objectives in games with sparse rewards.
problem Training agents in games with sparse rewards requires significant exploration.
method Action guidance, a novel technique that combines exploration with reward shaping.
result Action guidance enables agents to optimize true objectives efficiently.
New RL method uses distance between states instead of rewards for sparse reward environments.
problem Sparse rewards or non-reward environments in reinforcement learning.
method Uses goal-distance gradient and bridge point planning for policy improvement.
result Significantly better performance on sparse reward and local optimal problems in complex environments.
Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.
problem Learning from sparse and delayed rewards in reinforcement learning.
method Randomized Return Decomposition (RRD) algorithm to redistribute rewards.
result Substantial improvement over baseline algorithms in experiments.
Enhances reward specification in RL with a novel language-based approach.
problem Reward specification in RL can lead to unintended, potentially harmful behaviours.
method Developed a novel class of language-based Reward Machines using RML's built-in memory.
result Can specify non-regular, non-Markovian reward functions for complex tasks.
Reward tweaking optimizes behavior for long-term goals by adjusting the reward function.
problem Optimizing behavior for long-term goals in reinforcement learning with unstable long planning horizons.
method Reward tweaking learns a surrogate reward function that induces optimal behavior for the original task.
result Reward tweaking guides agents towards better long-term returns while planning for short horizons.
We propose a generic, Bayesian, information geometric approach to the exploration--exploitation trade-off in multi-armed bandit problems. Our approach, BelMan, uniformly supports pure exploration, exploration--exploitation, and two-phase bandit problems. The knowledge on bandit arms and their reward distributions is su…
Many reinforcement-learning researchers treat the reward function as a part of the environment, meaning that the agent can only know the reward of a state if it encounters that state in a trial run. However, we argue that this is an unnecessary limitation and instead, the reward function should be provided to the learn…
In this paper, we present a simple stock market model (the market game) which incorporates, as ab initio dynamics delayed majority dynamics, according to which agents (with heterogeneous strategies and price expectations) are rewarded if their actions at time t are the actions of the majority of agents at time t+1. We …
Extends reinforcement learning alignment to scalar rewards, improving math reasoning.
problem Designing reinforcement learning algorithms for general LLM alignment.
method Introduces f-GRPO and f-HAL, estimating f-divergences between reward-aligned and unaligned distributions.
result Improves math reasoning RLVR tasks and mitigates reward hacking.
Exact relationships found between ATM slope, volatility swap, and zero vanna.
problem Understanding relationships between implied volatilities and swaps.
method Analyzes exact relationships between ATM slope, volatility swap, and zero vanna.
result Exact relationships between ATM slope, volatility swap, and zero vanna.
Study local volatility from rough volatility models, finding new skew rule.
problem Understanding local volatility from rough volatility models.
method Analyzing asymptotic behavior of local volatility surface generated by rough stochastic volatility models.
result New skew rule: ratio of implied and local vol skews tends to 1/(H + 3/2).
This work characterizes reward function partial identifiability and its impact on policy optimization.
problem Reward function partial identifiability in complex tasks.
method Formal characterisation of partial identifiability using various reward learning data sources.
result Unified framework for comparing data sources and downstream tasks by their invariances.