Policy gradient methods ignore the potential value of adjusting environment variables: unobservable state features that are randomly determined by the environment in a physical setting, but are controllable in a simulator. This can lead to slow learning, or convergence to suboptimal policies, if the environment variabl…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Policy Optimisation”
Paper shows equivalence between two alignment methods and introduces a new algorithm.
problem Ensuring human alignment of large language models for useful, safe, and pleasant user experience.
method Introduces IPO-MD algorithm, showing equivalence between IPO and Nash-MD methods.
result Equivalence between IPO and Nash-MD methods proven when considering online version of IPO.
Bayesian optimisation has been successfully applied to a variety of reinforcement learning problems. However, the traditional approach for learning optimal policies in simulators does not utilise the opportunity to improve learning by adjusting certain environment variables: state features that are unobservable and ran…
In the automation of many kinds of processes, the observable outcome can often be described as the combined effect of an entire sequence of actions, or controls, applied throughout its execution. In these cases, strategies to optimise control policies for individual stages of the process might not be applicable, and in…
We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are com…
Deep RL algorithms struggle with noisy rewards in portfolio optimisation.
problem Evaluating deep reinforcement learning for portfolio optimisation with market impact.
method Simulated data with geometric Brownian motion and market impact model; Kelly criterion as upper bound; PPO and A2C with GAE; clipping; hidden Markov model for regime changes.
result PPO and A2C with GAE perform better with noisy rewards; PPO with HMM learns different policies for regime changes.
Building upon the recent success of deep reinforcement learning methods, we investigate the possibility of on-policy reinforcement learning improvement by reusing the data from several consecutive policies. On-policy methods bring many benefits, such as ability to evaluate each resulting policy. However, they usually d…
POLAR learns efficient data acquisition policies using pretrained belief representations.
problem Challenges in learning effective policies for adaptive data acquisition.
method POLAR decouples representation learning from policy learning by leveraging pretrained predictive foundation models as belief-state encoders.
result POLAR outperforms state-of-the-art methods across diverse tasks while requiring fewer training samples.
A risk-aware RL approach using RDEU and Wasserstein ball for robust performance.
problem Optimizing risk-aware performance criteria in uncertain environments.
method Rank dependent expected utility (RDEU) for risk assessment, Wasserstein ball for robustness, actor/agent framework.
result Explicit policy gradient formulae for robust optimization.
This work characterizes reward function partial identifiability and its impact on policy optimization.
problem Reward function partial identifiability in complex tasks.
method Formal characterisation of partial identifiability using various reward learning data sources.
result Unified framework for comparing data sources and downstream tasks by their invariances.
The performance of policy gradient methods is sensitive to hyperparameter settings that must be tuned for any new application. Widely used grid search methods for tuning hyperparameters are sample inefficient and computationally expensive. More advanced methods like Population Based Training that learn optimal schedule…
In statistical dialogue management, the dialogue manager learns a policy that maps a belief state to an action for the system to perform. Efficient exploration is key to successful policy optimisation. Current deep reinforcement learning methods are very promising but rely on epsilon-greedy exploration, thus subjecting…
The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning. This process, referred to as distillation, has been used to great success, for example, by enhancing the optimisation of agents, leading to stronger performance faster, on harder domains [26, 32, 5, 8]. Despite the…
Study optimal dividend and capital injection in insurance portfolios with self-exciting claim arrivals.
problem Optimal dividend and capital injection in insurance portfolios with Hawkes process claim arrivals.
method Analytical properties, explicit threshold, HJB variational inequality, finite-difference scheme, policy-gradient, actor-critic methods.
result Learned strategies closely match the PDE benchmark and remain stable across initial conditions.
FACMAC combines deep policy gradients with factored critic for multi-agent reinforcement learning.
problem Cooperative multi-agent reinforcement learning in discrete and continuous action spaces.
method FACMAC uses a centralised but factored critic, combining per-agent utilities into a joint action-value function.
result FACMAC outperforms MADDPG and other baselines on multi-agent particle environments and StarCraft II tasks.
Applying probabilistic models to reinforcement learning (RL) enables the application of powerful optimisation tools such as variational inference to RL. However, existing inference frameworks and their algorithms pose significant challenges for learning optimal policies, e.g., the absence of mode capturing behaviour in…
Unified reinforcement learning methods using hybrid inference.
problem Combining model-based and model-free reinforcement learning approaches.
method Control as Hybrid Inference (CHI) framework.
result CHI algorithm balances model-based and model-free learning.
We propose a method to optimise the parameters of a policy which will be used to safely perform a given task in a data-efficient manner. We train a Gaussian process model to capture the system dynamics, based on the PILCO framework. Our model has useful analytic properties, which allow closed form computation of error …
We present a data-efficient reinforcement learning algorithm resistant to observation noise. Our method extends the highly data-efficient PILCO algorithm (Deisenroth & Rasmussen, 2011) into partially observed Markov decision processes (POMDPs) by considering the filtering process during policy evaluation. PILCO conduct…
The Strong, Weak and Benign Goodhart's law. An independence-free and paradigm-agnostic formalisationstat.ML
Study on Goodhart's law without independence assumptions, finds new patterns in optimisation.
problem Understanding when and how optimisation of a proxy metric leads to over-optimisation of the intended goal.
method Formalized Goodhart's law without independence and paradigm assumptions, studied different cases of goal and discrepancy tailness.
result Dependence between proxy metric and goal does not change Goodhart's effect for light-tailed cases, but over-optimisation occurs in heavy-tailed discrepancy cases.
IntraLayer connects digital finance platforms efficiently.
problem Inefficient interconnectivity in digital finance.
method Core infrastructure and adaptive fiscal policy.
result Optimised transactional efficiency for various agents.
Optimizes loan recovery timing by forecasting cash flows.
problem Minimizing overall credit loss in loan portfolios.
method Forecast future cash flows using probabilistic and Markov chain models.
result Empirical illustration of loss-optimal recovery timing.
In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimisation task, which attempts to find a policy describing how to respond to humans, in the form of a function taking the current state of the dia…
Action-BED: Task-Driven Bayesian Experimental Design
problem Bayesian experimental design with doubly intractable objectives
method Formulating BED in terms of expected future loss (EFL) and optimising it with stochastic gradients
result Simplified and task-driven framework for BED
The mutual information is a core statistical quantity that has applications in all areas of machine learning, whether this is in training of density models over multiple data modalities, in maximising the efficiency of noisy transmission channels, or when learning behaviour policies for exploration by artificial agents…
Portfolio optimisation under non-linear drawdown constraints in a semimartingale financial modelq-fin.PM
A drawdown constraint forces the current wealth to remain above a given function of its maximum to date. We consider the portfolio optimisation problem of maximising the long-term growth rate of the expected utility of wealth subject to a drawdown constraint, as in the original setup of Grossman and Zhou (1993). We wor…
Bayesian design improves by reducing policy training cost.
problem Double intractability in expected information gain limits policy learning.
method Score matching to isolate EIG, then train policy singly intractably.
result Reduced computational burden for policy training, allowing multiple iterations.
Higher conservative training increases reward-hacking in reasoning models.
problem Reward hacking during online adaptation in reasoning models.
method Conservative offline training with varying levels of conservatism (β) was applied to a Qwen3-14B policy, and online adaptation was measured against a reward ensemble.
result Higher conservatism (β) increases reward-hacking damage, measured by the Goodhart gap and AUGC.
Algorithm optimizes lockdown policies balancing health and economy.
problem Balancing health and economic impacts of lockdowns during pandemics.
method Reinforcement learning to automatically compute lockdown policies.
result Algorithm learns optimal lockdown policies from disease and population data.
We present an off-policy actor-critic algorithm for Reinforcement Learning (RL) that combines ideas from gradient-free optimization via stochastic search with learned action-value function. The result is a simple procedure consisting of three steps: i) policy evaluation by estimating a parametric action-value function;…
Paper speeds up policy optimization for large recommendation systems.
problem Offline optimization of large-scale recommendation systems is computationally expensive.
method Derives an approximation of policy learning algorithms that scales logarithmically with the catalogue size.
result Our algorithm is an order of magnitude faster than naive approaches while producing equally good policies.
Reinforcement learning (RL) has advanced greatly in the past few years with the employment of effective deep neural networks (DNNs) on the policy networks. With the great effectiveness came serious vulnerability issues with DNNs that small adversarial perturbations on the input can change the output of the network. Sev…
Deep RL solves complex macroeconomic models.
problem Solving dynamic stochastic general equilibrium models with bounded rationality.
method Using deep reinforcement learning to model agents as neural networks.
result Artificially intelligent agents can solve models in all policy regimes.
This paper explores optimising acquisition functions in Bayesian optimisation.
problem Optimising acquisition functions in Bayesian optimisation is challenging due to their non-convex nature.
method The authors derive compositional forms for acquisition functions and use them to recast maximisation as a compositional optimisation problem.
result The compositional approach to maximising acquisition functions shows empirical advantages across various tasks.
Optimizes loan recovery timing across various portfolios.
problem Comparing and evaluating bank's loan recovery decision rules.
method Simulation-based expert system considering time value of money and costs.
result Threshold optima exist across different risk scenarios and portfolio compositions.
Deep reinforcement learning is poised to revolutionise the field of AI and represents a step towards building autonomous systems with a higher level understanding of the visual world. Currently, deep learning is enabling reinforcement learning to scale to problems that were previously intractable, such as learning to p…
A new algorithm for deep Q-learning with robustness to state transition uncertainty.
problem Model uncertainty in state transitions for non-tabular, continuous state spaces.
method Distributionally robust approach using worst-case transition ball and dualized Bellman operator with Sinkhorn distance.
result Optimal policy found through solving non-linear Bellman equation with neural network parameterization.
Many advanced Learning from Demonstration (LfD) methods consider the decomposition of complex, real-world tasks into simpler sub-tasks. By reusing the corresponding sub-policies within and between tasks, they provide training data for each policy from different high-level tasks and compose them to perform novel ones. E…
We consider the problem of learning by demonstration from agents acting in unknown stochastic Markov environments or games. Our aim is to estimate agent preferences in order to construct improved policies for the same task that the agents are trying to solve. To do so, we extend previous probabilistic approaches for in…
We consider the problem of learning by demonstration from agents acting in unknown stochastic Markov environments or games. Our aim is to estimate agent preferences in order to construct improved policies for the same task that the agents are trying to solve. To do so, we extend previous probabilistic approaches for in…
Derives a new objective to learn from human preferences without approximations.
problem Learning from human preferences through RLHF relies on approximations that can lead to pitfalls.
method Derives a new general objective PO that bypasses both approximations.
result Demonstrates the superiority of the new objective to Direct Preference Optimisation (DPO) empirically.
User Simulators are one of the major tools that enable offline training of task-oriented dialogue systems. For this task the Agenda-Based User Simulator (ABUS) is often used. The ABUS is based on hand-crafted rules and its output is in semantic form. Issues arise from both properties such as limited diversity and the i…
New framework converts multi-objective to single-objective optimisation.
problem Solving multi-objective optimisation problems.
method Formalises scalarisation into mathematical framework, uses R2 utilities.
result R2 utilities are monotone and submodular, optimised by greedy algorithms.
Unified Bayesian Optimisation for mixed variables improves performance.
problem Efficient optimisation of problems with both categorical and continuous variables.
method Derive value proposals from the Expected Improvement criterion to optimise both categorical and continuous variables under a single acquisition metric.
result Unified approach significantly outperforms existing methods across mixed-variable tasks.
A new method learns priors for Bayesian optimisation to improve performance.
problem Bayesian optimisation tasks often assume strong similarity, which is violated in many cases.
method Replace strong similarity assumption with shape similarity, learn priors for hyperparameters.
result PLeBO and prior transfer find good inputs in fewer evaluations.
Dynamic risk constraints help limit risky behavior in financial portfolios.
problem Static risk measures fail to control tail-risk-seeking traders.
method Introduces dynamic risk constraints applied throughout the trading horizon.
result Dynamic risk constraints can effectively limit risky behavior in portfolios.
Study improves Bayesian optimisation with ensemble transfer learning.
problem Improving sample efficiency in Bayesian optimisation of expensive functions.
method Empirical analysis of ensemble-based transfer learning methods and pipeline components.
result Two components (warm start initialisation and positive weight constraint) improve transfer learning Bayesian optimisation performance.
Meta-gradient RL learns to learn from experience.
problem Learning efficiency and adaptability in reinforcement learning.
method Meta-gradient descent to discover its own objective function.
result Meta-gradient RL adapts to learn more efficiently over time.