A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
This work analyzes the value of future reward information in RL.
We propose RUDDER, a novel reinforcement learning approach for delayed rewards in finite Markov decision processes (MDPs). In MDPs the Q-values are equal to the expected immediate reward plus the expected future rewards. The latter are related to bias problems in temporal difference (TD) learning and to high variance p…
New RL approach uses future state and action visitation measures for better exploration.
Enhances portfolio performance using deep reinforcement learning and future rewards.
Value functions are crucial for model-free Reinforcement Learning (RL) to obtain a policy implicitly or guide the policy updates. Value estimation heavily depends on the stochasticity of environmental dynamics and the quality of reward signals. In this paper, we propose a two-step understanding of value estimation from…
Method predicts future rewards from past actions in a linear Gaussian system.
This work simplifies IRL by using potential-based reward shaping.
Paper simplifies complex AI exploration by predicting future rewards.
We introduce a method by which a generative model learning the joint distribution between actions and future states can be used to automatically infer a control scheme for any desired reward function, which may be altered on the fly without retraining the model. In this method, the problem of action selection is reduce…
The paper introduces a new intrinsic reward method for exploration in reinforcement learning.
In reinforcement learning the Q-values summarize the expected future rewards that the agent will attain. However, they cannot capture the epistemic uncertainty about those rewards. In this work we derive a new Bellman operator with associated fixed point we call the `knowledge values'. These K-values compress both the …
There exist a number of reinforcement learning algorithms which learnby climbing the gradient of expected reward. Their long-runconvergence has been proved, even in partially observableenvironments with non-deterministic actions, and without the need fora system model. However, the variance of the gradient estimator ha…
Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.
We develop a theory for solving continuous time optimal stopping problems for non-linear expectations. Our motivation is to consider problems in which the stopper uses risk measures to evaluate future rewards.
Learning robust value functions given raw observations and rewards is now possible with model-free and model-based deep reinforcement learning algorithms. There is a third alternative, called Successor Representations (SR), which decomposes the value function into two components -- a reward predictor and a successor ma…
Curiosity enhanced by audio-visual associations improves learning efficiency.
A minimalist approach improves LLM reasoning by filtering incorrect responses.
We explore a novel setting of the Multi-Armed Bandit (MAB) problem inspired from real world applications which we call bandits with "stochastic delayed composite anonymous feedback (SDCAF)". In SDCAF, the rewards on pulling arms are stochastic with respect to time but spread over a fixed number of time steps in the fut…
This work presents a game-theoretic method for AVs that handles imperfect communication and individual rewards.
Off-policy evaluation for MNAR rewards in MDPs
Reinforcement learning with sparse rewards is still an open challenge. Classic methods rely on getting feedback via extrinsic rewards to train the agent, and in situations where this occurs very rarely the agent learns slowly or cannot learn at all. Similarly, if the agent receives also rewards that create suboptimal m…
Reinforcement learning (RL) algorithms allow artificial agents to improve their selection of actions to increase rewarding experiences in their environments. Temporal Difference (TD) Learning -- a model-free RL method -- is a leading account of the midbrain dopamine system and the basal ganglia in reinforcement learnin…
This work frames reward modelling from preferences as a causal problem.
Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic r…
The design of a reward function often poses a major practical challenge to real-world applications of reinforcement learning. Approaches such as inverse reinforcement learning attempt to overcome this challenge, but require expert demonstrations, which can be difficult or expensive to obtain in practice. We propose var…
New algorithm for multi-armed bandits with delayed, partially observed rewards.
In reinforcement learning, we can learn a model of future observations and rewards, and use it to plan the agent's next actions. However, jointly modeling future observations can be computationally expensive or even intractable if the observations are high-dimensional (e.g. images). For this reason, previous works have…
COCOA improves credit assignment in reinforcement learning by measuring contributions to rewards.
Inverse reinforcement learning (IRL) is used to infer the reward function from the actions of an expert running a Markov Decision Process (MDP). A novel approach using variational inference for learning the reward function is proposed in this research. Using this technique, the intractable posterior distribution of the…
CRB tackles rising rewards in combinatorial online learning.
Learning in sparse reward settings remains a challenge in Reinforcement Learning, which is often addressed by using intrinsic rewards. One promising strategy is inspired by human curiosity, requiring the agent to learn to predict the future. In this paper a curiosity-driven agent is extended to use these predictions di…
In reinforcement learning, Return, which is the weighted accumulated future rewards, and Value, which is the expected return, serve as the objective that guides the learning of the policy. In classic RL, return is defined as the exponentially discounted sum of future rewards. One key insight is that there could be many…
New RL agent learns sparse rewards efficiently.
Enhances RL agents with predictive internal representations.
Active inference enhances RL by balancing exploration and exploitation.
Motivated by applications of bandit algorithms in education, we consider a stochastic multi-armed bandit problem with -contaminated rewards. We allow an adversary to give arbitrary unbounded contaminated rewards with full knowledge of the past and future. We impose the constraint that for each time the…
Algorithm learns optimal coordination for strategic agents in uncertain settings.
Policy gradient algorithms typically combine discounted future rewards with an estimated value function, to compute the direction and magnitude of parameter updates. However, for most Reinforcement Learning tasks, humans can provide additional insight to constrain the policy learning. We introduce a general method to i…
New approach categorizes objective functions for embodied agents.
Automated algorithm selection and hyperparameter tuning facilitates the application of machine learning. Traditional multi-armed bandit strategies look to the history of observed rewards to identify the most promising arms for optimizing expected total reward in the long run. When considering limited time budgets and c…
In light of the growing interest in agent-based market models, we bring together several earlier works in which we considered the topic of self-consistent market modelling. Building upon the binary game structure of Challet and Zhang, we discuss generalizations of the strategy reward scheme such that the agents seek to…
Method uses ANN to estimate incentive salience from large behavioral data.
Reinforcement learning algorithms use correlations between policies and rewards to improve agent performance. But in dynamic or sparsely rewarding environments these correlations are often too small, or rewarding events are too infrequent to make learning feasible. Human education instead relies on curricula--the break…
Quantum model outperforms classical in training but underperforms in real-world metrics.
Optimizes forecast distributions for financial risk management.
In classical Q-learning, the objective is to maximize the sum of discounted rewards through iteratively using the Bellman equation as an update, in an attempt to estimate the action value function of the optimal policy. Conventionally, the loss function is defined as the temporal difference between the action value and…
Prognosticator improves performance in non-stationary MDPs.
We use martingale and stochastic analysis techniques to study a continuous-time optimal stopping problem, in which the decision maker uses a dynamic convex risk measure to evaluate future rewards. We also find a saddle point for an equivalent zero-sum game of control and stopping, between an agent (the "stopper") who c…