Paper introduces control variate to reduce variance in off-policy reinforcement learning.
problem High variance in off-policy reinforcement learning.
method Introduces control variate technique to Expected Sarsa(λ) and proposes ES(λ)-CV algorithm.
result Proposed ES(λ)-CV algorithm has lower variance than Expected Sarsa(λ).
Paper extends GP-SARSA for more efficient reinforcement learning.
problem Efficient reinforcement learning with Temporal Difference methods.
method Derives recursive formulas for a variant of GP-SARSA.
result Promotes greater memory efficiency and multiple timescale updates.
The paper analyzes SARSA with linear function approximation, providing finite-sample error bounds.
problem Finite-sample analysis of SARSA with linear function approximation under non-i.i.d. data.
method Developed a novel technique to characterize stochastic bias of SARSA with time-varying kernels, enabling finite-sample convergence analysis.
result Provided finite-sample analysis on mean square error of SARSA and fitted SARSA algorithms.
SS-SARSA tackles recovering bandits by treating rounds as states.
problem Real-world rewards vary with the elapsed rounds since last pull.
method Proposes SS-SARSA, a new RL algorithm treating rounds as states.
result Achieves efficient learning with asymptotic convergence to optimal policy.
EPG unifies SPG and DPG for reinforcement learning.
problem Improving reinforcement learning algorithms for policy optimization.
method EPG integrates across actions for gradient estimation, using practical results for Gaussian policies and extending to broader classes of policies.
result EPG reduces gradient variance without deterministic policies and with minimal overhead.
Implicit Q-learning and SARSA adjust step-sizes automatically, improving stability and performance.
problem Numerical instability and slow progress in Q-learning and SARSA due to step-size calibration.
method Reformulate iterative updates as fixed-point equations, scaling step-sizes inversely with feature norms.
result Implicit methods maintain stability over broader step-size ranges and achieve comparable convergence rates.
Swift-Sarsa combines TD learning with Sarsa to control tasks robustly.
problem Learning effective control policies from noisy signals.
method Combines True Online Sarsa(λ) with step-size optimization and decay. result Swift-Sarsa learns relevant signals without prior knowledge.
FedSARSA converges with heterogeneous agents, achieving linear speed-up.
problem Convergence analysis of Federated SARSA with heterogeneous agents.
method Linear function approximation, local training, multi-step error expansion.
result FedSARSA achieves linear speed-up with respect to the number of agents.
Unified algorithm for reinforcement learning with function approximation.
problem Limited scalability of Q(σ,λ) for large-scale learning.
method Proposes GQ(σ,λ) with linear function approximation to extend tabular Q(σ,λ).
result Empirical results show GQ(σ,λ) outperforms full-sampling and pure-expectation methods.
A new method for marine robot navigation using sparse Gaussian Process TD learning.
problem Data efficiency and online prediction challenges in marine robot navigation.
method Sparse Gaussian Process Temporal Difference (SPGP-TD) learning with a sparse approximation.
result SPGP-SARSA outperforms state-of-the-art sparse methods and replicates exact prediction quality.
This study evaluates multi-step reinforcement learning methods in the Mountain Car environment.
problem Lack of statistical significance in evaluating multi-step reinforcement learning methods.
method Combines n-step action-value algorithms with DQN architecture, tests in Mountain Car environment.
result Performance varies with off-policy correction, backup length, and target network update frequency.
EPG unifies SPG and DPG, reducing variance and improving exploration.
problem Improving policy gradient methods for reinforcement learning.
method EPG integrates across actions for gradient estimation, reducing variance.
result EPG reduces variance without requiring deterministic policies.
This paper extends Q(σ) to Q(σ, λ) and Double Q(σ), improving reinforcement learning control.
problem Improving reinforcement learning control methods.
method Extends Q(σ) to Q(σ, λ) using eligibility traces and introduces Double Q(σ).
result The new Q(σ, λ) algorithm outperforms classical TD control methods.
Study evaluates reinforcement learning for trading diverse stocks, finds Q-learning outperforms.
problem Evaluating reinforcement learning for trading diverse stocks.
method Implemented Value Iteration (VI), State-action-reward-state-action (SARSA), and Q-Learning on a diverse stock portfolio dataset.
result Q-learning performs better than VI and SARSA during testing, but performance varies based on market conditions.
A new approach to reinforcement learning by adjusting stepsize based on state criticality.
problem The bias-variance trade-off in n-step reinforcement learning algorithms.
method Adapting the stepsize (n) for each state based on a human trainer's criticality measure.
result Improved reinforcement learning performance by dynamically adjusting the stepsize.
Paper proposes a new RL approach combining IL and RL methods to improve decision-making.
problem Challenges in RL with large state and action spaces, and difficulty in reward determination.
method Combines Imitation Learning and RL methods (SARSA and A3C) to learn sequential decision-making policies.
result Significantly decreases human effort and exploration time in learning decision-making policies.
A new softmax operator improves reinforcement learning algorithms.
problem Softmax operators can misbehave in reinforcement learning, leading to suboptimal policies.
method Developed a differentiable softmax operator and introduced a SARSA variant using it.
result The new algorithm converges and performs well in practice.
TDprop uses Jacobi preconditioning to improve adaptive optimizers in Deep RL.
problem Improving performance of adaptive optimizers in Deep RL.
method TDprop computes per-parameter learning rates based on Jacobi preconditioning of the TD update rule.
result TDprop matches or exceeds Adam's performance in Deep RL experiments, suggesting Jacobi preconditioning can improve adaptive methods.
Paper presents an energy-efficient RL method for sensor networks.
problem Energy consumption in sensor networks for health monitoring.
method Adaptive Reinforcement Learning framework using SARSA algorithm.
result Achieves performance enhancement and energy savings over time.
Unified algorithm Q(σ,λ) combines eligibility traces to improve memory and computation efficiency.
problem Memory and computation inefficiency in multi-step temporal-difference learning.
method Introduced Q(σ) algorithm, combined with eligibility traces to propose Q(σ,λ). result Proposed algorithm Q(σ,λ) converges to optimal value function exponentially. New RL algorithm learns state aggregation architecture adaptively.
problem Adapting reinforcement learning value function architectures.
method Adapts state aggregation architecture using state visit frequency feedback.
result Improves RL performance on various test problems.
We propose a reinforcement learning solution to the \emph{soccer dribbling task}, a scenario in which a soccer agent has to go from the beginning to the end of a region keeping possession of the ball, as an adversary attempts to gain possession. While the adversary uses a stationary policy, the dribbler learns the best…
New algorithms for n-player games using a player-centered approach.
problem Learning near-perfect strategies in multiplayer games.
method Player-centered reinforcement learning with Final Adaptation RL.
result FARL is crucial for achieving near-perfect strategies in various games.
This paper examines methods to estimate uncertainty in neural networks for dialogue policy optimization.
problem Efficient exploration in neural network-based dialogue policy optimization.
method Extensive benchmark of deep Bayesian methods to extract uncertainty estimates from DQN.
result Combining uncertainty estimation methods with DQN improves sample efficiency and user experience.
Study reinforcement learning with model misspecification, extending robust MDPs.
problem Reinforcement learning under model misspecification.
method Extending robust MDPs to model-free RL, defining robust Q-learning, SARSA, and TD-learning, proving convergence.
result Convergence to approximately optimal robust policy and value function.
This paper proposes a benchmarking environment for RL-based dialogue management.
problem Lack of a common benchmarking framework for RL models in dialogue management.
method Developed a set of simulated environments and compared RL models (DQN, A2C, Natural Actor-Critic, GP-SARSA).
result Demonstrated the effectiveness of various RL models in dialogue management.
A two-stream reinforcement learning model improves decision-making across human and neuropsychiatric studies.
problem Improving reinforcement learning models to better simulate human decision-making and neuropsychiatric conditions.
method Proposes a two-stream reinforcement learning model that processes positive and negative rewards and incorporates reward-processing biases.
result The two-stream model outperforms standard Q-learning and SARSA methods on various tasks and datasets.
Transformers can implement reinforcement learning algorithms from data without updates.
problem Training reinforcement learning algorithms from data without parameter updates.
method Design a teacher-mimicking training procedure for transformers to implement policy-improvement methods.
result Gradient flow converges to an optimal parameter manifold corresponding to the desired RL update.
Tabular Q-learning outperforms advanced RL methods in monetary policy.
problem Dynamic setting of short-term interest rates to stabilize inflation and unemployment under uncertain macroeconomic conditions.
method Discrete-action Markov Decision Process with tabular Q-learning, SARSA, Actor-Critic, Deep Q-Networks, Bayesian Q-learning, POMDP formulations.
result Standard tabular Q-learning achieved the best performance (-615.13 +- 309.58 mean return) compared to advanced RL methods and traditional policy rules.
Projective simulation outperforms standard reinforcement learning in navigation tasks.
problem Benchmarking projective simulation in navigation problems.
method Projective simulation model applied to grid world and mountain car problems, compared to Q-learning and SARSA.
result Projective simulation outperforms standard reinforcement learning in terms of parameter simplicity and computational efficiency.
Continuous-time Q-learning theory developed for reinforcement learning.
problem Continuous-time reinforcement learning challenges.
method Entropy-regularized, exploratory diffusion process formulation; first-order approximation of Q-function; martingale conditions.
result Developed a q-learning theory independent of time discretization.
Develops hyperfinite G-expectation theory for continuous-time processes.
problem Creating a discrete model for continuous-time G-expectation. method Introduces hyperfinite G-expectation and develops its theory, proving existence of liftings. result Establishes existence theorem for liftings of continuous-time G-expectation. Survey finds LLMs match human economic expectations closely.
problem Understanding human economic expectations and their deviations.
method Survey of LLM's expectations based on news articles.
result LLM's expectations closely match existing surveys and exhibit deviations.
This paper proves Expected Shortfall is concave, not convex.
problem Understanding the convexity/concavity of Expected Shortfall.
method Analytical proof of concavity with respect to probability distributions.
result Expected Shortfall is concave, not convex.
Defines g-expectation of distributions and its applications.
problem Defining g-expectation of distributions. method Two special cases of nonlinear g and law-invariant g-expectation. result Explicit derivation of g-expectation of distributions. Introduces a new conditional expectation under distorted probabilities, addressing time-inconsistency.
problem Time-inconsistency in nonlinear expectations under probability distortion.
method Localizes probability distortion and constructs a time-consistent conditional expectation.
result Constructs a conditional expectation that is time-consistent and corresponds to a parabolic differential equation.
Choquet and minimax expectations are equivalent in European option pricing.
problem Pricing European options in incomplete markets.
method Comparing Choquet and minimax expectations in the context of European options.
result Choquet and minimax expectations are equal for European options.
Active inference minimizes expected free energy for optimal behavior.
problem Understanding and optimizing behavior in complex systems.
method Combines Bayesian decision theory, optimal Bayesian design, and the free energy principle.
result Active inference emerges as a unified framework for information-seeking, utility maximization, and goal-directed behavior.
Extends sublinear expectations to random sets, identifying extremal and constructing methods.
problem Extending sublinear expectations to random sets.
method Identifying extremal expectations and presenting general construction methods.
result Identification of extremal sublinear and superlinear expectations.
Study on expectile and expected shortfall for tail risk assessment.
problem Comparing expectile and expected shortfall for tail risk assessment.
method Duality results and optimized certainty equivalent.
result Derived bounds and asymptotic behavior of expectile with respect to expected shortfall.
We provide a general construction of time-consistent sublinear expectations on the space of continuous paths. It yields the existence of the conditional G-expectation of a Borel-measurable (rather than quasi-continuous) random variable, a generalization of the random G-expectation, and an optional sampling theorem that…
Paper introduces EEMs for pricing contingent claim returns.
problem Computing expected future prices of contingent claims.
method Dynamic change of measure approach to construct EEMs.
result EEMs provide physical and pricing expectations of contingent claim prices.
New method for predicting portfolio dynamics using non-Euclidean geometry.
problem Predicting efficient portfolios with geometric structure.
method Non-Euclidean conditional expectation and filtering equations.
result Accurate numerical forecasts of portfolio dynamics.
Study on expected critical points of real Lefschetz pencils.
problem Counting critical points in real Lefschetz pencils.
method Asymptotic probabilistic real Riemann-Hurwitz formula.
result Asymptotic expected number and distribution of critical points.
Paper proposes using expectation models for planning in stochastic environments.
problem Intractability of learning distribution and sample models in large state and action spaces.
method Proposes using approximate expectation models for MBRL, analyzes linear and non-linear parametrizations, and presents a policy evaluation algorithm.
result Planning with an expectation model is equivalent to planning with a distribution model under certain conditions.
Dual representation and properties of expectile-based expected shortfall studied.
problem Studying the expectile-based expected shortfall as a risk measure.
method Provided dual representation in terms of Bochner integral, showed boundedness properties, and computed for selected distributions.
result Explicit dual representation and boundedness properties of expectile-based expected shortfall.
The paper proposes a new risk measure, Expected Downside Risk, to explain risk-preference.
problem Contradictory empirical findings between risk and reward.
method Introducing Expected Downside Risk (EDR) as a new risk measure.
result EDR better explains investors' utility perception and can model both positive and negative risk-reward relationships.
We refine Expected Shortfall by controlling different tail portions, offering tailored risk assessments.
problem Risk assessment in financial positions, especially in tail regions.
method Introducing adjusted Expected Shortfall measures that control different tail portions.
result Adjusted Expected Shortfall measures ensure risk does not exceed specified thresholds for various probability levels.