Study minimax optimal RL in factored MDPs with bonus exploration.
problem Optimal reinforcement learning in episodic factored MDPs.
method Proposes two model-based algorithms with bonus exploration for minimax optimal regret.
result Achieves minimax optimal regret guarantees for rich factored structures.
Algorithm finds safe zones in policy Markov Decision Processes to limit trajectory escape.
problem Finding safe zones in policy Markov Decision Processes to limit trajectory escape.
method Bi-criteria approximation learning algorithm with polynomial sample complexity.
result Achieves almost 2 approximation for both escape probability and safe zone size.
An efficient method to price bonds with optional sinking feature is presented. Such instruments equip their issuer with the option (but not the obligation) to redeem parts of the notional prior to maturity, therefore the future cash flows are random. In a one-factor model for the issuer's default intensity we show that…
The paper tackles robust policy learning in MDPs using statistical methods.
problem Offline data-driven sequential decision making in MDPs.
method Evaluates policies using average rewards centered at policy-induced stationary distributions. Developed a statistically efficient method for estimating robust optimal policies.
result Established a rate-optimal regret bound up to a logarithmic factor.
New Q-learning method achieves optimal sample complexity for average-reward problems.
problem Challenges in achieving optimal sample complexity for average-reward Q-learning.
method Synchronous and asynchronous Q-learning with a new contraction principle.
result Optimal O ~ ( ε − 2 ) \widetilde{O}(\varepsilon^{-2}) O ( ε − 2 ) sample complexity guarantees. New methods improve temporal difference learning for policy evaluation in Markov decision processes.
problem Improving temporal difference learning for policy evaluation in Markov decision processes.
method Introduced variance-reduced forms of stochastic approximation to achieve non-asymptotic, instance-dependent optimality.
result Temporal difference learning is strictly suboptimal, but variance-reduced forms achieve optimality up to logarithmic factors.
This paper improves MARL for networked systems through new protocols and discount factors.
problem Improving control in networked systems using multi-agent reinforcement learning.
method Formulated as a spatiotemporal Markov decision process, introduced a spatial discount factor, and proposed NeurComm.
result Appropriate spatial discount factor enhances learning curves of non-communicative MARL algorithms.
A new method for risk-averse decision-making in Markov processes with improved regret bounds.
problem Risk-averse decision-making in Markov processes.
method Introduces mini-batch measures and multipattern risk-averse problems in a feature-based Q Q Q -learning method. result Proves a high-probability regret bound of O ( H 2 N H K ) \mathcal{O}\big(H^2 N^H \sqrt{ K}\big) O ( H 2 N H K ) for the Q Q Q -learning method. New algorithm learns efficiently in multi-agent settings.
problem Efficient learning in multi-agent Markov decision processes.
method Cooperative Prioritized Sweeping: model-based reinforcement learning with sample efficiency.
result Outperforms state-of-the-art on SysAdmin and randomized environments.
AlphaCFG discovers alpha factors using grammar-guided search.
problem Discovering formulaic alpha factors in finance.
method AlphaCFG uses a grammar-based framework to define and discover alpha factors with syntactic and semantic constraints.
result AlphaCFG outperforms state-of-the-art methods in trading profitability and efficiency.
Paper tests Markov assumption in sequential decision making.
problem Testing the Markov assumption in sequential decision making.
method Forward-Backward Learning procedure to test MA without assuming parametric forms.
result The proposed test plays a crucial role in identifying optimal policies in complex decision processes.
Style Miner generates stable and significant style factors for time series analysis.
problem Finding significant and stable explanatory factors in high-dimensional time series data.
method Proposes a reinforcement learning method to balance explanatory power and stability constraints.
result Outperforms existing methods by a large margin and achieves a 10% gain in R-squared explanatory power.
New formalism for decision making combines causal structures with MDPs, improving reinforcement learning performance.
problem Sequential decision making with causal knowledge to improve performance.
method Causal Markov Decision Processes (C-MDPs) and C-UCBVI algorithm exploiting causal structure.
result C-UCBVI achieves an i l d e O ( H S Z T ) ilde{O}(HS\sqrt{ZT}) i l d e O ( H S Z T ) regret bound, independent of actions. Model predicts stock prices using GAN and RoI Pooling.
problem Predicting stock prices influenced by macroeconomic factors.
method Markov Decision Process, GAN, RoI Pooling.
result Identifies macroeconomic factors' influence on stock prices.
New RL theory reduces sample complexity for mixing MDPs.
problem Optimal sample complexity for reinforcement learning in mixing MDPs.
method Regeneration-type ideas to analyze mixing times.
result Optimal sample complexity depends on mixing time, not just discount factor.
New algorithm solves uncertain Markov decision processes using Wasserstein uncertainty.
problem Solving Markov decision processes with uncertain transition probabilities.
method Distributionally robust Q Q Q -learning algorithm for Wasserstein uncertainty. result Convergence of the algorithm proved and demonstrated with real data.
Improved model-free RL algorithm with reduced sample complexity.
problem Learning optimal policies in discounted MDPs efficiently.
method Proposes new algorithms with improved sample complexity bounds.
result Achieves better sample complexity than existing methods.
Improved exploration in factored average-reward MDPs reduces regret.
problem Minimizing regret in unknown Factored Markov Decision Processes (FMDPs).
method DBN-UCRL strategy, inspired by UCRL2, uses Bernstein-type confidence sets for individual elements of the transition function.
result Achieves a regret bound with a leading term strictly improving over existing bounds.
We propose a new complexity measure for Markov decision processes (MDPs), the maximum expected hitting cost (MEHC). This measure tightens the closely related notion of diameter [JOA10] by accounting for the reward structure. We show that this parameter replaces diameter in the upper bound on the optimal value span of a…
UCBVI-γ algorithm minimizes regret in discounted MDPs.
problem Minimizing regret in discounted MDPs.
method Optimism in the face of uncertainty principle and Bernstein-type bonus.
result UCBVI-γ achieves nearly minimax optimal regret.
This paper presents a unifying framework for reinforcement learning and planning.
problem Sequential decision making in AI, formalized as MDP optimization.
method A unifying algorithmic framework (FRAP) for reinforcement learning and planning.
result Identifies common dimensions in MDP planning and learning algorithms.
Paper presents an algorithm for optimal regret in communicating Markov decision processes.
problem Achieving optimal regret in Markov decision processes with a communicating assumption.
method The algorithm explicitly tracks the constant K(M) to learn optimally, balancing exploration, co-exploration, and exploitation.
result The algorithm achieves asymptotically optimal regret K ( M ) log ( T ) + o ( log ( T ) ) K(M) \log(T) + \mathrm{o}(\log(T)) K ( M ) log ( T ) + o ( log ( T )) for communicating Markov decision processes. We consider reinforcement learning in parameterized Markov Decision Processes (MDPs), where the parameterization may induce correlation across transition probabilities or rewards. Consequently, observing a particular state transition might yield useful information about other, unobserved, parts of the MDP. We present a…
New RL algorithm for linear MDPs with nearly optimal regret.
problem Optimizing reinforcement learning for linear mixture Markov decision processes.
method Proposed a new Bernstein-type concentration inequality for self-normalized martingales and a computationally efficient algorithm UCRL-VTR+.
result UCRL-VTR+ achieves nearly minimax optimal regret of i l d e O ( d H T ) ilde O(dH\sqrt{T}) i l d e O ( d H T ) . Study optimality in safety-constrained Markov decision processes using asynchronous value iteration and modified Q-learning.
problem Optimality in safety-constrained Markov decision processes with multichain structure.
method Formulated as a zero-sum game, constructed asynchronous value iteration scheme and modified Q-learning algorithm.
result Resolved Bellman's principle of optimality for multichain Markov decision processes and provided learning algorithms.
Temporal difference learning explained through gradient splitting, improving convergence times.
problem Learning value functions in Markov Decision Processes with linear approximations.
method Interpreting TD learning as gradient splitting and applying convergence proofs from gradient descent.
result Improved convergence times for TD learning, especially with a minor variation.
Advances in mobile computing technologies have made it possible to monitor and apply data-driven interventions across complex systems in real time. Markov decision processes (MDPs) are the primary model for sequential decision problems with a large or indefinite time horizon. Choosing a representation of the underlying…
This paper deals with discrete-time Markov control processes on a general state space. A long-run risk-sensitive average cost criterion is used as a performance measure. The one-step cost function is nonnegative and possibly unbounded. Using the vanishing discount factor approach, the optimality inequality and an optim…
New algorithms learn in complex decision-making problems with smooth transitions.
problem Learning in complex decision-making problems with smooth transitions.
method UCB and PSRL philosophies applied to episodic Markov decision processes with kernel approximation.
result Low regret learning achieved in continuous state and action spaces.
Develops an actor-critic algorithm for risk-sensitive Markov decision processes.
problem Risk-sensitive cost criterion in Markov decision processes.
method Actor-critic algorithm with function approximation.
result Asymptotic convergence of the actor-critic algorithm.
Overview of risk-sensitive Markov decision processes with Optimized Certainty Equivalent.
problem Optimizing decision-making under risk in Markov processes.
method Analyzes risk-sensitive criteria using Optimized Certainty Equivalent, including entropic risk and Conditional Value-at-Risk.
result Conditions for the existence of optimal policies and solution procedures are provided.
Paper proposes a reinforcement learning method for trading using expert trajectories.
problem Inability of existing methods to handle long-term goals and delayed rewards in futures trading.
method Modeling futures trading as MDP, using reinforcement learning with expert trajectories and multiple short-term alpha factors.
result The proposed method outperforms traditional and deep learning methods in trading performance.
Improved online Q-learning for MDPs with concentration bounds.
problem Online Q-learning in infinite-horizon discounted MDPs with sublinear regret for large gaps.
method Smoothed ε n ε_n ε n -Greedy exploration scheme combining ε n ε_n ε n -greedy and Boltzmann exploration, analyzed using concentration bounds for contractive Markovian stochastic approximation. result Near- i l d e O ( N 9 / 10 ) ilde{O}(N^{9/10}) i l d e O ( N 9/10 ) regret bound for Smoothed ε n ε_n ε n -Greedy exploration scheme. Paper develops a new estimator for MDPs' risk functionals with lower variance and bias.
problem Estimating the distribution of returns in MDPs with high variance and bias.
method Developed a doubly robust (DR) estimator for the CDF of returns in MDPs, incorporating model-based estimation to mitigate variance issues.
result The DR estimator achieves lower variance and bias compared to IS estimators, and matches minimax lower bounds.
We consider deterministic Markov decision processes (MDPs) and apply max-plus algebra tools to approximate the value iteration algorithm by a smaller-dimensional iteration based on a representation on dictionaries of value functions. The setup naturally leads to novel theoretical results which are simply formulated due…
The paper develops a reinforcement learning model to estimate ad impact considering delayed and cumulative effects.
problem Accurately estimating ad impact considering delayed and long-term effects, cumulative impacts, and customer heterogeneity.
method Modeling ad bidding as a Contextual Markov Decision Process (CMDP) with delayed Poisson rewards, proposing a two-stage maximum likelihood estimator and reinforcement learning algorithm.
result Achieves a near-optimal regret bound of O ~ ( d H 2 T ) \tilde{O}{(dH^2\sqrt{T})} O ~ ( d H 2 T ) , validating the approach through simulation experiments. Risk measures applied to dynamic Markov processes with varying risk aversion.
problem Investigating dynamic risk measures in Markov decision processes with varying risk aversion.
method Distributional viewpoint on law-invariant convex risk measures, applied to Markov decision processes with latent costs and random actions.
result Existence of optimal policies in finite and infinite time horizons under mild assumptions.
A new RL paradigm reduces state-action-value function approximation inefficiency.
problem Challenges in state-action-value function approximation for RL.
method State Action Separable Reinforcement Learning (sasRL) decouples action space from value function learning.
result sasRL achieves up to 75% better performance than state-of-the-art MDP-based RL algorithms.
We address the problem of inverse reinforcement learning in Markov decision processes where the agent is risk-sensitive. In particular, we model risk-sensitivity in a reinforcement learning framework by making use of models of human decision-making having their origins in behavioral psychology, behavioral economics, an…
Bayesian method infers local rules for collective animal movement.
problem Learn local rules governing long-term group behaviors.
method Bayesian Inverse Reinforcement Learning with Linearly-Solvable Markov Decision Process.
result Recover true costs and find value of collective movement.
The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.
problem Maximizing long-term average reward in Markov Decision Processes with batch learning.
method Doubly robust estimator for average reward, optimization algorithm for optimal policy, finite-sample regret guarantee.
result The proposed method achieves semiparametric efficiency and provides a finite-sample regret guarantee.
Lattice investment projects support process model with corruption is formulated and analyzed. The model is based on the Ising lattice model of ferromagnetic but takes deal with the social phenomenon. Set of corruption agents is considered. It is supposed that agents are placed in sites of the lattice. Agents take decis…
New algorithm identifies best policy in MDPs faster.
problem Identifying the best policy in Markov Decision Processes.
method Problem-dependent lower bound and first algorithm with instance-specific sample complexity.
result First algorithm with reduced exploration rate for faster convergence.
Study examines sample complexity for RL with safety constraints.
problem Ensuring safety in RL for systems with constraints.
method Analyzes two RL approaches: generative and online.
result Sample complexity increased by a logarithmic factor in constraints.
We study discretizations of polynomial processes using finite state Markov processes satisfying suitable moment matching conditions. The states of these Markov processes together with their transition probabilities can be interpreted as Markov cubature rules. The polynomial property allows us to study such rules using …
CPPO learns policies from partial offline data in MDPs with structural assumptions.
problem Offline Reinforcement Learning with partial coverage assumption.
method Constrained Pessimistic Policy Optimization (CPPO) using a function class and model class constraint.
result CPPO achieves PAC guarantee with partial coverage, learning competitive policies.
We develop robust Markov Decision Processes with risk measures for uncertain environments.
problem Uncertainty in Markov Decision Processes and its impact on risk measures.
method Formulation as a Stackelberg game, robust cost and value iterations, existence of optimal policies.
result Existence of deterministic optimal policies for robust optimization and risk measures.
Q-learning for average cost MDPs gets a concentration bound.
problem Finding bounds for Q-learning in average cost MDPs.
method Derives a concentration bound using shortest path problem equivalence.
result Numerical comparison with relative value iteration shows the bound's effectiveness.