Extends reinforcement learning to continuous state spaces with safety constraints.
problem Safety-critical reinforcement learning in continuous state spaces with unknown dynamics.
method Introduces a novel Budgeted Bellman Optimality operator and applies it to continuous state spaces.
result Validated on spoken dialogue and autonomous driving applications.
New RL method tackles dynamic MDPs with evolving rewards and states.
problem Dynamic MDPs with evolving rewards and states.
method Sliding Window Upper-Confidence bound for Reinforcement Learning (SWUCRL2-CW) and Bandit-over-Reinforcement Learning (BORL).
result Achieves dynamic regret bound for non-stationary MDPs.
Improved algorithm for online planning with tighter bounds.
problem Online planning in Markov Decision Processes with open-loop policies and budget constraints.
method Proposed KLOLOP algorithm with tighter upper-confidence bounds and efficient implementation.
result KLOLOP leads to better practical performances with sample complexity bound retained.
A meta-learning approach for efficient algorithm selection in budget-limited scenarios.
problem Efficiently selecting the best-performing machine learning algorithm with limited computational resources.
method A Markov Decision Process framework where an agent decides whether to train, wake up, or start new algorithms based on partial learning curves.
result Meta-learning from learning curves improves algorithm selection, especially when learning curves do not intersect frequently.
Automated Budget Constrained Training optimizes model training under time constraints.
problem Balancing model quality and computational cost in constrained time.
method Developed a hyperparameter optimisation algorithm that learns the relationship between hyperparameters, model quality, and computational cost.
result The algorithm optimally decides whether to terminate or continue training, and what hyperparameters to use.
Paper tackles non-monotonic resource utilization in sequential decision-making.
problem Sequential decision-making under uncertainty with resource constraints.
method Introduces a new MDP policy with constant regret against LP relaxation.
result Develops a learning algorithm with logarithmic regret for unknown outcome distributions.
In crowd labeling, a large amount of unlabeled data instances are outsourced to a crowd of workers. Workers will be paid for each label they provide, but the labeling requester usually has only a limited amount of the budget. Since data instances have different levels of labeling difficulty and workers have different r…
Develops algorithms for reinforcement learning in non-stationary MDPs with dynamic regret bounds.
problem Sequential decision-making under temporal drifts in Markov decision processes.
method Sliding Window Upper-Confidence bound for Reinforcement Learning (SWUCRL2-CW) and Bandit-over-Reinforcement Learning (BORL) algorithms.
result Achieves superior empirical performance and low dynamic regret bounds in non-stationary RL settings.
Paper tests Markov assumption in sequential decision making.
problem Testing the Markov assumption in sequential decision making.
method Forward-Backward Learning procedure to test MA without assuming parametric forms.
result The proposed test plays a crucial role in identifying optimal policies in complex decision processes.
Paper presents an efficient algorithm for linear MDP with low switching cost.
problem Large state space reinforcement learning problems with low switching cost.
method First algorithm for linear MDP with low switching cost, achieving near-optimal regret and switching cost.
result Regret bound of $\widetilde{O}\left(\sqrt{d^3H^4K}
ight)$ and near-optimal switching cost of $O\left(d H\log K
ight)$ .
Study on policy testing in MDPs with lower bounds and new algorithm.
problem Deciding if policy value exceeds a threshold with limited samples.
method Derived lower bound, proposed new algorithm, reformulated problem, used policy optimization in reversed MDP.
result New algorithm outperforms existing methods in policy testing.
New RL algorithm tackles nonstationary MDPs with linear approximations and varying rewards.
problem Nonstationary reinforcement learning with evolving reward and state transition functions.
method Developed a new algorithm LSVI-UCB-Restart with periodic restart, and parameter-free Ada-LSVI-UCB-Restart for unknown variation budgets.
result First minimax dynamic regret lower bound for nonstationary linear MDPs and linear MDPs lower bound.
Paper develops a new RL method for MDPs with uncertainty, achieving better regret bounds.
problem Online reinforcement learning in environments with both endogenous and exogenous uncertainty.
method Developed a VB-UCRL algorithm that restarts based on variation schedules.
result Established a regret bound of saving at most S \sqrt{S} S or S 1 6 T 1 12 S^{\frac{1}{6}}T^{\frac{1}{12}} S 6 1 T 12 1 . New algorithm solves uncertain Markov decision processes using Wasserstein uncertainty.
problem Solving Markov decision processes with uncertain transition probabilities.
method Distributionally robust Q Q Q -learning algorithm for Wasserstein uncertainty. result Convergence of the algorithm proved and demonstrated with real data.
Model-free RL solves financial portfolio optimization without knowing component dynamics.
problem Optimizing financial portfolios without knowing specific component dynamics.
method Reformulate portfolio optimization as a MDP and solve using model-free RL.
result Model-free RL can solve portfolio optimization problems.
Survey of integrating planning and learning in model-based reinforcement learning.
problem Sequential decision making in AI, formalized as MDP optimization.
method Systematic coverage of dynamics model learning and planning-learning integration.
result Broad conceptual overview of model-based reinforcement learning.
Rank aggregation based on pairwise comparisons over a set of items has a wide range of applications. Although considerable research has been devoted to the development of rank aggregation algorithms, one basic question is how to efficiently collect a large amount of high-quality pairwise comparisons for the ranking pur…
New method optimizes costly evaluations in Bayesian optimization.
problem Costly evaluations in BO methods, especially in hyperparameter tuning.
method Formulated as a CMDP, developed rollout approximation for optimal policy.
result Improved cost efficiency over standard BO methods.
Bridges uplift modeling and sequential decision-making with online budget allocation.
problem Treatment allocation under budget constraints in digital advertising.
method Budget-Constrained Causal Bandits (BCCB) integrates learning, exploration, and budget pacing.
result Data-efficiency crossover: BCCB operates effectively from the first user, 3-5x lower performance variance.
Paper presents an algorithm for optimal regret in communicating Markov decision processes.
problem Achieving optimal regret in Markov decision processes with a communicating assumption.
method The algorithm explicitly tracks the constant K(M) to learn optimally, balancing exploration, co-exploration, and exploitation.
result The algorithm achieves asymptotically optimal regret K ( M ) log ( T ) + o ( log ( T ) ) K(M) \log(T) + \mathrm{o}(\log(T)) K ( M ) log ( T ) + o ( log ( T )) for communicating Markov decision processes. New algorithm reduces regret and constraint violation in adversarial CMDP learning.
problem Online learning for episodic stochastically constrained Markov decision processes (CMDPs) with adversarial loss.
method Upper Confidence Primal-Dual Reinforcement Learning (UC-PDL) algorithm.
result Achieves O ~ ( L ∣ S ∣ ∣ A ∣ T ) \widetilde{\mathcal{O}}(L|\mathcal{S}|\sqrt{|\mathcal{A}|T}) O ( L ∣ S ∣ ∣ A ∣ T ) upper bounds of both regret and constraint violation. Study optimality in safety-constrained Markov decision processes using asynchronous value iteration and modified Q-learning.
problem Optimality in safety-constrained Markov decision processes with multichain structure.
method Formulated as a zero-sum game, constructed asynchronous value iteration scheme and modified Q-learning algorithm.
result Resolved Bellman's principle of optimality for multichain Markov decision processes and provided learning algorithms.
Advances in mobile computing technologies have made it possible to monitor and apply data-driven interventions across complex systems in real time. Markov decision processes (MDPs) are the primary model for sequential decision problems with a large or indefinite time horizon. Choosing a representation of the underlying…
New algorithms learn in complex decision-making problems with smooth transitions.
problem Learning in complex decision-making problems with smooth transitions.
method UCB and PSRL philosophies applied to episodic Markov decision processes with kernel approximation.
result Low regret learning achieved in continuous state and action spaces.
New framework for RL with opponents, improving learning outcomes.
problem Learning in RL with potential adversaries.
method Threatened Markov Decision Processes (TMDPs) and level-k thinking.
result Improved RL performance by accounting for adversaries.
Develops an actor-critic algorithm for risk-sensitive Markov decision processes.
problem Risk-sensitive cost criterion in Markov decision processes.
method Actor-critic algorithm with function approximation.
result Asymptotic convergence of the actor-critic algorithm.
A deep Q-learning strategy optimizes portfolio trading efficiency.
problem Optimizing dynamic portfolio allocation schemes.
method Formulated a Markov decision process model with deep Q-learning for discrete combinatorial actions.
result Outperforms benchmark strategies in real-world trading simulations.
Overview of risk-sensitive Markov decision processes with Optimized Certainty Equivalent.
problem Optimizing decision-making under risk in Markov processes.
method Analyzes risk-sensitive criteria using Optimized Certainty Equivalent, including entropic risk and Conditional Value-at-Risk.
result Conditions for the existence of optimal policies and solution procedures are provided.
Paper models MOOC users behavior using MDP for easier classification.
problem Difficulty in defining and classifying MOOC users based on their behavior.
method Uses Markov Decision Process (MDP) to model user behavior from log data.
result Classifies users based on their intentions inferred from MDP rewards.
We optimize saddle-point problems for large-scale Markov decision processes.
problem Optimizing policies in large-scale Markov decision processes.
method Characterized conditions for convergence and designed an optimization algorithm.
result Our algorithm converges faster and is state-space independent.
Algorithm finds safe zones in policy Markov Decision Processes to limit trajectory escape.
problem Finding safe zones in policy Markov Decision Processes to limit trajectory escape.
method Bi-criteria approximation learning algorithm with polynomial sample complexity.
result Achieves almost 2 approximation for both escape probability and safe zone size.
Monte Carlo algorithms simulate some prescribed number of samples, taking some random real time to complete the computations necessary. This work considers the converse: to impose a real-time budget on the computation, which results in the number of samples simulated being random. To complicate matters, the real time t…
Risk measures applied to dynamic Markov processes with varying risk aversion.
problem Investigating dynamic risk measures in Markov decision processes with varying risk aversion.
method Distributional viewpoint on law-invariant convex risk measures, applied to Markov decision processes with latent costs and random actions.
result Existence of optimal policies in finite and infinite time horizons under mild assumptions.
Bayesian method infers local rules for collective animal movement.
problem Learn local rules governing long-term group behaviors.
method Bayesian Inverse Reinforcement Learning with Linearly-Solvable Markov Decision Process.
result Recover true costs and find value of collective movement.
We address the problem of inverse reinforcement learning in Markov decision processes where the agent is risk-sensitive. In particular, we model risk-sensitivity in a reinforcement learning framework by making use of models of human decision-making having their origins in behavioral psychology, behavioral economics, an…
The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.
problem Maximizing long-term average reward in Markov Decision Processes with batch learning.
method Doubly robust estimator for average reward, optimization algorithm for optimal policy, finite-sample regret guarantee.
result The proposed method achieves semiparametric efficiency and provides a finite-sample regret guarantee.
New algorithm identifies best policy in MDPs faster.
problem Identifying the best policy in Markov Decision Processes.
method Problem-dependent lower bound and first algorithm with instance-specific sample complexity.
result First algorithm with reduced exploration rate for faster convergence.
New objective function preserves Bellman's principle for policy gradient.
problem Lack of objective function capturing Bellman's principle optimally.
method Proposed a new objective function and its gradient.
result Preserves Bellman's principle of optimality in policy gradient.
Frequently, acquiring training data has an associated cost. We consider the situation where the learner may purchase data during training, subject TO a budget. IN particular, we examine the CASE WHERE each feature label has an associated cost, AND the total cost OF ALL feature labels acquired during training must NOT e…
Study resource allocation strategies in sequential decisions with unknown rewards.
problem Sequential resource allocation with unknown rewards.
method Design combinatorial multi-armed bandit algorithms for discrete or continuous budgets.
result Prove algorithms achieve logarithmic cumulative regret under semi-bandit feedback.
We develop robust Markov Decision Processes with risk measures for uncertain environments.
problem Uncertainty in Markov Decision Processes and its impact on risk measures.
method Formulation as a Stackelberg game, robust cost and value iterations, existence of optimal policies.
result Existence of deterministic optimal policies for robust optimization and risk measures.
New algorithm learns Markov decision processes from test data.
problem Learning Markov decision processes from limited test data.
method L*-based active learning with sampling.
result Sampling-based algorithm outperforms passive learning.
Q-learning for average cost MDPs gets a concentration bound.
problem Finding bounds for Q-learning in average cost MDPs.
method Derives a concentration bound using shortest path problem equivalence.
result Numerical comparison with relative value iteration shows the bound's effectiveness.
New Q-learning method achieves optimal sample complexity for average-reward problems.
problem Challenges in achieving optimal sample complexity for average-reward Q-learning.
method Synchronous and asynchronous Q-learning with a new contraction principle.
result Optimal O ~ ( ε − 2 ) \widetilde{O}(\varepsilon^{-2}) O ( ε − 2 ) sample complexity guarantees. This paper analyzes risk-sensitive reinforcement learning with Conditional Value-at-Risk (CVaR) for robust Markov Decision Processes.
problem Risk-sensitive reinforcement learning for robust Markov Decision Processes (RMDPs) with state-action-dependent ambiguity sets.
method The paper establishes a connection between robustness and risk sensitivity, defining a new risk measure NCVaR and proposing value iteration algorithms.
result The proposed approach using NCVaR optimization and value iteration algorithms can solve problems with state-action-dependent ambiguity sets.
Unified framework connects reinforcement learning and optimal control.
problem Sequential decision-making across different communities.
method Unified modeling framework based on optimizing policies.
result Unified framework includes four universal policy classes.
We characterize value functions in partially observable MDPs as semi-algebraic sets.
problem Understanding feasible value functions in partially observable Markov decision processes.
method Characterization of feasible value functions as semi-algebraic sets defined by polynomial inequalities.
result The feasible set of value functions in POMDPs is a semi-algebraic set, not a polytope as in MDPs.
New RL framework models continuous-time dynamics using neural ODEs.
problem Modeling continuous-time dynamics in semi-Markov decision processes.
method Model-based reinforcement learning with neural ODEs.
result High-performing policies developed with minimal data.