New RL algorithms handle stochastic action sets, addressing divergence issues.
problem Handling stochastic action sets in reinforcement learning.
method Developed new policy gradient algorithms with variance reduction techniques.
result Proved conditions for convergence of new algorithms.
Stochastic Q-learning tackles large action spaces with reduced computation.
problem Effective decision-making in complex environments with large discrete action spaces.
method Stochastic value-based RL approaches that consider a sublinear number of actions in each iteration.
result Stochastic Q-learning achieves near-optimal returns with significantly reduced computation time.
New method for linear bandits with unknown sparsity, improving sparse regret bounds.
problem Sparse regret bounds for unknown sparsity and adversarial action sets.
method Combines online to confidence set conversions with randomized model selection over nested confidence sets.
result First sparse regret bounds for unknown sparsity and adversarial action sets.
New algorithm reduces sleeping bandits' regret to O(sqrt(T)).
problem Improving sleeping bandits with stochastic actions and adversarial rewards.
method Inspired by EXP3, new algorithm with O ( T ) O(\sqrt{T}) O ( T ) regret. result Achieves O ( T ) O(\sqrt{T}) O ( T ) regret, improving over existing O ( T 2 / 3 ) O(T^{2/3}) O ( T 2/3 ) . New algorithms handle unpredictable actions in sequential learning.
problem Learning with unreliable composite actions in online optimization.
method Follow-The-Perturbed-Leader method with Counting Asleep Times loss estimation.
result Significant improvement in performance guarantees for sleeping bandit problem.
New algorithm for safe bandits with unknown parameters and constraints.
problem Safety-critical systems with unknown parameters and constraints.
method Safe-LUCB algorithm with two phases: pure exploration and safe exploration-exploitation.
result General and problem-dependent regret bounds for the Safe-LUCB algorithm.
Deep Q-Learning method for Nash equilibria in stochastic games.
problem Model-free learning for multi-agent stochastic games, especially for general-sum games.
method Data-efficient Deep-Q-learning using local linear-quadratic expansion parametrized by deep neural networks.
result The algorithm learns optimal actions for stochastic games without experiencing all state-action pairs.
New algorithm reduces regret in stochastic linear bandits with heteroscedastic noise.
problem Optimizing performance in stochastic linear bandits with varying noise levels.
method Variance-adaptive algorithm VAEE with active exploration strategy.
result Achieves simple regret with a nearly harmonic-mean dependent rate.
This paper investigates the adversarial Bandits with Knapsack (BwK) online learning problem, where a player repeatedly chooses to perform an action, pays the corresponding cost, and receives a reward associated with the action. The player is constrained by the maximum budget B B B that can be spent to perform actions, an…
New method reduces ensemble size for linear bandits, achieving near optimal regret.
problem Achieving near optimal regret in linear bandits with limited ensemble size.
method Ensemble sampling with a size of order d log T d \log T d log T for a d d d -dimensional stochastic linear bandit. result Regret is at most ( d log T ) 5 / 2 T (d \log T)^{5/2} \sqrt{T} ( d log T ) 5/2 T , improving over linear scaling with T T T . Improved reinforcement learning for episodes with varying action sets.
problem Reinforcement learning with context-dependent action sets.
method Extends MVP algorithm to handle adversarial and stochastic contexts.
result Established minimax regret bounds of O ( S A H 3 K log L ) O(\sqrt{SAH^3K\log L}) O ( S A H 3 K log L ) for adversarial contexts. Method predicts future rewards from past actions in a linear Gaussian system.
problem Maximizing cumulative reward in a stochastic multi-armed bandit with linear Gaussian dynamics.
method Proposes a method using a modified Kalman filter to predict future rewards based on past rewards.
result Reward from any action can be used to predict another action's future reward.
TOFU-POV tackles partially observed linear bandits, achieving sublinear regret with low-dimensional action vectors.
problem Stochastic linear bandits with partially observed actions in settings like recommendation and healthcare.
method TOFU-POV estimates latent action subspace, imputes missing actions, and runs OFUL in low-dimensional coordinates.
result TOFU-POV achieves T \sqrt{T} T regret scaling with intrinsic subspace dimension, improving upon natural baselines. This paper finds an MFE in large stochastic games using reinforcement learning.
problem Finding an equilibrium in large stochastic games is difficult.
method Lower-myopic best response dynamics and posterior sampling for reinforcement learning.
result Policy and action distributions converge to optimal strategies in an MFE.
Method trains neural network for optimal decisions from stochastic simulators.
problem Suboptimal decisions from SBI approximations of posterior distributions.
method Trains neural network on simulated data to predict optimal actions.
result Induces similar cost as true posterior for optimal actions.
Paper solves stochastic contextual linear bandits using linear bandit algorithms.
problem Stochastic contextual linear bandits with unknown context distribution.
method Establishes a reduction framework to convert to linear bandit problems.
result Achieves nearly optimal regret bound of O ( d T log T ) O(d\sqrt{T\log T}) O ( d T log T ) . We introduce efficient algorithms which achieve nearly optimal regrets for the problem of stochastic online shortest path routing with end-to-end feedback. The setting is a natural application of the combinatorial stochastic bandits problem, a special case of the linear stochastic bandits problem. We show how the diffi…
Unified reinforcement learning and stochastic processes with action-driven processes.
problem Combining reinforcement learning and stochastic processes for efficient control.
method Action-driven processes, leveraging control-as-inference, and minimizing Kullback-Leibler divergence.
result Action-driven processes unify reinforcement learning and stochastic processes, equivalent to maximum entropy reinforcement learning.
Study robust control for systems with continuous states using adversarial perturbations.
problem Fragile policies in Markov control models under internal or external perturbations.
method Distributionally robust stochastic control with adaptive adversarial perturbations.
result Optimal robust policies for continuous state systems with uniform learning guarantees.
Intelligent agents can learn to represent the action spaces of other agents simply by observing them act. Such representations help agents quickly learn to predict the effects of their own actions on the environment and to plan complex action sequences. In this work, we address the problem of learning an agent's action…
Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…
Noiseless IO bounds inferred from demonstrations, matching adversarial settings.
problem Inferring decision-maker's objective from observed data.
method High-probability generalization bounds for induced action set.
result Generalization bound of O ( d T ) {O}(\frac{d}{T}) O ( T d ) for consistent estimators. Quantum stochastic flow computes heat kernel traces for Ricci flat manifolds.
problem Computing heat kernel traces for Ricci flat manifolds.
method Quantum stochastic differential equation (qsde) on Fock space over L 2 L^2 L 2 differential 1-forms, adapted flow construction. result Trace of the connection Laplacian heat kernel can be computed over any compact Ricci-flat Riemannian manifold.
Study shows how to learn optimal policies quickly in stochastic control problems.
problem Learning optimal policies in large, continuous state and action spaces with limited data.
method Analyzes three geometric exponents to quantify fast policy regret convergence.
result Shows that fast policy regret convergence is induced by specific geometric structures.
This paper explains CART random forests using stochastic control theory.
problem Understanding the inner workings of CART random forests.
method Developed a stochastic-control perspective on CART random forests, interpreting feature subsampling as a random feasible action set and the split rule as a policy.
result Established that the CART policy is locally stabilizing but globally suboptimal for the forest objective.
Abstract reviews Markov processes with jumps on manifolds and Lie groups.
problem Analyzing Markov processes with jumps in geometric settings.
method Stochastic differential equations, Courrège theorem, invariant Markov processes.
result Developments in Lie groups and manifolds under various actions.
The paper tackles counterfactual learning for stochastic policies with continuous actions.
problem Learning stochastic policies with continuous actions from logged data.
method Introduces a joint kernel embedding of contexts and actions to model continuous actions, and uses proximal point algorithms and smooth estimators for optimization.
result Demonstrates the benefits of using proximal point algorithms and smooth estimators for counterfactual learning.
Improved Bayesian regret bound for linear Thompson sampling with general distributions.
problem Proving an improved Bayesian regret bound for linear Thompson sampling with general distributions.
method Generalized elliptical potential lemma for non-Gaussian noise and prior distributions.
result Minimax optimal regret bound for changing action sets with general prior and noise distributions.
New method uses hindsight to make exploration robust in stochastic environments.
problem Exploration in sparse-reward or reward-free environments, especially in stochastic settings.
method Learn representations of the future that capture unpredictable aspects, using them to predict and reward only the predictable parts of the world.
result Improves exploration in Atari games and Montezuma's Revenge, robust to stochasticity.
Proposes a new resampling method for off-policy evaluation in stochastic control.
problem Estimating policy performance from data generated under a different policy.
method K-nearest neighbor resampling procedure for off-policy evaluation.
result Statistical consistency results for the proposed method under weak conditions.
Algorithm maximizes rewards with a budget and giving up option.
problem Sequential decision-making with stochastic rewards and resource consumption.
method Upper Confidence Bound (UCB) algorithm for maximizing cumulative reward.
result Logarithmic regret bound with improved dependence on problem parameters.
We study the regret of optimal strategies for online convex optimization games. Using von Neumann's minimax theorem, we show that the optimal regret in this adversarial setting is closely related to the behavior of the empirical minimization algorithm in a stochastic process setting: it is equal to the maximum, over jo…
Improved ExO method achieves near-optimal bounds in both stochastic and adversarial settings.
problem Finding optimal exploration strategies in online decision-making with limited feedback.
method Exploration by Optimization with hybrid regularizers for locally observable games.
result Achieved nearly optimal bounds of O ( ∑ a e q a ∗ k 2 m 2 log T / Δ a ) O(\sum_{a
eq a^*} k^2 m^2 \log T / Δ_a) O ( ∑ a e q a ∗ k 2 m 2 log T / Δ a ) in stochastic and adversarial environments. We pursue an inverse approach to utility theory and consumption & investment problems. Instead of specifying an agent's utility function and deriving her actions, we assume we observe her actions (i.e. her consumption and investment strategies) and ask if it is possible to derive a utility function for which the observ…
This study examines how learning algorithms affect collective action in machine learning.
problem The impact of collective action on machine learning is limited when not considering the choice of learning algorithms.
method Focuses on distributionally robust optimization and stochastic gradient descent, analyzing their effects on collective success.
result The choice of learning algorithm significantly impacts the effective size and success of a collective in machine learning.
Paper introduces a new analysis framework for stochastic linear bandits.
problem Optimizing decision-making in online experiments with noisy rewards.
method Develops a general analysis framework and algorithms for stochastic linear bandits.
result Introduces new algorithms like SG that improve performance and provide new regret bounds.
A new bandit model with stochastic context distributions and UCB algorithm.
problem Learning optimal actions in a stochastic environment with hidden contexts.
method Stochastic contextual bandit model and UCB algorithm.
result Order-optimal high-probability bound on cumulative regret for linear and kernelized reward functions.
Optimal strategy proposed for maximizing cumulative reward in continuum-armed bandits.
problem Maximizing cumulative reward in a scenario with limited resources and unknown stochastic rewards.
method Proposed an optimal strategy for a nonparametric setting with side information on actions.
result Optimal regret scales as \(O(T^{1/3})\) up to poly-logarithmic factors when \(T\) is proportional to \(N\).
Paper solves discounted stochastic games with near-optimal time and sample complexity.
problem Solving discounted stochastic two-player games with optimal complexity.
method Generalizes Q-learning to two-player strategy computation, overcoming limitations of existing methods.
result Near-optimal ε ε ε -strategy computation with polylogarithmic factors in 1 − γ 1 - γ 1 − γ and ε − 2 ε^{-2} ε − 2 . Bayesian optimisation tackles stochastic MPC hyper-parameter tuning.
problem Fine-tuning hyper-parameters in stochastic MPC models.
method Heteroscedastic Bayesian optimisation framework.
result Framework effectively tunes hyper-parameters in control problems.
PLOTS learns procedural actions from observed sequences, up to 100x faster.
problem Learning procedural actions from observed sequences efficiently.
method Exploits subtask structure to incrementally build action plans, optimistically explores actions.
result Explicit procedural learning is 100x faster than policy-gradient methods and model-based approaches.
DQNs can approximate optimal Q-functions with high accuracy on compact sets.
problem Approximating optimal Q-functions in continuous-time Markov Decision Processes.
method Stochastic control, FBSDEs, residual network approximation theorems, large deviation bounds, viscosity solutions.
result DQNs can approximate optimal Q-functions on compact sets with arbitrary accuracy and high probability.
New model for display advertising with stochastic and adversarial components.
problem Display advertising with stochastic and adversarial click-through-rates.
method Adversarial scaling model; two algorithms tested: action elimination and mirror descent.
result Two algorithms are robust to adversarial scaling.
IRL addresses weaknesses in DDPG and A3C for continuous reinforcement learning.
problem Theoretical weaknesses in DDPG and A3C for continuous reinforcement learning.
method IRL based on stochastic differential equations, ensuring action continuity and variance control.
result IRL method guarantees action continuity and variance control, allowing positive interaction with the environment.
FinFlowRL combines imitation and reinforcement learning for better financial control.
problem Traditional stochastic control methods fail in real-world finance due to changing market conditions.
method FinFlowRL uses imitation learning to pretrain an adaptive meta policy, then finetunes it with reinforcement learning.
result FinFlowRL consistently outperforms individual strategies across various market conditions.
We consider stochastic multi-armed bandit problems with complex actions over a set of basic arms, where the decision maker plays a complex action rather than a basic arm in each round. The reward of the complex action is some function of the basic arms' rewards, and the feedback observed may not necessarily be the rewa…
New algorithm identifies optimal actions in large reward spaces efficiently.
problem Finding the best action from a large set of options with minimal trials.
method GenTS-Explore algorithm for real-valued combinatorial pure exploration.
result Achieves optimal sample complexity for large action sets.
New algorithm tackles delayed feedback in Lipschitz bandits with sublinear regret.
problem Delayed feedback in Lipschitz bandits.
method Design of algorithms for bounded and unbounded stochastic delays.
result Sublinear regret guarantees for both bounded and unbounded delays.