Enhances HiP-MDP for scalable, robust transfer learning.
problem Scalability and robustness in transfer learning for complex tasks.
method Introduces HiP-MDP with latent embeddings, Bayesian Neural Network, and scalable inference.
result Improved scalability and robustness in transfer learning for high-dimensional tasks.
HiP-MDPs help personalize HIV treatment across patient variations.
problem Physiological variation leads to different responses to treatments.
method Embed tasks in a low-dimensional space, updating HiP-MDP framework.
result Robust personalized medicine strategies developed for HIV treatment.
RL agents learn from a few tasks to generalize to new ones.
problem Creating efficient RL agents that can solve multiple tasks.
method GHP-MDPs model with latent variables for hidden parameters.
result State-of-the-art performance and sample-efficiency on new tasks.
This work extends HiP-MDPs to robust state abstractions for multi-task and meta-reinforcement learning.
problem Limited observability of state in HiP-MDPs for real-world scenarios with rich observation spaces.
method Inspired by Block MDPs, the work extends HiP-MDPs to enable robust state abstractions for multi-task and meta-reinforcement learning.
result Transfer and generalization bounds based on task and state similarity, and sample complexity bounds that depend on the aggregate number of samples across tasks.
Spectral method learns hidden state mapping for RL in rich-observation MDPs.
problem Challenges in RL with large state spaces and hidden low-dimensional structure.
method Spectral decomposition method to learn hidden state to observation state mapping.
result Achieves low regret with weak dependence on observed space dimensionality.
Study shows policy gradient convergence for entropy-regularized MDPs with neural nets in mean-field regime.
problem Global convergence of policy gradient for entropy-regularized MDPs with neural network approximation.
method Softmax policy with neural network approximation in mean-field regime, gradient flow in 2-Wasserstein metric, exponential convergence under sufficient regularization.
result Gradient flow converges exponentially fast to the unique stationary solution under sufficient regularization.
Optimizes learning policies in average-reward MDPs with improved sample complexity.
problem Learning optimal policies in average-reward MDPs with limited samples.
method Reduces to discounted MDPs and uses improved bounds for variance parameters.
result Establishes minimax optimal sample complexity bound of O(SA(H/ε^2))
New RL method learns to skip states in linearly q π q^π q π -realizable MDPs, simplifying to linear MDPs.
problem Online RL in episodic MDPs with linearly q π q^π q π -realizable action-values. method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.
result First polynomial-sample-complexity online RL algorithm for linearly q π q^π q π -realizable MDPs. Optimizes learning policies in MDPs with weakly communicating structure.
problem Learning optimal policies in weakly communicating MDPs with generative model.
method Span-based approach, reducing to discounted MDPs for analysis.
result First minimax optimal sample complexity bound for weakly communicating MDPs.
ARL algorithm reduces adversarial MDP to bandit problems for reliable policy learning.
problem Learning reliable policies in non-stationary, adversarial MDPs.
method Adversarial Reinforcement Learning (ARL) algorithm that converts MDP to a sequence of adversarial bandit problems.
result Achieves optimal regret bound of O ( S A T H 3 ) O(\sqrt{SATH^3}) O ( S A T H 3 ) . We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimen…
No-regret algorithm for contextual RL with GLM mappings.
problem Learning near-optimal policies in episodic MDPs with contextual information.
method Proposes no-regret online RL algorithm using optimistic and randomized exploration methods.
result Improves previous bounds and provides a lower bound for the setting.
New RL method handles hidden actions in offline learning.
problem Learning from unseen actions in real-world RL datasets.
method LURE (Learning from the Unseen: Robust Estimator) method using next-state variable as proxy.
result Valid statistical inference and improved RL conclusions with hidden actions.
A new method assigns hidden parameters deterministically to improve learning efficiency.
problem Traditional learning methods struggle with high computational burden.
method Two-stage learning with deterministic assignment of hidden parameters.
result Deterministic assignment of hidden parameters almost matches traditional learning's generalization performance.
New RL algorithm tackles nonstationary MDPs with linear approximations and varying rewards.
problem Nonstationary reinforcement learning with evolving reward and state transition functions.
method Developed a new algorithm LSVI-UCB-Restart with periodic restart, and parameter-free Ada-LSVI-UCB-Restart for unknown variation budgets.
result First minimax dynamic regret lower bound for nonstationary linear MDPs and linear MDPs lower bound.
Study on regret minimization in deterministic MDPs.
problem Minimizing regret in deterministic reinforcement learning.
method Logarithmic regret lower bounds, leveraging graph theory and cycles.
result Explicitly quantifies the fundamental limit of performance achievable by any learning algorithm.
Proposes a new DQN framework using variational inference.
problem Efficient exploration and large-scale MDPs.
method Variational inference for value function parameters.
result Achieves efficient exploration on large-scale MDPs.
Any reinforcement learning algorithm that applies to all Markov decision processes (MDPs) will suffer Ω ( S A T ) Ω(\sqrt{SAT}) Ω ( S A T ) regret on some MDP, where T T T is the elapsed time and S S S and A A A are the cardinalities of the state and action spaces. This implies T = Ω ( S A ) T = Ω(SA) T = Ω ( S A ) time to guarantee a near-optimal policy. In many settings…
We consider reinforcement learning in parameterized Markov Decision Processes (MDPs), where the parameterization may induce correlation across transition probabilities or rewards. Consequently, observing a particular state transition might yield useful information about other, unobserved, parts of the MDP. We present a…
Transformers can implement reinforcement learning algorithms from data without updates.
problem Training reinforcement learning algorithms from data without parameter updates.
method Design a teacher-mimicking training procedure for transformers to implement policy-improvement methods.
result Gradient flow converges to an optimal parameter manifold corresponding to the desired RL update.
Paper analyzes convergence of dynamic policy gradient for MDPs, improving performance in finite-time problems.
problem Optimal policies in finite-time MDPs are not stationary and require epoch-specific training.
method Introduces dynamic policy gradient combining dynamic programming and policy gradient, analyzes convergence for softmax parametrisation.
result Dynamic policy gradient training exploits finite-time structure, leading to better convergence bounds.
New RL algorithms learn from context in MDPs with smooth dynamics.
problem Learning in MDPs with varying dynamics based on context.
method Proposes algorithms for CMDPs with smooth context-MDP mappings.
result PAC learning algorithm for linear CMDPs.
DARLING tackles non-stationary RL with guarantees, improving dynamic regret.
problem Non-stationary reinforcement learning in unknown change points.
method Detection Augmented Reinforcement Learning (DARLING) for tabular and linear MDPs.
result DARLING matches minimax lower bounds in tabular and linear MDPs.
TUCRL efficiently explores and exploits in non-communicating MDPs without prior knowledge.
problem Efficient exploration-exploitation in non-communicating Markov Decision Processes (MDPs).
method Introduces TUCRL, the first algorithm for efficient exploration-exploitation in any finite MDP without prior knowledge.
result Derives a O ~ ( D e x t t t C Γ e x t t t C S e x t t t C A T ) \widetilde{O}(D^{ exttt{C}} \sqrt{Γ^{ exttt{C}} S^{ exttt{C}} AT}) O ( D e x ttt C Γ e x ttt C S e x ttt C A T ) regret bound for weakly-communicating MDPs. RSVF improves robust MDPs by relaxing ambiguity set constraints.
problem Computing robust policies with provable worst-case guarantees in uncertain environments.
method RSVF uses a Bayesian prior to optimize ambiguity set size and location, relaxing the requirement that the set be a confidence interval.
result RSVF achieves less conservative solutions with the same worst-case guarantees.
New method models unknown systems with hidden parameters using neural networks.
problem Modeling unknown dynamical systems with hidden parameters.
method Training a deep neural network (DNN) model using trajectory data of the unknown system.
result DNN model accurately predicts unknown dynamical systems with new initial conditions.
New complexity measure MEHC refines MDP upper bounds and rewards informativeness.
problem Refining complexity measures for MDPs and understanding reward informativeness.
method Introducing MEHC, a new complexity measure that tightens MDP diameter by accounting for reward structure.
result MEHC replaces diameter in upper bounds on optimal value span and UCRL2-like algorithms' regret.
New findings on hidden symmetries in ReLU networks.
problem Understanding the redundancy and symmetries in ReLU network parameter space.
method Analyzing parameter settings and function classes for various network architectures.
result For certain network architectures, there are no hidden symmetries.
Study on the limits of learning HMM parameters under various conditions.
problem Understanding the conditions under which hidden Markov model parameters can be learned.
method Nonasymptotic minimax upper and lower bounds, thresholds analysis.
result Nonasymptotic minimax bounds match up to constants, showing learnable thresholds.
New algorithm reduces reinforcement learning regret to sqrt(T) without strong dynamics assumptions.
problem Infinite-horizon average-reward reinforcement learning with linear MDPs.
method Approximate by discounted-reward MDPs and apply optimistic value iteration.
result Achieves O(sqrt(T)) regret with polynomial complexity.
New method optimizes ambiguity sets for robust MDPs, improving policy robustness.
problem Sensitivity of optimal policies to model misspecification in MDPs.
method Proposes a new approach to optimize the shape of ambiguity sets, departing from norm-bounded uniform and symmetric sets.
result Empirical results show practical promise of weighted near-optimal ambiguity sets.
Algorithm learns mixtures of Markov chains and MDPs from short trajectories.
problem Learning mixtures of Markov chains and MDPs from short unlabeled trajectories.
method Subspace estimation, spectral clustering, EM algorithm, model estimation, classification.
result 96.6% average accuracy on a mixture of two MDPs in gridworld, outperforming EM algorithm with random initialization.
Approximate Newton methods are a standard optimization tool which aim to maintain the benefits of Newton's method, such as a fast rate of convergence, whilst alleviating its drawbacks, such as computationally expensive calculation or estimation of the inverse Hessian. In this work we investigate approximate Newton meth…
The report explores parameter estimation methods in HMRF and related models.
problem Estimating hyper-parameters in HMRF and related models.
method Metropolis-Hastings algorithm, MCMC, pseudo-likelihood approximation, MAP estimation, EM algorithm.
result Effective parameter estimation methods for HMRF and related models.
Study reinforcement learning with model misspecification, extending robust MDPs.
problem Reinforcement learning under model misspecification.
method Extending robust MDPs to model-free RL, defining robust Q-learning, SARSA, and TD-learning, proving convergence.
result Convergence to approximately optimal robust policy and value function.
New state-only IL algorithm tackles MDP transition mismatch.
problem Transition dynamics mismatch between expert and imitator MDPs.
method Adversarial state-only IL with two subproblems solved iteratively.
result Effective performance improvement in transition dynamics mismatch scenarios.
A neural network with a single hidden layer can't represent certain multivariable functions.
problem Representing certain multivariable functions with a neural network having only one hidden layer.
method Developed a continuum version of a one-hidden-layer neural network with ReLU activation, and proved constraints on its parameters and second derivative.
result Existence of a smooth binary function that cannot be precisely represented by any such neural network.
MQGrad uses reinforcement learning to dynamically adjust gradient quantization bits.
problem Reduction of communication overhead in large-scale machine learning model training.
method Reinforcement learning applied to gradient quantization in parameter server.
result MQGrad accelerates deep neural network learning while maintaining prediction accuracy.
Meta RL learns task structure from experience.
problem Designing efficient reinforcement learning algorithms.
method Separately learns policy and task belief using privileged information.
result Effective at solving complex meta-RL environments.
New algorithm reduces reinforcement learning regret for linear MDPs with unknown transitions.
problem Adversarial linear mixture MDPs with bandit feedback and unknown transition.
method Proposes a new algorithm with a least square estimator and self-normalized concentration.
result Achieves improved regret bound with high probability.
We extend the Bayesian Information Criterion (BIC), an asymptotic approximation for the marginal likelihood, to Bayesian networks with hidden variables. This approximation can be used to select models given large samples of data. The standard BIC as well as our extension punishes the complexity of a model according to …
Deep nets solve MDPs without high dimensions.
problem Solving Bellman equations for MDPs in high dimensions.
method Deep neural networks with ReLU activation approximating payoff and transition functions.
result Deep nets can approximate Q Q Q -functions in polynomially bounded parameters. New method estimates sigmoids' parameters using gradient estimation.
problem Learning parameters of models with hidden variables.
method Estimate gradients at random points, cluster, use cluster centers as parameter estimates.
result Proven that estimated gradients concentrate around true parameter vectors.
PROPO tackles non-stationary MDPs with efficient policy optimization.
problem Non-stationary MDPs with varying reward and transition kernels.
method PROPO, a periodic restarted optimistic policy optimization algorithm with sliding-window-based policy evaluation and improvement.
result PROPO achieves near-optimal performance in non-stationary MDPs.
Proposes a new theoretical framework for PbRL that requires less human feedback.
problem Lack of theoretical work capturing practical PbRL frameworks.
method Introduces a reward-agnostic PbRL framework that acquires exploratory trajectories before human feedback.
result Demonstrates improved sample complexity for learning optimal policies in linear and low-rank MDPs.
New RL algorithms show model-based methods are more efficient than model-free ones in complex decision processes.
problem Efficient reinforcement learning in contextual decision processes with strategic exploration.
method Design of new model-based RL algorithms with sample complexity governed by witness rank.
result Exponential separation between model-based and model-free RL in some rich-observation settings.
Deep RL learns to play Pong from frames alone.
problem Scaling up RL problems leads to computational bottlenecks.
method End-to-end DRL approach using ANN and Policy Gradients.
result Successfully learned to play Pong from frames.
NoMoPy models noise as HMM/FHMM in Python.
problem Modeling noise in data.
method Approximate and exact EM algorithms, cross-validation, confidence region estimation.
result Validated on example problems.