Paper proposes DAC-ML, a cognitive architecture that learns quickly from few episodes.
problem Sample inefficiency in AI learning action policies.
method Incorporates hippocampus-inspired sequential memory system into DAC theory of mind.
result DAC-ML rapidly converges to effective action policies maximizing reward.
Random projection improves stability in neural episodic control.
problem Stability and efficiency in end-to-end deep reinforcement learning.
method Incorporating random projection into Neural Episodic Control (NEC) architecture.
result The proposed architecture reduces the number of parameters to learn, improving stability.
MEMEC improves sample efficiency in reinforcement learning.
problem Lack of sample efficiency in reinforcement learning.
method Proposes MEMEC, a Boltzmann policy with state-dependent temperature for more principled exploration.
result MEMEC outperforms other methods on classic RL environments and Atari games.
State of the art deep reinforcement learning algorithms take many millions of interactions to attain human-level performance. Humans, on the other hand, can very quickly exploit highly rewarding nuances of an environment upon first discovery. In the brain, such rapid learning is thought to depend on the hippocampus and…
AE-DDPG improves DDPG for continuous control in complex environments with less data and time.
problem Data insufficiency and training inefficiency in DDPG for complex environments.
method Asynchronous episodic data collection, modified experience replay, and new action noise.
result AE-DDPG achieves higher rewards and less training time in complex environments.
Logarithmic regret achieved in continuous-time linear-quadratic reinforcement learning.
problem Optimizing control actions in unknown continuous-time systems over a finite time horizon.
method Least-squares algorithm based on continuous-time observations and controls, with perturbation analysis and parameter estimation error analysis.
result Logarithmic regret bound of order O ( ( ln M ) ( ln ln M ) ) O((\ln M)(\ln\ln M)) O (( ln M ) ( ln ln M )) . Single episode policy transfer in RL without access to rewards.
problem Performing near-optimally in a single attempt with unknown dynamics.
method Optimizes a probe and inference model to estimate latent variables for universal control policy.
result Significantly outperforms existing adaptive approaches in diverse domains.
Efficiently stores and retrieves past states for faster learning in reinforcement learning.
problem Data inefficiency and memory limitations in reinforcement learning.
method Dynamic online k-means for state clustering and prioritization.
result Dynamic online k-means improves performance with smaller memory sizes.
Study optimal control in unknown nonlinear systems with near-optimal regret bound.
problem Sequential control in unknown, nonlinear dynamical systems.
method LC^3 algorithm, based on information theory.
result Near-optimal O ( T ) O(\sqrt{T}) O ( T ) regret bound for episodic settings. Bootstrap method for Markov chains in reinforcement learning.
problem Distributional consistency in finite controlled Markov chains with unknown control policies.
method Model-based bootstrap with novel LLN and CLT for visitation counts and transition increments.
result Asymptotically valid confidence intervals for value and Q Q Q -functions in offline RL. Deep reinforcement learning methods attain super-human performance in a wide range of environments. Such methods are grossly inefficient, often taking orders of magnitudes more data than humans to achieve reasonable performance. We propose Neural Episodic Control: a deep reinforcement learning agent that is able to rap…
New algorithm learns reward signals from episodic returns for better reinforcement learning.
problem Difficulty in designing reward functions for real-world reinforcement learning tasks.
method Introduces a new algorithm that decomposes episodic returns into time-step rewards using deep neural networks.
result Learning reward signals from episodic returns improves reinforcement learning efficiency.
Study shows how to control jump-diffusion processes with stable feedback controls in reinforcement learning.
problem Control jump-diffusion processes with unknown coefficients in reinforcement learning.
method Lipschitz continuous optimal feedback controls, stability analysis of forward-backward SDEs, least-squares algorithm.
result Achieves O ( N ln N ) O(\sqrt{N\ln N}) O ( N ln N ) regret for linear-convex learning problems with jumps. Proposes a new resampling method for off-policy evaluation in stochastic control.
problem Estimating policy performance from data generated under a different policy.
method K-nearest neighbor resampling procedure for off-policy evaluation.
result Statistical consistency results for the proposed method under weak conditions.
New framework for RL with linear-convex models reduces performance gap.
problem Continuous-time episodic reinforcement learning with unknown coefficients and convex objectives.
method Probabilistic framework and phase-based learning algorithm for optimal exploration-exploitation trade-off.
result Sublinear regrets achieved, matching best possible results in literature.
Study of multidimensional control problems with reflection controls.
problem Solving control problems with reflection controls in multidimensional settings.
method Gradient descent algorithm for polytope approximations, data-driven domain estimator, episodic learning algorithm.
result Data-driven solutions for unknown diffusion dynamics with sublinear regret.
New Q-learning algorithms reduce regret in inventory control problems.
problem Efficiently learning optimal policies in inventory control problems with limited feedback.
method Proposed Elimination-Based Half-Q-Learning (HQL) and Full-Q-Learning (FQL) algorithms with theoretical regret bounds.
result HQL incurs i l d e O ( H 3 T ) ilde{\mathcal{O}}(H^3\sqrt{ T}) i l d e O ( H 3 T ) regret, FQL incurs i l d e O ( H 2 T ) ilde{\mathcal{O}}(H^2\sqrt{ T}) i l d e O ( H 2 T ) regret, independent of state and action space sizes. This paper tackles adaptive control of unknown Markov jump systems with sample complexity and regret bounds.
problem Adaptive control of unknown Markov jump systems with changing dynamics.
method Identification-based adaptive control using a system identification algorithm and certainty equivalent control.
result The proposed adaptive control scheme achieves O ( T ) \mathcal{O}(\sqrt{T}) O ( T ) regret, improving to O ( p o l y l o g ( T ) ) \mathcal{O}(polylog(T)) O ( p o l y l o g ( T )) with partial knowledge. Proposes PIC and POIC for measuring task difficulty in RL.
problem Lack of metrics to measure task difficulty in RL.
method Introduces policy information capacity (PIC) and policy-optimal information capacity (POIC) as metrics based on mutual information.
result Empirically shows PIC and POIC correlate with task solvability better than alternatives.
The success of popular algorithms for deep reinforcement learning, such as policy-gradients and Q-learning, relies heavily on the availability of an informative reward signal at each timestep of the sequential decision-making process. When rewards are only sparsely available during an episode, or a rewarding feedback i…
Paper proposes an intersection decision algorithm for autonomous vehicles.
problem Navigating intersections with non-automated vehicles.
method Combines reinforcement learning for high-level decisions and model predictive control for low-level planning.
result The proposed algorithm outperforms another controller in success rate and training episodes.
Control of non-episodic, finite-horizon dynamical systems with uncertain dynamics poses a tough and elementary case of the exploration-exploitation trade-off. Bayesian reinforcement learning, reasoning about the effect of actions and future observations, offers a principled solution, but is intractable. We review, then…
New approach uses Gaussian processes to learn and track complex systems with guaranteed accuracy.
problem Inaccurate first principle models for complex systems due to data complexity.
method Bayesian prediction error bound for Gaussian process regression, derived from kernel-based data density.
result Achieves vanishing tracking error with increasing data density, providing time-varying accuracy guarantees.
New algorithm reduces regret in stochastic shortest path problems.
problem Planning and control in environments with unknown dynamics and variable episode lengths.
method Developed an algorithm with a new regret bound of O ( B ⋆ ∣ S ∣ ∣ A ∣ K ) O(B_\star |S| \sqrt{|A| K}) O ( B ⋆ ∣ S ∣ ∣ A ∣ K ) . result Guaranteed a significant reduction in regret compared to previous methods.
Tiny Eats GRU detects eating episodes on a microcontroller.
problem Automatic dietary monitoring on low-power devices.
method Shallow gated recurrent unit (GRU) architecture on Arm Cortex M0+.
result Tiny Eats GRU achieves 95.15% accuracy with 4% memory usage and 6 ms latency.
New RL algorithm achieves nearly optimal performance for linear MDPs.
problem Optimal reinforcement learning for episodic linear MDPs.
method Weighted linear regression with variance estimator and rare-switching policy.
result Achieves nearly minimax optimal regret i l d e O ( d H 3 K ) ilde O(d\sqrt{H^3K}) i l d e O ( d H 3 K ) . This work optimizes RL algorithms using entropy regularisation for continuous-time LQ problems.
problem Designing RL algorithms to balance exploration and exploitation in noisy environments.
method Entropy regularisation in two formulations: exploratory control and proximal policy update.
result Regret of O ( N ) \mathcal{O}(\sqrt{N}) O ( N ) for both learning algorithms over N N N episodes. A new method learns from multi-modal sequences with external memory.
problem Learning new modes in a dynamic environment without prior knowledge.
method Maintains a neural episodic memory with a Dirichlet Process prior to store mode descriptors and transfers knowledge through retrieval.
result Performs continual learning favorably compared to mainstream approaches.
Meta-learning control algorithm with finite-time guarantees for unknown systems.
problem Online control of unknown linear systems with constraints.
method Provable regret guarantees for an iterative control algorithm.
result Regret bounds of O ( T 3 / 4 ) O(T^{3/4}) O ( T 3/4 ) for controller cost and constraint violation. Study nonparametric estimator for Markov chain transition matrices in offline setting.
problem Estimating transition matrices of finite controlled Markov chains from logged data.
method Developed sample complexity bounds and conditions for minimaxity.
result Achieving certain statistical risk requires balancing mixing properties and sample size.
New algorithms for planning with adversarial changes in costs.
problem Planning with adversarial changes in costs over time.
method Developed algorithms for adversarial SSP with high probability regret bounds.
result Obtained sub-linear regret bounds for adversarial SSP.
Transforming sparse outcomes into dense process rewards for efficient reinforcement learning.
problem Training RL policies to maximize sparse outcomes.
method Incentivizing policy matching state-action visitations of successful episodes.
result Significantly faster RL finetuning performance.
Study preference-based reinforcement learning in episodic kernel MDPs.
problem Learning from episodic human preferences in reinforcement learning.
method Developed preference-based value estimation and confidence sets for kernel-based MDPs.
result Proved high-probability regret bounds that converge to optimal policy value.
Algorithm tackles constrained reinforcement learning with concave-convex and knapsack constraints.
problem Constrained episodic reinforcement learning with concave rewards and convex constraints.
method Modular analysis with strong theoretical guarantees for concave-convex and knapsack settings.
result Significantly outperforms existing approaches in constrained episodic environments.
Algorithm improves multi-armed bandit performance by transferring reward samples.
problem Sequential multi-armed bandit problem with changing reward distributions.
method UCB algorithm with reward sample transfer.
result Significant improvement in cumulative regret over standard UCB.
Researchers tackle malaria control as a reinforcement learning problem with limited data.
problem Learning optimal malaria control policies with scarce data.
method Applied Genetic Algorithm, Bayesian Optimization, and Q-learning with sequence breaking.
result Q-Learning with sequence breaking achieved 7th place in the KDD Cup challenge.
BerlinUCB learns from episodic rewards in nonstationary contexts.
problem Online learning with episodic rewards in nonstationary environments.
method BerlinUCB integrates clustering for self-supervision.
result BerlinUCB outperforms standard contextual bandits in various scenarios.
New method extends supervised learning for non-stationary control problems.
problem Optimal control in non-stationary, reset-free environments.
method Prospective Learning with Control (PLuC) using Empirical Risk Minimization (ERM).
result ERM asymptotically achieves Bayes optimal policy in non-stationary environments.
When applied to training deep neural networks, stochastic gradient descent (SGD) often incurs steady progression phases, interrupted by catastrophic episodes in which loss and gradient norm explode. A possible mitigation of such events is to slow down the learning process. This paper presents a novel approach to contro…
TSDS framework reduces edge LLM agent compute by 43%-73% while maintaining safety and reliability.
problem Managing reasoning budget and uncertainty in edge LLM agents.
method Integrates a lightweight convergence probe and a perplexity-based deferral rule calibrated via multi-objective LTT.
result Reduces per-episode thinking compute by 43%-73% over deferral-only baselines.
Improved regret bound for online learning in unknown MDPs.
problem Online learning in unknown episodic MDPs with changing loss functions.
method Adapts adversarial MDP model to convex performance criteria using entropic regularization.
result Achieved i l d e O ( L ∣ X ∣ ∣ A ∣ T ) ilde{O}(L|X|\sqrt{|A|T}) i l d e O ( L ∣ X ∣ ∣ A ∣ T ) regret bound. NIPA aims to translate brain learning mechanisms into scalable Bayesian inference.
problem Scalable Bayesian inference for large-scale statistical machine learning problems.
method Neural-inspired algorithm combining model-based, model-free, and episodic-control modules.
result Advances Bayesian methods and facilitates their application to deep learning.
Algorithm reduces episode count for CMDPs with constraints.
problem Online decision-making with constraints in episodic CMDPs.
method Optimistic planning using linear programming for PAC guarantee.
result Probably approximately correct (PAC) guarantee on episode count.
Many interesting real world domains involve reinforcement learning (RL) in partially observable environments. Efficient learning in such domains is important, but existing sample complexity bounds for partially observable RL are at least exponential in the episode length. We give, to our knowledge, the first partially …
Algorithm improves online learning in adversarial bandits.
problem Online learning in adversarial multi-armed bandits with non-uniform best arm distribution.
method Online-within-online setup, inner and outer learners, leveraging non-uniform empirical distribution of best arms.
result Improves regret bounds for non-uniform best arm distributions.
Typical neural networks with external memory do not effectively separate capacity for episodic and working memory as is required for reasoning in humans. Applying knowledge gained from psychological studies, we designed a new model called Differentiable Working Memory (DWM) in order to specifically emulate human workin…
Reinforcement learning (RL) algorithms have made huge progress in recent years by leveraging the power of deep neural networks (DNN). Despite the success, deep RL algorithms are known to be sample inefficient, often requiring many rounds of interaction with the environments to obtain satisfactory performance. Recently,…
Tiny episodic memory significantly improves continual learning performance.
problem Transfer knowledge between tasks in continual learning.
method Store and replay a small number of examples from previous tasks.
result A simple baseline outperforms CL approaches with episodic memory.