New method uses feedback to improve deep reinforcement learning efficiency.
problem Vast amounts of data needed for reasonable performance in deep reinforcement learning.
method Binary corrective feedback combined with probabilistic conditional exploration.
result Achieves drastic improvements in sample efficiency and robustness.
A bandit algorithm reduces regret in noisy, communication-constrained feedback.
problem Distributed stochastic multi-armed bandit with noisy, communication-constrained feedback.
method Proposes a multi-phase bandit algorithm, UE-UCB++, that matches an information-theoretic lower bound.
result Matches an information-theoretic lower bound of Ω(√(KT/σ²)) on the minimax regret.
New model for personalized online advertising with multi-user interaction.
problem Realistic online advertising scenarios with multiple users interacting simultaneously.
method Introduces Multi-User Contextual Cascading Bandit (MCCB) model and proposes UCBBP and AUCBBP algorithms.
result Proves UCBBP and AUCBBP achieve optimal regret bounds for multi-user context.
The paper enhances a virtual assistant's humor to improve user satisfaction.
problem Improving a virtual assistant's ability to deliver humorous responses.
method Combines traditional NLP techniques with self-attentional networks and multi-task learning, using implicit feedback for labeling.
result Deep-learning models outperform heuristic methods in real-world user satisfaction.
Paper tackles noisy bandit feedback for multiclass classification.
problem Learning multiclass classifier with corrupted feedback.
method Proposes an unbiased estimator technique to estimate noise rates and an end-to-end framework.
result Algorithm achieves mistake bounds of O ( T ) O(\sqrt{T}) O ( T ) in high noise and O ( T i c e f r a c 23 ) O(T^{
icefrac{2}{3}}) O ( T i ce f r a c 2 3 ) in worst case. User preferences for items can be inferred from either explicit feedback, such as item ratings, or implicit feedback, such as rental histories. Research in collaborative filtering has concentrated on explicit feedback, resulting in the development of accurate and scalable models. However, since explicit feedback is oft…
Learning weights in a spiking neural network with hidden neurons, using local, stable and online rules, to control non-linear body dynamics is an open problem. Here, we employ a supervised scheme, Feedback-based Online Local Learning Of Weights (FOLLOW), to train a network of heterogeneous spiking neurons with hidden l…
Curious Meta-Controller alternates between model-based and model-free control to improve sample efficiency.
problem Combining the benefits of model-based and model-free control to enhance sample efficiency.
method Adaptive alternation between model-based and model-free control using curiosity feedback.
result Significantly improved sample efficiency and near-optimal performance on robotic tasks.
TransCF improves recommendation by modeling user-item relationships with translation vectors.
problem Triangle inequality violation in matrix factorization-based recommendation methods.
method TransCF uses translation vectors to model latent user-item relationships in implicit feedback.
result TransCF outperforms state-of-the-art methods by up to 17% in hit ratio.
In a wide variety of applications, including personalization, we want to measure the difference in outcome due to an intervention and thus have to deal with counterfactual inference. The feedback from a customer in any of these situations is only 'bandit feedback' - that is, a partial feedback based on whether we chose…
A new algorithm for conversational recommendation systems using dueling bandits in GLMs.
problem Limited user feedback in existing conversational bandit methods.
method Integrates dueling bandits with relative feedback in generalized linear models.
result Theoretical and empirical validation of ConDuel's efficacy.
Modeling alignment as resource-limited cognitive processes, researchers derive performance bounds.
problem Systematic deviations in feedback-based alignment of large language models.
method Modeling alignment as a two-stage cascade U o H o Y U o H o Y U oH o Y given S S S , with cognitive and total capacities. result Capacity-coupled Alignment Performance Interval derived from Fano and PAC-Bayes bounds.
Work shows hallucination detection by LLMs is impossible without expert feedback.
problem Detecting hallucinations in LLMs is theoretically impossible without expert-labeled feedback.
method Investigated hallucination detection using a theoretical framework inspired by language identification.
result Automated hallucination detection is impossible for most language collections without expert-labeled feedback.
Paper tackles invoice line item matching in P2P processes using agent feedback.
problem Matching product/service descriptions in invoices with purchase orders.
method Two approaches using agent feedback data: similarity ranking and classification.
result Proposed approaches outperform benchmarks and real-world data sets.
BYOL-Explore learns to explore visually-rich environments by predicting world dynamics.
problem Exploration in visually complex environments.
method Optimizes a single prediction loss in latent space to learn world representation, dynamics, and exploration policy.
result Achieves superhuman performance on Atari games with simpler design.
A new multi-objective RL framework improves intrinsic exploration performance.
problem Sub-optimal exploration performance due to ad-hoc handling of intrinsic exploration.
method A multi-objective RL framework where both exploration and exploitation are optimized as separate objectives.
result EMU-Q method outperforms classic and other intrinsic RL methods on benchmarks.
ISL algorithm tackles deep exploration efficiently.
problem Deep exploration in reinforcement learning.
method Derives ISL algorithm by augmenting RL objective with a novel regularization term.
result Empirically shows state-of-the-art performance on deep-exploration benchmarks.
EBE enables efficient exploration in reinforcement learning.
problem Inefficient exploration in reinforcement learning.
method Entropy-based exploration (EBE) quantifies learning and adaptively explores unexplored regions.
result EBE enables faster learning without hyperparameter tuning.
This work learns latent representations to speed up exploration in complex environments.
problem Challenging exploration in high-dimensional state and action spaces with sparse rewards.
method Representation learning using prior experience to learn effective latent representations.
result Learned latent representations reduce the dimensionality of the search space for effective exploration.
Go-Explore improves performance on hard-exploration problems in Atari games.
problem Challenges in reinforcement learning, especially with sparse or deceptive rewards.
method Exploits principles of remembering states, returning to promising states, and solving simulated environments.
result Scores significantly higher than previous state-of-the-art on Montezuma's Revenge and Pitfall.
R3L uses planning algorithms to efficiently explore sparse reward environments.
problem Balancing exploration and exploitation in sparse reward reinforcement learning.
method Formulate exploration as a search problem using RRT, leverage demonstrations from initial solutions to refine RL policy.
result R3L outperforms classic and intrinsic exploration techniques, requiring fewer samples and achieving better asymptotic performance.
DE via conjugate policies improves exploration and policy performance.
problem Effective exploration in policy gradient methods.
method DE via conjugate policies.
result DE improves policy performance and exploration effectiveness.
Study incentivizes exploration in non-stationary MAB with compensation.
problem Incentivized exploration for non-stationary stochastic bandits with biased feedback.
method Proposed algorithms for abruptly-changing and continuously-changing non-stationary environments.
result Achieves sublinear regret and compensation over time.
New exploration bonuses improve reinforcement learning efficiency.
problem Efficient exploration in unknown environments with limited feedback.
method Improved exploration bonuses scaling with 1/n and improved stopping time analysis.
result Faster learning rates and improved sample complexity in pure-exploration settings.
New findings on when to use action space exploration in reinforcement learning.
problem Understanding when to use action space exploration over traditional methods.
method Theoretical analysis and empirical testing of simple exploration methods.
result Exploration in action space is preferred when parametric complexity exceeds action space dimensionality and horizon length.
Proposes a method to avoid excessive exploration in reinforcement learning.
problem Avoiding excessive exploration in reinforcement learning to deploy it in practice.
method Designs a novel algorithm using UCB reinforcement learning policy with adaptive exploration constraints.
result Proves that the approach remains conservative while minimizing regret in tabular settings and validates on real-world tasks.
New algorithm reduces regret by allowing free exploration in multi-armed bandits.
problem Designing an adaptive policy to minimize regret with a free exploration budget.
method Introduced ( α , β ) (α,β) ( α , β ) -probably saving policies and a two-phase algorithm UFE-KLUCB-H. result UFE-KLUCB-H accumulates strictly less regret than non-free exploration policies.
New meta-RL method avoids exploration-exploitation trade-off.
problem Learning to explore and exploit simultaneously in meta-RL.
method Developed new objectives for exploration and exploitation.
result DREAM outperforms existing methods on complex tasks.
The paper tackles pure exploration in multi-armed bandits with low rank structure using oblivious sampling.
problem Pure exploration in multi-armed bandits with low rank reward sequences.
method The approach involves separating the exploration strategy from feedback, using oblivious sampling, and incorporating kernel information of reward vectors.
result Efficient algorithms with regret bound O ( d ( ln N ) / n ) O(d\sqrt{(\ln N)/n}) O ( d ( ln N ) / n ) for both time-varying and fixed cases, with a lower bound gap of O ( ln N ) O(\sqrt{\ln N}) O ( ln N ) . The paper evaluates various bonus-based exploration methods in the ALE and finds limited improvement in performance.
problem Improving exploration in reinforcement learning algorithms, especially in challenging games.
method Empirical evaluation of different reward bonuses on the Arcade Learning Environment.
result Recently developed bonus-based exploration methods do not significantly improve performance in challenging games.
MAME models a separate exploration policy for faster adaptation.
problem Efficient exploration strategies for quick task adaptation in meta-reinforcement learning.
method Explicitly models a separate exploration policy for task distribution, using self-supervised or supervised learning objectives for adaptation.
result Superior performance compared to prior works in meta-reinforcement learning.
Regularization-induced exploration improves contextual bandit performance.
problem Complex reward models in real-world contextual bandits are hard to explore effectively.
method Regularization-induced exploration using stochasticity in cross-validation.
result Regularization-induced exploration leads to reliable exploration in large-scale business environments.
The Exploration-Exploitation tradeoff arises in Reinforcement Learning when one cannot tell if a policy is optimal. Then, there is a constant need to explore new actions instead of exploiting past experience. In practice, it is common to resolve the tradeoff by using a fixed exploration mechanism, such as ε ε ε -greedy ex…
We introduce the community exploration problem that has many real-world applications such as online advertising. In the problem, an explorer allocates limited budget to explore communities so as to maximize the number of members he could meet. We provide a systematic study of the community exploration problem, from off…
The paper proposes an algorithm to learn efficient and effective exploration policies in reinforcement learning.
problem Balancing exploration and exploitation in reinforcement learning.
method Formalized a counterfactual metric for exploration utility and used meta-learning to learn an end-to-end exploration policy.
result Demonstrated improved performance in high-dimensional control tasks in MuJoCo simulator compared to previous methods.
MADE improves exploration in RL by maximizing deviation from explored regions.
problem Efficient exploration in high-dimensional RL tasks with sparse rewards.
method Proposes a new exploration approach via maximizing the deviation of the occupancy of the next policy from explored regions, adding it as an adaptive regularizer to the RL objective.
result Significantly improves sample efficiency in navigation and locomotion tasks.
Efficient exploration is one of the key challenges for reinforcement learning (RL) algorithms. Most traditional sample efficiency bounds require strategic exploration. Recently many deep RL algorithms with simple heuristic exploration strategies that have few formal guarantees, achieve surprising success in many domain…
Maximizes Rényi entropy for efficient exploration in reward-free RL.
problem Challenges of exploration in reward-free reinforcement learning.
method Maximizes Rényi entropy over state-action space in exploration phase; uses batch RL for planning phase.
result Effective and sample-efficient exploration leading to superior policies.
An agent explores indefinitely in an environment with unlimited rewards.
problem Balancing exploration and exploitation in environments with unlimited rewards.
method Simple example of an environment with unbounded rewards and optimal agent behavior.
result An optimal agent always explores to maximize rewards, regardless of accumulated knowledge.
Proposes EVE for efficient exploration in reinforcement learning.
problem Efficient exploration in reinforcement learning.
method EVE: a recipe for posterior over parameters, facilitating efficient exploration.
result Competitive performance on benchmarks, efficient exploration confirmed.
VASE uses Bayesian neural networks to improve exploration in sparse reward environments.
problem Exploration in environments with continuous control and sparse rewards.
method VASE uses a Bayesian neural network model of the environment dynamics and variational inference to alternately update the model's accuracy and policy.
result VASE outperforms other surprise-based exploration techniques in continuous control sparse reward environments.
ASE safely explores unknown MDPs with unknown dynamics, improving sample efficiency.
problem Balancing exploration and safety in unknown MDPs with stochastic dynamics.
method Exploits analogies between state-action pairs to safely learn near-optimal policies.
result Empirically improves sample efficiency compared to existing methods.
This work uses model uncertainty for efficient exploration in sparse reward environments.
problem Challenging exploration in sparse reward reinforcement learning environments.
method Implicit generative modeling approach to estimate Bayesian uncertainty of the agent's belief of the environment dynamics.
result Our implicit generative model consistently outperforms competing approaches in data efficiency for exploration.
Achieving efficient and scalable exploration in complex domains poses a major challenge in reinforcement learning. While Bayesian and PAC-MDP approaches to the exploration problem offer strong formal guarantees, they are often impractical in higher dimensions due to their reliance on enumerating the state-action space.…
Directed exploration improves reinforcement learning efficiency and robustness.
problem Achieving good sample efficiency in reinforcement learning with efficient exploration.
method Directed exploration through goal-conditioned policies that are independent of uncertainty.
result Directed exploration is more efficient and robust to uncertainty than reward bonuses.
Two new exploration methods for multi-agent systems improve team performance.
problem Exploration in transition-dependent multi-agent settings.
method EITI and EDTI, using mutual information and VoI to encourage coordinated exploration.
result Significant improvement in multi-agent performance through coordinated exploration.
AdaLinUCB optimizes exploration-exploitation for contextually varying costs.
problem Optimizing decision-making in environments with varying exploration costs.
method Adaptive Upper-Confidence-Bound (AdaLinUCB) algorithm for opportunistic learning.
result AdaLinUCB achieves O((log T)^2) regret bound, significantly outperforming other algorithms.
This paper proposes a self-supervised exploration method using disagreement of dynamics models.
problem Efficient exploration in stochastic environments with real robots.
method Train an ensemble of dynamics models and incentivize exploration to maximize disagreement.
result Sample-efficient exploration achieved without external rewards or reinforcement learning.