We introduce the community exploration problem that has many real-world applications such as online advertising. In the problem, an explorer allocates limited budget to explore communities so as to maximize the number of members he could meet. We provide a systematic study of the community exploration problem, from off…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Online Exploration”
A new method combines online and offline learning to tackle contextual bandits with missing action support.
problem Learning optimal policies with logged data when the logging policy has deficient support.
method Hybrid approach using online exploration to exploit supported actions and offline learning to avoid unnecessary explorations.
result Determines an optimal policy with theoretical guarantees using minimal online explorations.
Maximize to Explore integrates RL components for efficient policy discovery.
problem Balancing exploration and exploitation in online RL with general function approximators.
method Integrates estimation, planning, and exploration into a single objective function.
result Achieves sublinear regret for MDPs and MGs with general function approximations.
SUPE combines unlabeled data with RL to efficiently explore tasks.
problem Efficient exploration in reinforcement learning with sparse rewards.
method Extract low-level skills using VAE, pseudo-label unlabeled data, and use as off-policy data for online RL.
result SUPE outperforms prior methods across 42 long-horizon tasks.
Interactive user interfaces need to continuously evolve based on the interactions that a user has (or does not have) with the system. This may require constant exploration of various options that the system may have for the user and obtaining signals of user preferences on those. However, such an exploration, especiall…
Algorithm reduces online regret by leveraging offline data in linear bandits.
problem Online regret minimization in linear bandits with offline data.
method OOPE algorithm using extended D-optimal design.
result Substantial reduction in online regret compared to prior work.
Batch Thompson Sampling reduces exploration-exploitation trade-off in online decision making.
problem Balancing exploration and exploitation in online decision making.
method Introducing a batch Thompson Sampling framework for stochastic multi-arm bandit and linear contextual bandit problems.
result Achieves asymptotic regret bound with batch queries, significantly reducing interactions.
New approach uses unlabeled prior data to accelerate exploration in sparse reward tasks.
problem Sparse reward tasks in reinforcement learning.
method Learn reward model from online experience, label prior data, and use concurrently.
result Rapid exploration in challenging sparse-reward domains.
Improves RLHF sample efficiency by scaling reward complexity polynomially.
problem Exponential sample complexity in RLHF algorithms for skewed preferences.
method SE-POPO, an online RLHF algorithm that achieves polynomial sample complexity.
result SE-POPO outperforms existing algorithms in sample efficiency.
Algorithm balances online and offline data for linear bandits.
problem Online learning with an offline dataset in linear bandits.
method Proposes a linear bandit algorithm that uses offline data early and increasingly favors exploration as the horizon grows.
result Establishes regret bounds showing competitive performance with both purely online and offline solutions.
Q-chunking improves RL for long tasks by chunking actions.
problem Improving sample efficiency and exploration in offline-to-online RL.
method Action chunking in TD-based RL methods.
result Q-chunking outperforms prior methods on long-horizon tasks.
New -Coverage objective simplifies exploration in reinforcement learning.
problem Challenges in exploration for high-dimensional domains.
method Introduces -Coverage objective to enable efficient exploration and planning.
result First computationally efficient algorithms for online reinforcement learning with low coverability.
Develops a method to estimate optimal policy value in online learning.
problem Challenges in evaluating ongoing policies in online learning environments.
method Doubly Robust Interval Estimation (DREAM) method.
result Valid inference on online conditional mean estimator with asymptotically normal distribution.
New algorithm fills gaps in offline data for hybrid RL, achieving similar gains without coverage assumptions.
problem Lack of provable benefits in hybrid RL with coverage assumptions.
method Warm-starting optimistic online algorithms with offline data in experience replay buffer.
result Hybrid RL gains similar to offline-only RL without coverage assumptions, demonstrating efficient exploration.
New algorithm learns optimal exploration parameters for contextual bandits.
problem Learning optimal exploration in contextual bandits.
method Proposes two algorithms that learn optimal exploration parameters online based on context and reward.
result Demonstrates improved performance in learning optimal exploration compared to traditional methods.
New algorithm uses density ratios for efficient online reinforcement learning.
problem Challenges in collecting exploratory data for online reinforcement learning.
method Density ratio modeling for online exploration, combining truncation and optimism.
result Sample-efficient online exploration achieved with GLOW and HyGLOW.
New method for efficient online exploration in RLHF reduces regret.
problem Efficiently collecting new preference data in RLHF to refine reward model and policy.
method Proposes a new exploration scheme that directs preference queries toward reducing uncertainty in reward differences most relevant to policy improvement.
result Establishes regret bounds of order for online RLHF, with polynomial scaling in all model parameters.
Paper proposes an -policy gradient for online pricing, reducing regret to .
problem Online pricing learning task
method Combines model-based and model-free reinforcement learning, using -greedy with gradient descent.
result Achieves expected regret of order .
A new algorithm improves stochastic linear bandit performance using residual bootstrap.
problem Improving performance in stochastic linear bandit problems.
method Residual bootstrap exploration to estimate mean reward and pull the arm with the highest estimate.
result Proposed algorithm exttt{LinReBoot} achieves high-probability sub-linear regret under mild conditions.
Proposes a new algorithm for efficient online model selection of LLMs considering the increasing-then-converging trend.
problem Balancing cost and performance in choosing the best LLM among a diverse set of models.
method Introduces a time-increasing bandit algorithm (TI-UCB) that predicts model performance increases and balances exploration and exploitation.
result Achieves a logarithmic regret upper bound, indicating a fast convergence rate in model selection.
Bayesian model for energy-efficient EV navigation.
problem Limited battery capacity in electric vehicles.
method Bayesian modeling and online learning framework with exploration strategies.
result Established rigorous regret bounds for Thompson Sampling.
Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcs.LG
XPO enhances RLHF by encouraging diverse responses, offering improved sample efficiency.
problem Limited exploration in RLHF leads to suboptimal models and bottlenecks.
method XPO is a simple one-line change to DPO, introducing a novel exploration bonus.
result XPO achieves strong theoretical guarantees and promising empirical performance.
Greedy policy maximizes information in unknown linear systems.
problem Exploration in unknown linear dynamical systems.
method Online greedy policy maximizing information.
result Competitive performance compared to gradient-based methods.
New insights show coverage conditions are crucial for efficient online reinforcement learning.
problem The role of coverage conditions in determining sample complexity of offline reinforcement learning.
method Established a connection between coverage conditions and sample efficiency in online reinforcement learning.
result Coverability, a structural property of MDPs, enables sample-efficient exploration in online reinforcement learning.
This paper uses bandit algorithms to reduce the cost of user interface experimentation in online retail.
problem Reducing the cost of user interface experimentation in online retail.
method Modeling user interface experimentation as an opportunistic bandit problem, reducing the cost of exploration.
result Significant regret reduction and improved contextual information for testing.
Study explores cyberbullying datasets and classifier generalization.
problem Cyberbullying detection across different online communities.
method Analysis of various cyberbullying datasets and classifier generalization.
result Insight into classifier generalization power and interaction.
RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST) for PO with mediator feedback.
problem Policy Optimization in continuous control tasks.
method RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST) for regret minimization in PO.
result Achieving constant regret under certain circumstances in PO with mediator feedback.
NROWAN-DQN: A Stable Noisy Network with Noise Reduction and Online Weight Adjustment for Explorationcs.LG
NROWAN-DQN improves stability and exploration in noisy networks.
problem Noisy networks struggle with stable exploration in complex tasks.
method Noise reduction and online weight adjustment for stable actions.
result NROWAN-DQN outperforms prior algorithms in stability and exploration.
Study online RL with mismatched dynamics, achieving sublinear regret.
problem Exploration challenges in online RL with mismatched training and deployment dynamics.
method Introduce supremal visitation ratio, propose efficient algorithm with -divergence.
result Achieves sublinear regret in online RMDPs with optimal dependence on supremal visitation ratio and interaction episodes.
In this paper, we study the problem of safe online learning to re-rank, where user feedback is used to improve the quality of displayed lists. Learning to rank has traditionally been studied in two settings. In the offline setting, rankers are typically learned from relevance labels created by judges. This approach has…
Adaptive and sequential experiment design is a well-studied area in numerous domains. We survey and synthesize the work of the online statistical learning paradigm referred to as multi-armed bandits integrating the existing research as a resource for a certain class of online experiments. We first explore the tradition…
New algorithms for efficient learning with partial information, reducing regret.
problem Online learning with partial observability and semi-bandit feedback.
method Implicit exploration strategy for near-optimal regret guarantees.
result First algorithms with near-optimal regret guarantees without knowing the observation system.
Optimizes recommender selection online with D-optimal design.
problem Finding the optimal recommender in online exploration-exploitation.
method Leverages D-optimal design from statistics to maximize information gain.
result Achieves maximum information gain during online exploration.
New approach uses autoregressive models to explore and quantify uncertainty in decision-making.
problem Quantifying and exploring uncertainty in online decision-making.
method Reformulates uncertainty as missing future outcomes, training autoregressive models for next-outcome prediction.
result Establishes a reduction from online learning to offline next-outcome prediction, controlling Bayesian regret by sequence prediction loss.
Paper proposes efficient online estimation of causal effects by deciding which data sources to query.
problem Data fusion problems with multiple data sources capturing distinct subsets of variables.
method Online moment selection (OMS) framework, balancing exploration and exploitation.
result OMS algorithms achieve zero asymptotic regret for estimating average treatment effects.
MADE improves exploration in RL by maximizing deviation from explored regions.
problem Efficient exploration in high-dimensional RL tasks with sparse rewards.
method Proposes a new exploration approach via maximizing the deviation of the occupancy of the next policy from explored regions, adding it as an adaptive regularizer to the RL objective.
result Significantly improves sample efficiency in navigation and locomotion tasks.
Study on sample complexity for pure exploration in feedback graph settings.
problem Sample complexity of pure exploration in online learning with feedback graphs.
method Derive instance-specific lower bounds and present asymptotically optimal algorithm TaS-FG.
result TaS-FG is asymptotically optimal and efficient across different graph configurations.
Study on computable online learning with new conditions and complexities.
problem Characterizing optimal online learning under varying optimality requirements.
method Introduced anytime optimal (a-optimal) online learning and explored computational separations.
result Found a computational separation between a-optimal and optimal online learning.
Central to robot exploration and mapping is the task of persistent localization in environmental fields characterized by spatially correlated measurements. This paper presents a Gaussian process localization (GP-Localize) algorithm that, in contrast to existing works, can exploit the spatially correlated field measurem…
AttViz offers online visualizations of neural language model attention mechanisms.
problem Limited interpretability of neural language models.
method Online toolkit for exploring self-attention mechanisms.
result Visualizations help understand model decision-making.
Optimal online data collection for semiparametric inference reduces regret.
problem Sequential data collection decisions for efficient estimation under budget constraints.
method Online Moment Selection framework; Explore-then-Commit and Explore-then-Greedy policies.
result Online data collection policies achieve zero regret relative to an oracle policy.
The paper tackles fair sharing of exploration costs across groups in online learning.
problem Sharing the cost of exploration fairly across multiple groups in online learning.
method The paper introduces the 'grouped' bandit model and uses axiomatic bargaining theory, specifically the Nash bargaining solution, to formalize fairness.
result The paper derives policies that are optimally fair and regret-optimal, showing that regret-optimal policies can be unfair.
This paper explores how imperfect reward models can improve online RLHF.
problem Sample efficiency in online RLHF from imperfect reward models.
method Identifies policy coverability and proposes TPO algorithm for transfer learning.
result TPO algorithm improves sample efficiency compared to standard online learning.
Two new algorithms improve online clustering of bandits by accelerating cluster identification without strong assumptions.
problem Challenges in accurately identifying unknown user clusters in online bandit settings.
method Proposes UniCLUB and PhaseUniCLUB algorithms with enhanced exploration mechanisms.
result Achieves comparable regret bounds to prior work with weaker assumptions.
Having a sequence-to-sequence model which can operate in an online fashion is important for streaming applications such as Voice Search. Neural transducer is a streaming sequence-to-sequence model, but has shown a significant degradation in performance compared to non-streaming models such as Listen, Attend and Spell (…
Bayesian reflex models AI learning like the autonomic nervous system.
problem Online learning in dynamic AI environments.
method Bayesian online algorithms with belief maintenance, sequential updating, and uncertainty-driven action balancing.
result Unified framework for adaptive AI learning.
This work tackles representation learning for RL in low-rank MDPs, improving sample efficiency.
problem How to learn a compact representation for RL in low-rank MDPs efficiently.
method Proposes REP-UCB for online RL and develops an algorithm for offline RL under partial coverage.
result Significantly improved sample complexity for online RL and competitive performance for offline RL.
We propose a plan online and learn offline (POLO) framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local traj…