Proposes learning latent reward model for planning from rewards.
problem Planning in high-dimensional state spaces with limited reward information.
method Directly learns a latent dynamics model from rewards, planning in latent state-space.
result Successfully learns accurate latent reward prediction model, achieving strong performance and high sample efficiency.
A novel approach for safe offline RL using latent safety constraints.
problem Balancing safety constraints and reward maximization in offline RL.
method Conditional Variational Autoencoders for latent safety modeling, Constrained Reward-Return Maximization.
result Our approach maintains safety compliance while optimizing rewards, outperforming existing methods.
Mitigates biases in reward models using variational inference.
problem Spurious correlations in reward models that align large language models with human preferences.
method Formulates data-generating process, identifies non-spurious latent variables, and uses variational inference to recover them.
result Effective mitigation of spurious correlation issues, yielding more robust reward models.
A new learning method for prosthetic arms without explicit rewards.
problem Learning a prosthetic arm to interact with users without explicit reward signals.
method Interaction-Grounded Learning, observing multidimensional context and feedback vectors, discovering latent reward signal.
result The algorithm can discover a latent reward signal and ground its policies for successful interaction.
SARA uses similarity to learn rewards robustly and adaptively.
problem Robustness to labeler errors and adaptability to diverse feedback formats.
method Contrastive framework that learns latent representations and computes rewards as similarities.
result Strong performance on offline RL benchmarks and diverse applications.
Adversarial RL recovers agent rewards from financial market data simulations.
problem Recovering agent rewards in volatile financial markets with unknown dynamics.
method Adversarial inverse reinforcement learning in latent space simulations.
result Adversarial RL can robustly recover agent rewards from latent space representations of real market data.
New algorithm tackles nonstationary linear bandits with latent dynamics.
problem Nonstationary bandit problem with latent states and unknown dynamics.
method Explore-then-commit algorithm with exploration and commitment phases.
result Achieves i l d e O ( T 2 / 3 ) ilde{\mathcal{O}}(T^{2/3}) i l d e O ( T 2/3 ) regret. A method to train multi-agent reinforcement learning models without intrinsic rewards.
problem Training multi-agent reinforcement learning models with sparse rewards is challenging.
method A learning-based exploration strategy using variational graph autoencoder to generate initial states.
result The method improves the training and performance of multi-agent reinforcement learning models.
Paper introduces Latent-CLIP for efficient text-image comparison in latent space.
problem Efficiently compare text and images in latent space without costly decoding.
method Trains CLIP model in latent space, uses Latent-CLIP rewards for noise optimization, and guides generation away from harmful content.
result Latent-CLIP matches CLIP performance on text-image classification and harmful content detection.
Algorithm learns near-optimal policies for reward-mixing MDPs with few latent contexts.
problem Episodic reinforcement learning in reward-mixing Markov decision processes with a few latent contexts.
method Sample-efficient algorithm EM^2 using higher-order method-of-moments approach.
result Provides an ε-optimal policy using O(ε^(-2) * S^d A^d * poly(H, Z)^d) episodes for arbitrary M ≥ 2.
We decode latent states in Block MDPs and learn near-optimal policies.
problem Model estimation and reward-free learning in Block MDPs.
method Information-theoretical lower bound and efficient model estimation algorithm.
result Our algorithm approaches the information-theoretical limit for latent state decoding and converges to optimal policies.
New framework tackles stochastic latent subgroup heterogeneity in online decision-making.
problem Stochastic latent heterogeneity in online decision-making where individual responses vary with unobserved subgroups.
method Latent heterogeneous bandit framework using EM-greedy algorithm to learn subgroup probabilities and reward parameters.
result Achieves optimal estimation and classification guarantees, revealing a fundamental stochastic barrier in online decision-making.
Direct approach for handling contextual bandits with latent state dynamics.
problem Handling contextual bandits with latent state dynamics, especially when rewards depend on posterior probabilities of hidden states.
method Direct reduction to standard linear contextual bandits, extended analysis of HMM parameters, periodic update of reward-model parameters.
result Periodic update of reward-model parameters allows handling complex dependencies in hidden states.
LaTRO optimizes latent reasoning in LLMs without external reward.
problem Training LLMs to perform complex reasoning tasks.
method Formulates reasoning as latent distribution sampling and optimizes via variational approaches.
result LLMs improve reasoning and evaluation quality through self-improvement.
RL agents learn from a few tasks to generalize to new ones.
problem Creating efficient RL agents that can solve multiple tasks.
method GHP-MDPs model with latent variables for hidden parameters.
result State-of-the-art performance and sample-efficiency on new tasks.
Algorithm maximizes user rewards under per-item budget constraints.
problem Maximizing cumulative rewards in collaborative bandits with budget constraints.
method Collaborative algorithm B-LATTICE that clusters users and collaborates across groups.
result Achieves sub-linear regret bounds matching minimax bounds.
Optimal algorithm for latent bandits with cluster structure reduces regret to nearly optimal.
problem Maximizing cumulative rewards in a multi-armed bandit problem with latent clusters.
method LATTICE algorithm exploiting cluster structure and arm information.
result Minimax optimal regret of O ( ( M + N ) T ) O(\sqrt{(\mathsf{M}+\mathsf{N})\mathsf{T}}) O ( ( M + N ) T ) with O ( log T ) O(\log{\mathsf{T}}) O ( log T ) calls to matrix completion oracle. New method boosts skill learning by encouraging optimistic exploration.
problem Intrinsic reward for exploration is inherently pessimistic.
method Derive an information gain auxiliary objective involving an ensemble of discriminators and rewarding policy disagreement.
result Improves skill learning in grid worlds and Atari games.
New method for LLMs to learn reasoning by optimizing latent variables.
problem Teaching LLMs to generate logical justifications for answers.
method Formalized reasoning as latent variable model, derived FEM objective, designed sampling schemes.
result Prompt Posterior Sampling (PPS) outperforms other schemes in learning to reason.
Inverse reinforcement learning (IRL) is used to infer the reward function from the actions of an expert running a Markov Decision Process (MDP). A novel approach using variational inference for learning the reward function is proposed in this research. Using this technique, the intractable posterior distribution of the…
New algorithm learns causal graph to minimize regret in bandits without full structure.
problem Learning optimal decisions in bandits with unknown causal graph and latent confounders.
method Two-stage approach: first learns ancestors and necessary confounders, second applies standard bandit algorithm.
result No full causal structure needed for optimal decisions; only necessary confounders are crucial.
This research enhances exploration in DDPG using latent trajectory optimization.
problem Limited exploration in DDPG with deterministic policies.
method Model-based trajectory optimization for exploration in DDPG, using a learned deep dynamics model.
result Improved performance in continuous control tasks, especially with sparse rewards and images.
Planning has been very successful for control tasks with known environment dynamics. To leverage planning in unknown environments, the agent needs to learn the dynamics from interactions with the world. However, learning dynamics models that are accurate enough for planning has been a long-standing challenge, especiall…
New algorithms use offline data to improve online decision-making with latent states.
problem Accelerating online sequential decision-making with latent states in offline data.
method Design end-to-end latent bandit algorithms for linear latent contextual bandits, learning latent subspace offline and using it online.
result Proves minimax optimal regret guarantees for online algorithms and practical efficiency.
Unified approach to correlated multi-armed bandits reduces regret significantly.
problem Correlated rewards in multi-armed bandits.
method Developed a unified approach to leverage reward correlations and presented algorithms with rigorous analysis.
result C-UCB algorithm pulls non-competitive arms only O(1) times, improving over classic algorithms.
Method tackles uncertainty in reward models for LLMs from heterogeneous human feedback.
problem Uncertainty in reward models for LLMs from heterogeneous human feedback.
method Heterogeneous preference framework and alternating gradient descent algorithm.
result Established theoretical guarantees for estimator convergence and asymptotic distribution.
A method for self-supervised representation learning in partially observable environments.
problem Sparse rewards and stochasticity in partially observable environments.
method World model in latent space for estimating missing information.
result Significant improvement in exploration compared to prior work.
Providing a suitable reward function to reinforcement learning can be difficult in many real world applications. While inverse reinforcement learning (IRL) holds promise for automatically learning reward functions from demonstrations, several major challenges remain. First, existing IRL methods learn reward functions f…
Unified Latent Dynamics unifies model-free and model-based reinforcement learning.
problem Combining the efficiency of model-free methods with the representational strengths of model-based approaches.
method Embedding state-action pairs into a latent space where the true value function is approximately linear, using synchronized updates of encoder, value, and policy networks.
result ULD achieves cross-domain competence with minimal tuning and a fraction of the parameter footprint.
Trajectory-level supervision allows efficient offline reinforcement learning.
problem Offline reinforcement learning
method Developing a statistical theory for offline policy optimization from trajectory-level labels
result Proving a high-probability guarantee of order O ~ ( H 2 C s a ( π ⋆ ) / n ) \widetilde O(H^2\sqrt{C_{sa}(\pi^\star)/n}) O ( H 2 C s a ( π ⋆ ) / n ) Hierarchical MARL learns complementary skills for team coordination.
problem Creating intelligent agents that coordinate like human sports teams.
method Two-level hierarchical MARL with unsupervised skill discovery.
result Emergence of useful and complementary skills in team games.
Meta-learner infers subtask graph to adapt quickly to unknown tasks.
problem Adapting to unknown tasks with unknown subtask dependencies in few-shot RL.
method Meta-learner with Subtask Graph Inference (MSGI) and UCB-inspired intrinsic reward.
result Accurately infers latent task parameters and adapts more efficiently.
New algorithms tackle latent bandit problems with lower regret.
problem Learning optimal actions in a latent state setting.
method General algorithms based on UCBs and Thompson sampling, aware of model uncertainty.
result Our methods have lower regret than classic bandit policies when latent states are fewer than actions.
Exploration is an extremely challenging problem in reinforcement learning, especially in high dimensional state and action spaces and when only sparse rewards are available. Effective representations can indicate which components of the state are task relevant and thus reduce the dimensionality of the space to explore.…
Motivated by online recommendation and advertising systems, we consider a causal model for stochastic contextual bandits with a latent low-dimensional confounder. In our model, there are L L L observed contexts and K K K arms of the bandit. The observed context influences the reward obtained through a latent confounder var…
PROWL uses robust reward estimates to improve ITR selection.
problem Reward uncertainty in ITR estimation leads to inflated performance.
method PAC-Bayesian framework with reward uncertainty certificates.
result PROWL achieves better robust treatment regime estimation.
New method optimizes policies without assuming known link functions between preferences and rewards.
problem Policy alignment with unknown and unrestricted link functions.
method Formulates an f f f -divergence-constrained reward maximization problem, learning policies directly. result Induces a semiparametric single-index binary choice model for policy alignment.
Director learns hierarchical behaviors from pixels, outperforming exploration methods.
problem Challenges in learning long-term hierarchical behaviors from pixels.
method Directly learns hierarchical behaviors from pixels using a latent space world model.
result Director outperforms exploration methods on various tasks.
New approach to meta-learning with variational Bayes for unlabeled data.
problem Improving machine learning systems' adaptability to small data.
method Generative meta-learning using AEVB and mean-field variational Bayes.
result Relevant VB updates do not depend on generative neural networks for certain models.
Combining model-based and model-free deep reinforcement learning has shown great promise for improving sample efficiency on complex control tasks while still retaining high performance. Incorporating imagination is a recent effort in this direction inspired by human mental simulation of motor behavior. We propose a lea…
Bayesian method improves reinforcement learning efficiency by encouraging exploration.
problem High sample complexity due to inefficient exploration.
method Bayesian linear regression and latent space embedding for intrinsic reward.
result Significant improvement in sample complexity on various tasks.
We introduce a method by which a generative model learning the joint distribution between actions and future states can be used to automatically infer a control scheme for any desired reward function, which may be altered on the fly without retraining the model. In this method, the problem of action selection is reduce…
Study noisy rewards in online decision-making with unknown distributions.
problem Learning optimal decisions in online settings with noisy and unknown reward distributions.
method Proposes algorithms integrating learning and decision-making via LCB thresholding.
result Achieves competitive ratios of 1 - 1/e and 1/2 in various settings.
BYOL-Explore learns to explore visually-rich environments by predicting world dynamics.
problem Exploration in visually complex environments.
method Optimizes a single prediction loss in latent space to learn world representation, dynamics, and exploration policy.
result Achieves superhuman performance on Atari games with simpler design.
A new estimator reduces variance in slate bandit OPE.
problem Large action spaces in slate bandits cause high variance in OPE.
method Develops Latent IPS (LIPS) to optimize slate abstractions for low variance and bias.
result LIPS substantially outperforms existing estimators in scenarios with non-linear rewards and large slate spaces.
Many reinforcement learning (RL) tasks provide the agent with high-dimensional observations that can be simplified into low-dimensional continuous states. To formalize this process, we introduce the concept of a DeepMDP, a parameterized latent space model that is trained via the minimization of two tractable losses: pr…
Optimal control in latent factor models uses Tsallis entropy for exploration.
problem Optimal control in models with latent factors.
method Reward exploration with Tsallis entropy and derive q q q -Gaussian distribution over states. result Optimal policy derived in a model-agnostic setting.
This study proposes hidden state curiosity to enhance RL models' resilience against noise.
problem Curiosity traps in RL models distract agents from discovering novel experiences.
method Proposed hidden state curiosity based on the Free Energy Principle to reward agents for KL divergence between predictive priors and posteriors.
result Agents with hidden state curiosity are more resilient against curiosity traps compared to those with prediction error curiosity.