CVRL tackles complex visual observations in reinforcement learning.
problem Complex visual observations in natural environments.
method Contrastive Variational Reinforcement Learning (CVRL) learns a contrastive variational model by maximizing mutual information between latent states and observations.
result CVRL achieves comparable performance with state-of-the-art model-based DRL methods and significantly outperforms them on tasks with complex observations.
Observational learning is a type of learning that occurs as a function of observing, retaining and possibly replicating or imitating the behaviour of another agent. It is a core mechanism appearing in various instances of social learning and has been found to be employed in several intelligent species, including humans…
New framework learns policies for partially observable systems.
problem Learning policies in partially observable dynamical systems.
method Partially Observable Bilinear Actor-Critic framework.
result Algorithm can learn against optimal policies in certain cases.
New algorithm improves reinforcement learning from partial observations.
problem Inferior performance of algorithms in real-world reinforcement learning due to partial observability.
method Representation-based approach to POMDPs, leading to a tractable algorithm.
result Empirically demonstrates superior performance with partial observations.
Uncorrelated optical space observation association represents a classic needle in a haystack problem. The objective being to find small groups of observations that are likely of the same resident space objects (RSOs) from amongst the much larger population of all uncorrelated observations. These observations being pote…
A new algorithm CAP learns optimal policies from observational data with confounding bias and missing observations.
problem Offline contextual bandit with confounding bias and missing observations.
method CAP policy learning, forming reward function as solution of integral equation system, building confidence set, and greedily taking action with pessimism.
result Developed an upper bound to the suboptimality of CAP for the offline contextual bandit problem.
Efficient RL in partially observable risk-sensitive environments with hindsight observations.
problem Risk-sensitive reinforcement learning in partially observable environments.
method Integrates hindsight observations into POMDP framework, develops novel RL algorithm.
result Achieves polynomial regret with provable efficiency, outperforming existing methods.
Deep Reinforcement Learning (RL) recently emerged as one of the most competitive approaches for learning in sequential decision making problems with fully observable environments, e.g., computer Go. However, very little work has been done in deep RL to handle partially observable environments. We propose a new architec…
Algorithm improves imitation learning from visual data in partially observable environments.
problem Imitation learning from visual observations with missing expert actions and partial observability.
method Theoretical analysis and Latent Adversarial Imitation from Observations algorithm combining adversarial and latent representations.
result Latent Adversarial Imitation from Observations achieves state-of-the-art performance in high-dimensional robotic tasks.
New framework for reinforcement learning with sporadic state observations.
problem Partial observability in reinforcement learning.
method Action-Triggered Sporadically Traceable Markov Decision Processes (ATST-MDPs).
result Optimistic algorithm achieving regret bound for episodic learning.
New algorithms learn POMDPs efficiently with hindsight observability.
problem Hardness of learning in POMDPs due to partial observability.
method Hindsight Observable Markov Decision Process (HOMDP) and new algorithms for tabular and function approximation settings.
result Sample-efficient learning in POMDPs with optimal dependence on latent state and observation cardinalities.
New algorithm learns POMDPs without computational oracles.
problem Learning near-optimal policies in POMDPs with computationally hard oracles.
method Quasipolynomial-time algorithm using barycentric spanners for policy covers.
result First oracle-free learning algorithm for observable POMDPs.
Intelligent agents can cope with sensory-rich environments by learning task-agnostic state abstractions. In this paper, we propose an algorithm to approximate causal states, which are the coarsest partition of the joint history of actions and observations in partially-observable Markov decision processes (POMDP). Our m…
New algorithm proves RL from partial obs is feasible.
problem Difficulty in learning from partial observability.
method Optimism combined with MLE for weakly revealing POMDPs.
result Simple algorithm guarantees polynomial sample efficiency.
We consider the problem of diagnosis where a set of simple observations are used to infer a potentially complex hidden hypothesis. Finding the optimal subset of observations is intractable in general, thus we focus on the problem of active diagnosis, where the agent selects the next most-informative observation based o…
The paper teaches robots to navigate by learning costs from expert demonstrations.
problem Teaching robots to navigate autonomously using only expert observations.
method Developed a map encoder and cost encoder to infer semantic class probabilities and a cost function from expert observations.
result Robots can learn to follow traffic rules in a simulator using only semantic observations.
Method learns dynamics from noisy partial observations.
problem Reconstructing stochastic dynamical systems from indirect noisy data.
method Amortized path generation method for nonlinear stochastic filtering.
result Learned conditional path generator quantifies uncertainty.
DPFRL uses particle filters for decision making with complex visual observations.
problem Decision making with partial complex visual observations.
method Discriminative Particle Filter Reinforcement Learning (DPFRL) with a differentiable particle filter in the neural network policy.
result DPFRL outperforms state-of-the-art POMDP RL models in complex visual observation tasks.
In reinforcement learning, we can learn a model of future observations and rewards, and use it to plan the agent's next actions. However, jointly modeling future observations can be computationally expensive or even intractable if the observations are high-dimensional (e.g. images). For this reason, previous works have…
RL struggles with generalization due to implicit partial observability.
problem Generalization in RL is difficult due to implicit partial observability.
method Re-cast RL problem as solving epistemic POMDPs and propose ensemble-based techniques.
result Simple ensemble-based technique achieves significant generalization gains.
The paper develops algorithms for competitive RL in partially observable MGs.
problem Challenges in reinforcement learning with function approximation and partial observability.
method Proposes posterior sampling methods for self-play and adversarial learning in zero-sum MGs.
result Developed algorithms achieve low regret bounds scaling sublinearly with GEC and episode number.
Improves active learning efficiency by warping input space based on observed outputs.
problem Insensitivity of Gaussian process uncertainty to actual observations.
method Input warping with learned monotone reparameterization to adjust acquisition function behavior.
result Significantly improved sample efficiency across various benchmarks, especially in non-stationary conditions.
New algorithm learns optimal decisions from imperfectly observed contexts.
problem Learning optimal decisions in bandits with unobserved contexts.
method Posterior sampling algorithm for imperfectly observed contexts.
result Efficient learning from noisy imperfect observations.
Physics-constrained deep learning predicts geophysical dynamics with boundedness.
problem Forecasting geophysical systems with hidden variables and incomplete observations.
method Physics-constrained neural ordinary differential equation (NODE) representations with boundedness constraints.
result The approach generalizes learned dynamics to arbitrary initial conditions.
Study efficient reinforcement learning for partially observed systems with linear structure.
problem Efficient reinforcement learning for partially observed Markov decision processes with linear structure.
method Proposes OP-TENET algorithm using a Bellman operator with finite memory, adversarial integral equation, and optimistic exploration.
result Achieves ε-optimal policy within O(1/ε^2) episodes with polynomial sample complexity in intrinsic dimension.
Controlled interventions provide the most direct source of information for learning causal effects. In particular, a dose-response curve can be learned by varying the treatment level and observing the corresponding outcomes. However, interventions can be expensive and time-consuming. Observational data, where the treat…
A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to de…
A new approach to unsupervised learning using recognition-parametrised models.
problem Discovering meaningful latent structure in observational data.
method Recognition-Parametrised Model (RPM) combining parametric and non-parametric components.
result Effective learning of latent structure without explicit generative models.
Combines observational and randomized data to estimate treatment effects.
problem Estimating heterogeneous treatment effects using only observational data is biased.
method Two-step framework: learn shared structure from observational data, then data-specific structures from randomized data.
result Combining observational and randomized data improves treatment effect estimation.
iTimER learns from reconstruction errors to represent irregularly sampled time series.
problem Learning from irregularly sampled time series with missing data.
method iTimER models reconstruction errors as a proxy for unobserved values, using a mixup strategy and a Wasserstein metric.
result iTimER outperforms state-of-the-art methods in classification, interpolation, and forecasting tasks.
Improves LSTM performance by initializing states via manifold learning.
problem Improving LSTM performance through better initialization.
method Learning an intrinsic data manifold to initialize LSTM internal states.
result Improved LSTM performance through consistent initialization.
Paper tackles reinforcement learning with complex observations and simple latent dynamics.
problem Understanding reinforcement learning with complex observations and simple latent dynamics.
method Statistical and algorithmic analysis of reinforcement learning under general latent dynamics.
result Identifies latent pushforward coverability as a condition for statistical tractability.
Method models other agents' behaviors without requiring direct observation.
problem Understanding and interacting effectively with other agents in reinforcement learning.
method Extracts representations from local observations of the controlled agent using encoder-decoder architectures.
result The method achieves higher returns than baseline methods in multi-agent environments.
In many cases an intelligent agent may want to learn how to mimic a single observed demonstrated trajectory. In this work we consider how to perform such procedural learning from observation, which could help to enable agents to better use the enormous set of video data on observation sequences. Our approach exploits t…
Enhanced feedback model improves sample-efficiency in POMDPs.
problem Exponential hardness of learning in POMDPs.
method Multiple observations in hindsight feedback model.
result Sample-efficient learning possible for new subclasses of POMDPs.
We propose and study a new model for reinforcement learning with rich observations, generalizing contextual bandits to sequential decision making. These models require an agent to take actions based on observations (features) with the goal of achieving long-term performance competitive with a large set of policies. To …
Temporal-difference (TD) networks are a class of predictive state representations that use well-established TD methods to learn models of partially observable dynamical systems. Previous research with TD networks has dealt only with dynamical systems with finite sets of observations and actions. We present an algorithm…
Method estimates observation functions in state-space models without supervision.
problem Unsupervised learning of non-invertible observation functions in nonlinear state-space models.
method Nonparametric generalized moment method using constrained regression.
result Estimates function space of identifiability from state process.
DOVI improves reinforcement learning with offline data, reducing trial-and-error in critical scenarios.
problem Lack of sample efficiency in deep reinforcement learning for critical applications.
method Proposes DOVI algorithm to incorporate confounded observational data provably efficiently.
result DOVI reduces regret by a multiplicative factor compared to pure online setting, especially when data are informative.
A new approach predicts next observations without explicit decoding for better control.
problem High-dimensional observations and unknown dynamics in real-world control tasks.
method Proposes a novel information-theoretic LCE approach using predictive coding to develop a decoder-free model.
result The model reliably learns a controllable latent space leading to superior performance.
Paper establishes baselines for offline RL from visual observations.
problem Challenges in offline reinforcement learning from visual observations with continuous action spaces.
method Simple baselines and benchmarking tasks for offline RL from visual observations.
result Simple modifications to existing online RL algorithms outperform existing offline RL methods.
The paper extends multiple instance learning to multiclass and regression problems.
problem Learning from aggregate observations where supervision is given to sets of instances.
method Probabilistic framework for various aggregate observations, including classification and regression.
result The proposed estimator has nice convergence properties under mild assumptions.
This paper advances sample-efficient learning for partially observable RL by introducing B-stability and new algorithms.
problem Hard sample complexity for learning near-optimal policies in partially observable RL.
method Proposes B-stability as a unified structural condition and develops new algorithms for sample-efficient learning.
result Any B-stable PSR can be learned with polynomial samples, improving over current best complexities.
We study the problem of learning influence functions under incomplete observations of node activations. Incomplete observations are a major concern as most (online and real-world) social networks are not fully observable. We establish both proper and improper PAC learnability of influence functions under randomly missi…
New method identifies causal variables from partially observed data.
problem Learning from unpaired observations with instance-dependent partial observability.
method Proposes two methods enforcing sparsity in the inferred representation.
result Establishes two identifiability results for linear and piecewise linear mixing functions.
ContraBAR uses contrastive learning to learn Bayes-optimal policies in RL.
problem Learning optimal policies for unknown tasks sampled from a known distribution.
method Proposes ContraBAR, a meta RL algorithm using contrastive predictive coding (CPC) for belief inference.
result ContraBAR achieves comparable performance to state-of-the-art methods and is computationally efficient.
We study Imitation Learning (IL) from Observations alone (ILFO) in large-scale MDPs. While most IL algorithms rely on an expert to directly provide actions to the learner, in this setting the expert only supplies sequences of observations. We design a new model-free algorithm for ILFO, Forward Adversarial Imitation Lea…
Stable Hadamard Memory improves reinforcement learning by efficiently managing memory.
problem Memory models struggle in partially observable reinforcement learning environments.
method Introduces a novel memory model using the Hadamard product for efficient memory management and updates.
result Significantly outperforms state-of-the-art memory-based methods on challenging benchmarks.