Reinforcement learning mimics expert behavior.
problem Learning from expert demonstrations in reinforcement learning.
method Reduction to reinforcement learning with a stationary reward.
result Expert reward can be recovered and imitation learning is bounded.
PWIL learns agent behavior from expert using Wasserstein distance.
problem Learning agent behavior from expert demonstrations efficiently.
method Primal Wasserstein Imitation Learning (PWIL) with offline reward function.
result PWIL achieves expert behavior on MuJoCo tasks with minimal interactions.
Paper tackles covariate shift in offline IL using less proficient behavior data.
problem Mitigating covariate shift in Imitation Learning with limited data coverage.
method Model-based IL from Offline Data (MILO) framework.
result MILO can combat covariate shift even with sub-optimal behavior policy data.
KL-regularized RL from expert demos can lead to slow, unstable learning.
problem Pathological training dynamics in KL-regularized RL from expert demonstrations.
method Empirical analysis and non-parametric behavioral reference policies.
result KL-regularized RL can be significantly improved by using non-parametric behavioral policies.
New model learns better policies from expert demonstrations with higher efficiency.
problem Learning accurate policies from expert demonstrations with high efficiency.
method Generative adversarial imitation learning (GAIL) model that learns f-divergence automatically. result Learns better policies with higher data efficiency in physics-based control tasks.
We propose a framework for ensuring safe behavior of a reinforcement learning agent when the reward function may be difficult to specify. In order to do this, we rely on the existence of demonstrations from expert policies, and we provide a theoretical framework for the agent to optimize in the space of rewards consist…
Learning from demonstration (LfD) is the process of building behavioral models of a task from demonstrations provided by an expert. These models can be used e.g. for system control by generalizing the expert demonstrations to previously unencountered situations. Most LfD methods, however, make strong assumptions about …
The paper proposes a method to explain expert decisions by modeling preferences with 'what if' outcomes.
problem Interpreting and auditing decision-making policies in institutions.
method Integrating counterfactual reasoning into batch inverse reinforcement learning.
result The method effectively recovers accurate and interpretable descriptions of expert behavior.
The paper provides theoretical guarantees for behavior cloning using generative models.
problem Behavior cloning of complex expert demonstrations using generative models.
method The paper proposes a theoretical framework invoking low-level controllers to stabilize imitation around expert demonstrations. It shows that with suitable low-level stability guarantees and powerful generative models, pure supervised behavior cloning can match expert trajectories.
result The paper proves that with a suitable low-level stability guarantee and a powerful enough generative model, pure supervised behavior cloning can generate trajectories matching the per-time step distribution of essentially arbitrary expert trajectories in an optimal transport cost.
In this paper we propose a novel gradient algorithm to learn a policy from an expert's observed behavior assuming that the expert behaves optimally with respect to some unknown reward function of a Markovian Decision Problem. The algorithm's aim is to find a reward function such that the resulting optimal policy matche…
Enhances BO with expert preferences about abstract properties.
problem Lack of expert knowledge in BO for black-box experimental design.
method Human-AI collaboration to incorporate expert preferences into surrogate modeling.
result Superior performance compared to baselines in synthetic and real-world datasets.
Compressed imitation learning uses simplicity priors for efficient expert behavior copying.
problem Efficiently learn expert behaviors with minimal data.
method Utilizes policy simplicity as a prior for sample-efficient imitation learning.
result Significantly higher scores achieved with limited expert demonstrations.
Recent work on imitation learning has generated policies that reproduce expert behavior from multi-modal data. However, past approaches have focused only on recreating a small number of distinct, expert maneuvers, or have relied on supervised learning techniques that produce unstable policies. This work extends InfoGAI…
In this paper we demonstrate how genetic algorithms can be used to reverse engineer an evaluation function's parameters for computer chess. Our results show that using an appropriate expert (or mentor), we can evolve a program that is on par with top tournament-playing chess programs, outperforming a two-time World Com…
FinHEAR combines LLMs with human expertise for better financial decision-making.
problem Challenges in financial decision-making for language models.
method Multi-agent framework with specialized LLMs for historical analysis, event interpretation, and expert retrieval.
result FinHEAR outperforms baselines in financial tasks with higher accuracy and risk-adjusted returns.
New method recovers diverse policies from expert data using state-action pair weighting.
problem Recovering diverse policies from expert trajectories.
method Pointwise mutual information weighted behavioral cloning.
result Effective in focusing on state-action pairs most representative of the style.
An assistant learns to mediate decisions between humans and experts, balancing risk and learning.
problem Learning to mediate decisions between imperfect humans and expert knowledge.
method Formalizes online decision mediation, proposes a policy to balance immediate loss and future generalization.
result Consistent gains over benchmarks in decision-making performance.
Advances in the field of inverse reinforcement learning (IRL) have led to sophisticated inference frameworks that relax the original modeling assumption of observing an agent behavior that reflects only a single intention. Instead of learning a global behavioral model, recent IRL methods divide the demonstration data i…
Bayesian models combine experts with a flexible gating mechanism for complex data.
problem Theoretical properties of Bayesian mixture-of-experts models with softmax gating remain unexplored.
method Investigated asymptotic behavior of posterior distribution for density estimation, parameter estimation, and model selection.
result Established posterior contraction rates for density estimation and parameter estimation, providing insights for practical model design.
Combines BC and GAIL for efficient imitation learning.
problem Efficient imitation learning without reward signals.
method Integrates Behavior Cloning and Generative Adversarial Imitation Learning.
result Combination leads to stable and sample-efficient learning.
This work improves Gaussian process inference using mixtures of experts and nested SMC samplers.
problem High computational and memory costs of Gaussian processes.
method Mixtures of Gaussian process experts with nested SMC samplers.
result Significantly improved inference compared to importance sampling.
New insights into the top-K sparse softmax gating function for deep learning.
problem Understanding the theoretical effects of the top-K sparse softmax gating function on density and parameter estimations.
method Using a Gaussian mixture of experts, novel loss functions, and theoretical analysis.
result The convergence rates of density and parameter estimations are parametric under certain conditions, but slow under over-specified models.
Model explains capital allocation and wealth distribution dynamics in a frictional economy.
problem Understanding capital allocation and wealth distribution dynamics in a frictional economy.
method Mean-field game approach to model interactions between expert and household groups.
result Experts accumulate capital during booms and quickly reverse behavior in busts, even without macro-shocks.
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
Improves imitation learning in RL by learning reward function efficiently.
problem Lack of effective reward function approximation in AIRL for imitation tasks.
method Proposes Off-Policy AIRL that combines adversarial learning with efficient reward function approximation.
result Shows superior imitation performance and efficiency compared to state-of-the-art AIL algorithms.
Making the right decision in traffic is a challenging task that is highly dependent on individual preferences as well as the surrounding environment. Therefore it is hard to model solely based on expert knowledge. In this work we use Deep Reinforcement Learning to learn maneuver decisions based on a compact semantic st…
Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficul…
DSAC improves robustness of imitation learning.
problem Learning from few expert data and distribution shift.
method DSAC uses adversarial reward function.
result DSAC performs well on PyBullet environments.
Imitation learning is an effective alternative approach to learn a policy when the reward function is sparse. In this paper, we consider a challenging setting where an agent and an expert use different actions from each other. We assume that the agent has access to a sparse reward function and state-only expert observa…
We consider a problem of learning the reward and policy from expert examples under unknown dynamics. Our proposed method builds on the framework of generative adversarial networks and introduces the empowerment-regularized maximum-entropy inverse reinforcement learning to learn near-optimal rewards and policies. Empowe…
Non-experts have long made important contributions to machine learning (ML) by contributing training data, and recent work has shown that non-experts can also help with feature engineering by suggesting novel predictive features. However, non-experts have only contributed features to prediction tasks already posed by e…
A new method for learning to defer decisions with expert advice improves over standard methods.
problem Learning to defer decisions with expert advice in systems where expert information can be modified after selection.
method An augmented surrogate that operates on the composite expert-advice action space, providing consistency guarantees and excess-risk bounds.
result The method improves over standard Learning-to-Defer and adapts its advice acquisition behavior to the cost regime.
While most approaches to the problem of Inverse Reinforcement Learning (IRL) focus on estimating a reward function that best explains an expert agent's policy or demonstrated behavior on a control task, it is often the case that such behavior is more succinctly represented by a simple reward combined with a set of hard…
Novel CNN-based gaze scanpath comparison distinguishes experts from novices in dental radiograph interpretation.
problem Distinguishing expertise in dental radiograph interpretation based on gaze behavior.
method Convolutional neural networks (CNN) process scene information at the fixation level, using image patches as input to compare gaze scanpaths.
result 93% accuracy in distinguishing experts from novices using image patch features.
Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated…
Develops statistical framework for resolving reward function ambiguity in inverse reinforcement learning.
problem Non-uniqueness of reward functions in inverse reinforcement learning.
method Entropy regularization combined with least-squares reconstruction of the reward from the soft Bellman residual.
result Least-squares reward function is unique and consistent with the expert policy.
This paper investigates a financial market where returns depend on an unobservable Gaussian drift process. While the observation of returns yields information about the underlying drift, we also incorporate discrete-time expert opinions as an external source of information. For estimating the hidden drift it is crucial…
Investigates least squares estimation in deterministic MoE models.
problem Complexity and difficulty in analyzing MoE models.
method Least squares estimation under deterministic MoE models.
result Rates of estimating strongly identifiable experts are faster than polynomial experts.
Model predicts human food choices based on demographics.
problem Predicting human food choices from demographic data.
method Non-deterministic model based on NHANES dataset and behavioral studies.
result Generates synthetic data similar to original dataset.
Study analyzes convergence of parameter estimation in contaminated mixture of experts.
problem Challenges in learning from prompts in large-scale models.
method Convergence analysis, distinguishability condition, partial differential equations.
result Comprehensive convergence rates and minimax lower bounds for parameter estimation.
Survey of methods for learning from observation without requiring expert actions.
problem Lack of access to expert actions in imitation learning.
method Survey and classification of state-only imitation learning (SOIL) methods.
result Identification of open problems and future research directions.
We present a novel approach for supervised domain adaptation that is based upon the probabilistic framework of Gaussian processes (GPs). Specifically, we introduce domain-specific GPs as local experts for facial expression classification from face images. The adaptation of the classifier is facilitated in probabilistic…
Imitation learning seeks to learn an expert policy from sampled demonstrations. However, in the real world, it is often difficult to find a perfect expert and avoiding dangerous behaviors becomes relevant for safety reasons. We present the idea of \textit{learning to avoid}, an objective opposite to imitation learning …
CoCoAFusE fuses expert predictions to model complex patterns with interpretability and uncertainty.
problem Modeling complex patterns with interpretability and uncertainty quantification.
method Competitive/Collaborative Fusion of Experts (CoCoAFusE) that fuses expert distributions in addition to mixing.
result CoCoAFusE avoids multimodality artifacts and provides tighter credible bounds on the response variable.
Analyzes convergence rates for Gaussian-gated MoE model.
problem Theoretical understanding of Gaussian-gated MoE model is incomplete.
method Maximum likelihood estimation with novel Voronoi loss functions.
result MLE has distinct behaviors under different settings of Gaussian gating function parameters.
ICIL learns policies invariant to multiple environments, improving generalization.
problem Learning policies from multiple environments leads to spurious correlations.
method ICIL learns invariant feature representations and a matching imitation policy.
result ICIL policies generalize better to unseen environments.
Continuous control imitation learning fails if expert actions are smooth.
problem Continuous control imitation learning fails if expert actions are smooth.
method Study of imitation learning in discrete-time, continuous state-and-action control systems.
result Any smooth, deterministic imitator policy suffers exponentially larger error than the expert.
On a daily investment decision in a security market, the price earnings (PE) ratio is one of the most widely applied methods being used as a firm valuation tool by investment experts. Unfortunately, recent academic developments in financial econometrics and machine learning rarely look at this tool. In practice, fundam…