This paper investigates how to efficiently transition and update policies, trained initially with demonstrations, using off-policy actor-critic reinforcement learning. It is well-known that techniques based on Learning from Demonstrations, for example behavior cloning, can lead to proficient policies given limited data…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Behavioral Cloning”
The paper provides theoretical guarantees for behavior cloning using generative models.
problem Behavior cloning of complex expert demonstrations using generative models.
method The paper proposes a theoretical framework invoking low-level controllers to stabilize imitation around expert demonstrations. It shows that with suitable low-level stability guarantees and powerful generative models, pure supervised behavior cloning can match expert trajectories.
result The paper proves that with a suitable low-level stability guarantee and a powerful enough generative model, pure supervised behavior cloning can generate trajectories matching the per-time step distribution of essentially arbitrary expert trajectories in an optimal transport cost.
This paper provides theoretical foundations for using quantized actions in behavior cloning.
problem Applying autoregressive models to continuous control requires discretizing actions through quantization, which is poorly understood.
method The paper analyzes quantization error propagation and statistical sample complexity, and proposes model-based augmentation.
result Behavior cloning with quantized actions achieves optimal sample complexity, matching existing lower bounds.
Behavior cloning training instabilities amplified by SGD noise over long horizons.
problem Training instabilities in behavior cloning with deep neural networks.
method Empirical dissection of minibatch SGD updates and their effects on long-horizon rewards.
result Exponential moving average (EMA) of iterates effectively mitigates gradient variance amplification (GVA).
We address the problem of imitation learning with multi-modal demonstrations. Instead of attempting to learn all modes, we argue that in many tasks it is sufficient to imitate any one of them. We show that the state-of-the-art methods such as GAIL and behavior cloning, due to their choice of loss function, often incorr…
Behavior cloning can achieve horizon-independent sample complexity in offline imitation learning.
problem Sample complexity in imitation learning increases with problem horizon.
method New analysis of behavior cloning with logarithmic loss.
result Behavior cloning can achieve linear dependence on horizon in offline IL under dense rewards.
New technique improves imitation learning by preventing local minima and exploring states.
problem Behavioral cloning gets stuck in local minima and lacks effective exploration.
method Two-phase model with sampling mechanisms and self-attention modules.
result Significantly outperforms previous state-of-the-art in various environments.
The majority of cancer treatments end in failure due to Intra-Tumor Heterogeneity (ITH). ITH in cancer is represented by clonal evolution where different sub-clones compete with each other for resources under conditions of Darwinian natural selection. Predicting the growth of these sub-clones within a tumour is among t…
BCO* improves BCO by concurrently training inverse dynamics and expert policy.
problem Efficiently learn from unlabeled demonstrations without requiring many initial interactions.
method Introduce BCO* that concurrently trains an inverse dynamics model and expert policy.
result BCO* eliminates the need for initial interactions and improves sample complexity.
Imitation learning trains a policy from expert demonstrations. Imitation learning approaches have been designed from various principles, such as behavioral cloning via supervised learning, apprenticeship learning via inverse reinforcement learning, and GAIL via generative adversarial learning. In this paper, we propose…
Compressed imitation learning uses simplicity priors for efficient expert behavior copying.
problem Efficiently learn expert behaviors with minimal data.
method Utilizes policy simplicity as a prior for sample-efficient imitation learning.
result Significantly higher scores achieved with limited expert demonstrations.
New method recovers diverse policies from expert data using state-action pair weighting.
problem Recovering diverse policies from expert trajectories.
method Pointwise mutual information weighted behavioral cloning.
result Effective in focusing on state-action pairs most representative of the style.
Action chunking and data exploration improve behavior cloning in robotics.
problem Exponential errors in learning from demonstrations for continuous control tasks.
method Action chunking and exploratory data collection.
result Control-theoretic stability is key to improving imitation learning.
Paper introduces DNTs to clone black-box models efficiently.
problem Cloning functionality of black-box models.
method Deep Neural Trees (DNTs) trained with active learning.
result Trained DNT can clone task-specific behavior of black-box models.
Interactive IL beats BC by state-wise annotation cost.
problem Behavior Cloning struggles with annotation cost in sequential decision making.
method Proved Stagger and Warm Stagger algorithms to outperform BC.
result Interactive and hybrid IL methods outperform BC with state-wise annotation.
It is difficult to be able to imitate well in unknown states from a small amount of expert data and sampling data. Supervised learning methods such as Behavioral Cloning do not require sampling data, but usually suffer from distribution shift. The methods based on reinforcement learning, such as inverse reinforcement l…
New method uses bi-level optimization to learn useful representations for imitation learning.
problem Learning useful representations for multiple tasks in imitation learning settings.
method Formulates representation learning as a bi-level optimization problem.
result Bi-level optimization framework provides sample complexity benefits for imitation learning.
New algorithm robustly learns from corrupted demonstrations, even with constant fraction of noise.
problem Learning from corrupted demonstrations where a fraction of data is noise or outliers.
method Proposes a novel robust algorithm using a Median-of-Means (MOM) objective.
result Guarantees accurate policy estimation even with constant fraction of outliers, similar to classical methods in expert demonstration settings.
Paper tackles covariate shift in offline IL using less proficient behavior data.
problem Mitigating covariate shift in Imitation Learning with limited data coverage.
method Model-based IL from Offline Data (MILO) framework.
result MILO can combat covariate shift even with sub-optimal behavior policy data.
The study classifies and imitates trading agents in financial markets.
problem Classifying and imitating trading agents in continuous double auctions.
method Developed an agent-based model for trading, applied opponent modeling for classification, and used behavioral cloning for imitation.
result Techniques for classification and imitation were experimentally compared and evaluated.
Imitation learning is the problem of recovering an expert policy without access to a reward signal. Behavior cloning and GAIL are two widely used methods for performing imitation learning. Behavior cloning converges in a few iterations but doesn't achieve peak performance due to its inherent iid assumption about the st…
Demonstration-Regularized RLstat.ML
Demonstration-regularized RL reduces sample complexity for policy identification.
problem Improving reinforcement learning's sample efficiency with expert demonstrations.
method KL-regularization of behavior cloning using expert demonstrations.
result Demonstration-regularized RL achieves optimal policy identification with reduced sample complexity.
New method learns robot skills from data, matching or outperforming existing methods.
problem Learning robot skills from fixed datasets.
method Offline Reinforcement Learning via Supervised Learning using implicit models.
result Implicit models can match or outperform explicit models in acquiring robotic skills.
ORIL learns a reward function from unlabeled data to improve robot learning.
problem Leveraging unlabeled data for robot learning.
method ORIL learns a reward function from demonstrator and unlabeled trajectories, annotates data, and trains an agent via offline reinforcement learning.
result ORIL consistently outperforms BC agents on various robotic tasks.
Unified framework for policy learning using weak supervision.
problem High-quality supervision is often infeasible or expensive in practice.
method Treat weak supervision as imperfect peer information and evaluate policies based on correlated agreement.
result Substantial performance improvements, especially in complex or noisy environments.
Paper tackles offline RL from mixed datasets with adaptive KL regularizer.
problem Challenges in optimizing RL and BC signals with varying action coverage and multiple action modes.
method Adaptively weighted reverse KL divergence regularizer based on TD3 algorithm.
result Empirically outperforms existing offline RL algorithms in MuJoCo locomotion tasks.
A simple approach to offline RL without additional complexity.
problem Learning from a fixed dataset of actions with value estimation errors.
method Adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data.
result Matches the performance of state-of-the-art offline RL algorithms with minimal changes.
Improves RL from historical data by stitching trajectories.
problem Lack of high-quality data for offline RL.
method Trajectory Stitching (TS) to augment historical data with synthetic actions.
result Improves RL policy performance over baseline.
Diffusion models mimic human actions in sequential tasks.
problem Cloning human behavior in dynamic environments is challenging.
method Adapting diffusion models to handle stochastic, multimodal, and correlated actions.
result Diffusion models closely replicate human behavior in robotic and gaming tasks.
The susceptibility of deep learning to adversarial attack can be understood in the framework of the Renormalisation Group (RG) and the vulnerability of a specific network may be diagnosed provided the weights in each layer are known. An adversary with access to the inputs and outputs could train a second network to clo…
New approach mitigates feedback divergence in imitation learning.
problem Divergence between held-out error and learner performance in imitation learning.
method Identifies covariate shift as the root cause and proposes a simulator-based solution.
result Naive behavioral cloning performs well in real-world decision making problems.
This thesis proposes a derivatives hedging framework using deep learning and reinforcement learning.
problem Traditional hedging models fail in complex, uncertain markets due to assumptions like continuous trading and zero transaction costs.
method Integrates deep learning and reinforcement learning, using a spatiotemporal attention-based Transformer for probabilistic forecasting and hedging.
result The proposed method significantly outperforms traditional approaches in U.S. and Chinese financial markets.
MAZE attacks a model without data, using synthetic inputs.
problem Model Stealing attacks that require data to replicate a target model.
method MAZE uses zeroth-order gradient estimation to create synthetic inputs.
result MAZE achieves high clone accuracy without data.
New RL method learns value function for many policies using few key states.
problem Evaluate and improve policies in continuous control problems.
method Combines actor-critic architecture and policy embedding to learn a single value function for many policies.
result Value function minimizes prediction error by learning a small set of 'probing states' and their impact on policies' returns.
Comma.ai's approach to Artificial Intelligence for self-driving cars is based on an agent that learns to clone driver behaviors and plans maneuvers by simulating future events in the road. This paper illustrates one of our research approaches for driving simulation. One where we learn to simulate. Here we investigate v…
AI agent plays CSGO deathmatch with human-like style.
problem Lack of API for CSGO limits data for reinforcement learning.
method Behavioural cloning on large noisy and expert datasets.
result Matches human difficulty level in deathmatch mode.
Deep Neural Networks (DNNs) are susceptible to model stealing attacks, which allows a data-limited adversary with no knowledge of the training dataset to clone the functionality of a target model, just by using black-box query access. Such attacks are typically carried out by querying the target model using inputs that…
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Samplingcs.LG
Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce a notion of conservati…
GDT improves reinforcement learning by matching future state information efficiently.
problem Efficient learning of multi-task policies from trajectory data.
method Generalized Decision Transformer (GDT) for offline hindsight information matching.
result GDT enables effective offline multi-task state-marginal matching and imitation learning.
The paper establishes theoretical foundations for low-rank knowledge distillation in LLMs.
problem Understanding the theoretical underpinnings of low-rank knowledge distillation in LLMs.
method Theoretical framework for low-rank knowledge distillation, including convergence rates and generalization bounds.
result Theoretical analysis reveals optimal rank for minimizing generalization error.
Study uses machine learning to predict potato clones suitable for processing.
problem Efficiently identifying high-yield, disease-resistant potato varieties.
method Leveraged machine learning algorithms on Russet potato clones data from Oregon.
result Non-linear models like SVM and HGBC outperform traditional linear models in agricultural trials.
NewtonianVAE: Proportional Control and Goal Identification from Pixels via Physical Latent Spacescs.LG
Framework simplifies vision-based control and goal discovery.
problem Learning proportional control from visual data.
method Introduces NewtonianVAE for proportional control and goal discovery.
result Dramatic simplification and acceleration of vision-based controllers.
Behavioral cloning reduces policy learning to supervised learning by training a discriminative model to predict expert actions given observations. Such discriminative models are non-causal: the training procedure is unaware of the causal structure of the interaction between the expert and the environment. We point out …
A contraction analysis improves model-based RL's error recovery.
problem Theoretical understanding of model-based reinforcement learning.
method Contraction analysis applied to both stochastic and deterministic state transitions.
result Error reduction in cumulative reward using branched rollouts.
FedCD improves non-IID federated learning performance.
problem Learning from non-IID data in federated learning.
method FedCD dynamically groups devices with similar data.
result FedCD achieves higher accuracy and faster convergence.
New groups constructed from tree automorphisms, proving finiteness.
problem Finiteness properties of self-similar and Röver groups.
method Braiding of self-similar groups, use of -ary cloning systems, analysis of embedded disks.
result Construction of a new group (braided Röver group) of type .
Deep reinforcement learning has led to several recent breakthroughs, though the learned policies are often based on black-box neural networks. This makes them difficult to interpret and to impose desired specification constraints during learning. We present an iterative framework, MORL, for improving the learned polici…
Model-free reinforcement learning (RL) requires a large number of trials to learn a good policy, especially in environments with sparse rewards. We explore a method to improve the sample efficiency when we have access to demonstrations. Our approach, Backplay, uses a single demonstration to construct a curriculum for a…