Two hitherto disconnected threads of research, diverse exploration (DE) and maximum entropy RL have addressed a wide range of problems facing reinforcement learning algorithms via ostensibly distinct mechanisms. In this work, we identify a connection between these two approaches. First, a discriminator-based diversity …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “maximum entropy reinforcement learning”
Proof of convergence for multi-objective optimization using inverse reinforcement learning.
problem Proving convergence in multi-objective optimization problems.
method Wasserstein inverse reinforcement learning with projective subgradient method and gradient descent.
result Convergence of inverse reinforcement learning for multi-objective optimization.
Enhances RL by controlling policy stochasticity through trajectory entropy constraints.
problem Non-stationary Q-value estimation and short-sighted entropy tuning in maximum entropy RL.
method Proposes TECRL framework with separate Q-functions for reward and entropy, enforcing a trajectory entropy constraint.
result DSAC-E algorithm achieves higher returns and better stability on OpenAI Gym benchmarks.
We introduce Implicit Policy, a general class of expressive policies that can flexibly represent complex action distributions in reinforcement learning, with efficient algorithms to compute entropy regularized policy gradients. We empirically show that, despite its simplicity in implementation, entropy regularization c…
The paper introduces a new intrinsic reward method for exploration in reinforcement learning.
problem Improving exploration in reinforcement learning agents.
method Intrinsic rewards proportional to the entropy of future state-action features.
result The new objective leads to improved visitation of features within individual trajectories.
Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errorscs.LG
A new method reduces compounding errors in model-based reinforcement learning.
problem Compounding errors in long horizon predictions from model-based reinforcement learning.
method Maximum Entropy Model Rollouts (MEMR) with non-uniform sampling and prioritized experience replay.
result Significantly reduces computation requirements compared to other model-based methods.
MaxEnt Model Correction improves reinforcement learning model accuracy.
problem Improving reinforcement learning model accuracy and convergence.
method MaxEnt Model Correction (MoCo) procedure to correct model's next-state distributions.
result MoCoVI and MoCoDyna algorithms converge faster and effectively use approximate models.
Improved exploration methods for reinforcement learning with reduced sample complexity.
problem Challenges in reinforcement learning exploration in unknown environments.
method Proposed game-theoretic and trajectory entropy algorithms with improved sample complexity.
result Established statistical advantage of entropy-regularized MDPs for exploration and reduced sample complexity.
This work extends ME-RL using diffusion models to sample optimal policies.
problem Sampling from the optimal policy trajectory distribution in ME-RL.
method Introducing Diffusion-Augmented Markov Decision Processes (DA-MDPs) to minimize reverse KL divergence.
result DA-MDPs enable seamless integration into various ME-RL methods and outperform baselines.
We consider the problem of learning from demonstrated trajectories with inverse reinforcement learning (IRL). Motivated by a limitation of the classical maximum entropy model in capturing the structure of the network of states, we propose an IRL model based on a generalized version of the causal entropy maximization pr…
Improves reinforcement learning extrapolation in Gridworlds.
problem Generalizing to unseen states in reinforcement learning.
method Avoiding deterministic action choice, using ego-centric representation, incorporating symmetry, and adding an entropy term.
result Significant improvement in extrapolation performance in Gridworld environments.
New algorithm reduces performance loss in IRL with mismatched transition dynamics.
problem Performance degradation in inverse reinforcement learning due to mismatched transition dynamics.
method Proposed a robust Maximum Causal Entropy (MCE) IRL algorithm leveraging robust reinforcement learning insights.
result Empirically demonstrated stable performance improvement under transition dynamics mismatches.
Maximum entropy deep reinforcement learning (RL) methods have been demonstrated on a range of challenging continuous tasks. However, existing methods either suffer from severe instability when training on large off-policy data or cannot scale to tasks with very high state and action dimensionality such as 3D humanoid l…
In this paper, we present a new class of Markov decision processes (MDPs), called Tsallis MDPs, with Tsallis entropy maximization, which generalizes existing maximum entropy reinforcement learning (RL). A Tsallis MDP provides a unified framework for the original RL problem and RL with various types of entropy, includin…
New RL approach uses future state and action visitation measures for better exploration.
problem Improving exploration in reinforcement learning.
method Intrinsic reward based on future state and action visitation measures, using contraction operators.
result Policies achieve good state-action space coverage and high performance.
Deep networks have enabled reinforcement learning to scale to more complex and challenging domains, but these methods typically require large quantities of training data. An alternative is to use sample-efficient episodic control methods: neuro-inspired algorithms which use non-/semi-parametric models that predict valu…
Max entropy exploration guides reinforcement learning agents to pursue achievable goals.
problem Achieving distant test-time goals in long-horizon tasks.
method Optimize entropy of historical achieved goals by focusing on sparsely explored areas.
result Order of magnitude better sample efficiency on long-horizon multi-goal tasks.
Agents trained with deep reinforcement learning algorithms are capable of performing highly complex tasks including locomotion in continuous environments. We investigate transferring the learning acquired in one task to a set of previously unseen tasks. Generalization and overfitting in deep reinforcement learning are …
Multi-task Inverse Reinforcement Learning (IRL) is the problem of inferring multiple reward functions from expert demonstrations. Prior work, built on Bayesian IRL, is unable to scale to complex environments due to computational constraints. This paper contributes a formulation of multi-task IRL in the more computation…
We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are com…
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcs.LG
Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning…
Entropy regularization is an important idea in reinforcement learning, with great success in recent algorithms like Soft Q Network (SQN) and Soft Actor-Critic (SAC1). In this work, we extend this idea into the on-policy realm. We propose the soft policy gradient theorem (SPGT) for on-policy maximum entropy reinforcemen…
This work connects IRL methods from ML and economics.
problem Solving the inverse reinforcement learning problem.
method Shows connections and differences between various IRL methods.
result Identifies key computational and algorithmic differences.
In Multi-Goal Reinforcement Learning, an agent learns to achieve multiple goals with a goal-conditioned policy. During learning, the agent first collects the trajectories into a replay buffer, and later these trajectories are selected randomly for replay. However, the achieved goals in the replay buffer are often biase…
We address the problem of learning hierarchical deep neural network policies for reinforcement learning. In contrast to methods that explicitly restrict or cripple lower layers of a hierarchy to force them to use higher-level modulating signals, each layer in our framework is trained to directly solve the task, but acq…
The paper analyzes RLHF with human feedback and provides convergence results for MLE and pessimistic MLE.
problem Improving RLHF with human feedback from pairwise or -wise comparisons.
method Theoretical framework for RLHF with convergence analysis of MLE and pessimistic MLE.
result MLE fails but pessimistic MLE provides improved policies under certain coverage assumptions.
Develops statistical framework for resolving reward function ambiguity in inverse reinforcement learning.
problem Non-uniqueness of reward functions in inverse reinforcement learning.
method Entropy regularization combined with least-squares reconstruction of the reward from the soft Bellman residual.
result Least-squares reward function is unique and consistent with the expert policy.
From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Predictioncs.CL
In this work, we study the credit assignment problem in reward augmented maximum likelihood (RAML) learning, and establish a theoretical equivalence between the token-level counterpart of RAML and the entropy regularized reinforcement learning. Inspired by the connection, we propose two sequence prediction algorithms, …
New method learns multiple reward functions for complex tasks.
problem Learning reward functions for tasks with multiple ways of solving.
method Combines maximum entropy approach with Dirichlet process clustering.
result Method accurately learns reward functions for complex tasks.
We propose a new policy iteration theory as an important extension of soft policy iteration and Soft Actor-Critic (SAC), one of the most efficient model free algorithms for deep reinforcement learning. Supported by the new theory, arbitrary entropy measures that generalize Shannon entropy, such as Tsallis entropy and R…
Study on Bayesian reinforcement learning performance bounds.
problem Achieving optimal performance in model-based Bayesian reinforcement learning.
method Defining minimum Bayesian regret, deriving upper bounds using relative entropy and Wasserstein distance, and applying these to specific MDP cases.
result Upper bounds on minimum Bayesian regret for MDPs, including specific cases like MAB and online optimization with partial feedback.
Study shows fast rates for inverse reinforcement learning with linear rewards.
problem Entropy-regularized min-max inverse reinforcement learning in finite-horizon MDPs.
method Structural and statistical analysis of Min-Max-IRL with pseudo-self-concordance.
result Both trajectory-level KL divergence and parameter error decay at .
Unified reinforcement learning and stochastic processes with action-driven processes.
problem Combining reinforcement learning and stochastic processes for efficient control.
method Action-driven processes, leveraging control-as-inference, and minimizing Kullback-Leibler divergence.
result Action-driven processes unify reinforcement learning and stochastic processes, equivalent to maximum entropy reinforcement learning.
NDI learns from expert demonstrations by estimating occupancy measures.
problem Imitation Learning (IL) for complex systems.
method Density estimation of expert's occupancy measure followed by RL.
result NDI achieves state-of-the-art performance on control tasks.
This paper shows ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.
problem Understanding the lookahead capabilities of next-token prediction models.
method Unified view of ARMs and EBMs, establishing a bijection and deriving equivalence.
result ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.
The paper explores how different loss functions impact reinforcement learning algorithms.
problem Improving reinforcement learning algorithms by optimizing loss functions.
method Comprehensive survey on loss functions in reinforcement learning, proving the benefits of specific loss functions.
result Binary cross-entropy loss leads to first-order bounds and is more efficient than squared loss.
In many settings (e.g., robotics) demonstrations provide a natural way to specify tasks; however, most methods for learning from demonstrations either do not provide guarantees that the artifacts learned for the tasks, such as rewards or policies, can be safely composed and/or do not explicitly capture history dependen…
GraphOpt learns the formation mechanism of graphs from observed structures.
problem Learning formation mechanisms from observed graphs with complex structural properties.
method GraphOpt uses maximum entropy inverse reinforcement learning to solve the link formation problem in a sequential decision-making process.
result GraphOpt discovers a latent objective function that can explain and transfer across different graphs.
Paper develops MRCs for supervised classification using generalized maximum entropy.
problem Developing robust classifiers for decision problems.
method Generalized maximum entropy principle applied to minimax risk classifiers.
result Learning techniques for determining MRCs with performance guarantees.
ESAC combines genetic methods with RL to improve scalability and efficiency.
problem Combining genetic scalability with RL's data efficiency and optimal control.
method Combines Evolution Strategies (ES) with Soft Actor-Critic (SAC) to enable skill transfer and reduce hyperparameter sensitivity.
result Demonstrates improved performance and sample efficiency in challenging tasks.
Learning customer preferences from an observed behaviour is an important topic in the marketing literature. Structural models typically model forward-looking customers or firms as utility-maximizing agents whose utility is estimated using methods of Stochastic Optimal Control. We suggest an alternative approach to stud…
A new method recovers rewards from behavior policies using classification and regression.
problem Recovering meaningful rewards from observed behavior in reinforcement learning.
method GenPQR, a modular procedure that estimates behavior policy, evaluates soft Q-function, and recovers normalized reward using classification and regression.
result GenPQR matches or improves reward recovery compared to DeepPQR, while being simpler and more modular.
State entropy regularization improves robustness in reinforcement learning, especially under structured perturbations.
problem Structured and spatially correlated perturbations in reinforcement learning.
method State entropy regularization, compared to policy entropy.
result State entropy regularization provides better robustness to structured and spatially correlated perturbations.
The paper shows how policy regularization acts like an adversary to improve robustness.
problem Improving robustness of learned policies in reinforcement learning.
method Using convex duality, the paper characterizes adversarial reward perturbations and provides generalization guarantees.
result Policy regularization acts as an adversary to improve robustness against worst-case reward perturbations.
Maximum entropy modeling is a flexible and popular framework for formulating statistical models given partial knowledge. In this paper, rather than the traditional method of optimizing over the continuous density directly, we learn a smooth and invertible transformation that maps a simple distribution to the desired ma…
The framework of reinforcement learning or optimal control provides a mathematical formalization of intelligent decision making that is powerful and broadly applicable. While the general form of the reinforcement learning problem enables effective reasoning about uncertainty, the connection between reinforcement learni…
We propose a new approach to inverse reinforcement learning (IRL) based on the deep Gaussian process (deep GP) model, which is capable of learning complicated reward structures with few demonstrations. Our model stacks multiple latent GP layers to learn abstract representations of the state feature space, which is link…
We present a new statistical learning paradigm for Boltzmann machines based on a new inference principle we have proposed: the latent maximum entropy principle (LME). LME is different both from Jaynes maximum entropy principle and from standard maximum likelihood estimation.We demonstrate the LME principle BY deriving …