arXiv research

Entropy regularization improves policy optimization in reinforcement learning.

Paper proposes a policy-search algorithm to learn entropy-maximizing exploration policies in reward-free environments.

DAC enhances exploration in reinforcement learning with entropy regularization.

New approach to portfolio optimization shows entropy regularization is ineffective.

Enhances RL by controlling policy stochasticity through trajectory entropy constraints.

We introduce Implicit Policy, a general class of expressive policies that can flexibly represent complex action distributions in reinforcement learning, with efficient algorithms to compute entropy regularized policy gradients. We empirically show that, despite its simplicity in implementation, entropy regularization c…

Bayesian Markowitz portfolio problem shows entropy regularization is ineffective.

Two hitherto disconnected threads of research, diverse exploration (DE) and maximum entropy RL have addressed a wide range of problems facing reinforcement learning algorithms via ostensibly distinct mechanisms. In this work, we identify a connection between these two approaches. First, a discriminator-based diversity …

Entropy-regularized NPG methods converge linearly in discounted MDPs.

This paper improves reinforcement learning policies in a scalable way.

Entropy regularization is used to get improved optimization performance in reinforcement learning tasks. A common form of regularization is to maximize policy entropy to avoid premature convergence and lead to more stochastic policies for exploration through action space. However, this does not ensure exploration in th…

Entropy regularization is commonly used to improve policy optimization in reinforcement learning. It is believed to help with \emph{exploration} by encouraging the selection of more stochastic policies. In this work, we analyze this claim using new visualizations of the optimization landscape based on randomly perturbi…

Entropy-regularized NPG converges linearly with linear function approximation.

This work optimizes RL algorithms using entropy regularisation for continuous-time LQ problems.

Entropy regularization is an important idea in reinforcement learning, with great success in recent algorithms like Soft Q Network (SQN) and Soft Actor-Critic (SAC1). In this work, we extend this idea into the on-policy realm. We propose the soft policy gradient theorem (SPGT) for on-policy maximum entropy reinforcemen…

Maximum entropy deep reinforcement learning (RL) methods have been demonstrated on a range of challenging continuous tasks. However, existing methods either suffer from severe instability when training on large off-policy data or cannot scale to tasks with very high state and action dimensionality such as 3D humanoid l…

This work extends ME-RL using diffusion models to sample optimal policies.

PEOC uses policy entropy to detect untrained states in RL.

Entropy regularization improves MFG learning efficiency and stability.

State entropy regularization improves robustness in reinforcement learning, especially under structured perturbations.

Trust-region methods have yielded state-of-the-art results in policy search. A common approach is to use KL-divergence to bound the region of trust resulting in a natural gradient policy update. We show that the natural gradient and trust region optimization are equivalent if we use the natural parameterization of a st…

Study examines market response to concentrated policy communication using entropy measures.

Algorithm learns Nash equilibria in stochastic games using entropy-regularized policies.

Paper improves policy updates in reinforcement learning to speed up learning.

New RL approach uses future state and action visitation measures for better exploration.

We consider the problem of group testing with sum observations and noiseless answers, in which we aim to locate multiple objects by querying the number of objects in each of a sequence of chosen sets. We study a probabilistic setting with entropy loss, in which we assume a joint Bayesian prior density on the locations …

Improves policy optimization with polylog(T) regret bounds for stochastic losses.

Maximizes Rényi entropy for efficient exploration in reward-free RL.

We analyze the problem of learning a single user's preferences in an active learning setting, sequentially and adaptively querying the user over a finite time horizon. Learning is conducted via choice-based queries, where the user selects her preferred option among a small subset of offered alternatives. These queries …

Softmax policy gradient methods converge at $O(1/t)$ rate with constants depending on problem and initialization.

AER dynamically adjusts entropy regularization for better LLM reinforcement learning.

Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor). The update to the actor involves a log-likelihood update weighted by the action-values, with the addition of entropy regularization for soft variants. In th…

Study shows policy gradient convergence for entropy-regularized MDPs with neural nets in mean-field regime.

Develops statistical framework for resolving reward function ambiguity in inverse reinforcement learning.

We study the sparse entropy-regularized reinforcement learning (ERL) problem in which the entropy term is a special form of the Tsallis entropy. The optimal policy of this formulation is sparse, i.e.,~at each state, it has non-zero probability for only a small number of actions. This addresses the main drawback of the …

Study risk-sensitive market making with entropy regularization for better quote control.

We propose a new policy iteration theory as an important extension of soft policy iteration and Soft Actor-Critic (SAC), one of the most efficient model free algorithms for deep reinforcement learning. Supported by the new theory, arbitrary entropy measures that generalize Shannon entropy, such as Tsallis entropy and R…

Double descent found in DRL, improving generalization with model capacity.

Softmax policy gradient achieves global optimality in wide neural networks with entropy regularization.

The paper interprets policy-gradient algorithms using continuation theory.

We propose a general framework for entropy-regularized average-reward reinforcement learning in Markov decision processes (MDPs). Our approach is based on extending the linear-programming formulation of policy optimization in MDPs to accommodate convex regularization functions. Our key result is showing that using the …

In recent years, deep reinforcement learning has been shown to be adept at solving sequential decision processes with high-dimensional state spaces such as in the Atari games. Many reinforcement learning problems, however, involve high-dimensional discrete action spaces as well as high-dimensional state spaces. This pa…

New RL method nearly optimally learns policies with generative models.

In this paper, we present a probability one convergence proof, under suitable conditions, of a certain class of actor-critic algorithms for finding approximate solutions to entropy-regularized MDPs using the machinery of stochastic approximation. To obtain this overall result, we prove the convergence of policy evaluat…

The policy gradient theorem is defined based on an objective with respect to the initial distribution over states. In the discounted case, this results in policies that are optimal for one distribution over initial states, but may not be uniformly optimal for others, no matter where the agent starts from. Furthermore, …

RL solves discrete LQ control with Gaussian optimal policy.

Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning…

Paper develops a new probabilistic method for American options using entropy regularization.