New method improves deep policy gradient algorithms by learning relative state values.
problem High sample complexity and instability in policy gradient methods.
method Uses a new state-value function approximation based on residual variance.
result Empirical improvement across diverse continuous control tasks and algorithms.
Parastatistic distribution of a total debt owed to a large number of creditors considered in relation to the duration of these debts. The process of debt calculation depends on the fractal dimension of economic system in which this process takes place. Two actual variants of these dimensions are investigated. Critical …
Smaller actor-critic models lead to performance degradation and overfitting, highlighting the critic's role in value underestimation.
problem Performance degradation and overfitting in actor-critic models with smaller actors.
method Broad empirical investigations and analyses of asymmetric actor-critic setups, exploring techniques to mitigate value underestimation.
result Value underestimation is a key cause of performance degradation in smaller actor-critic models, and the critic plays a crucial role in mitigating this.
New method improves off-policy critic evaluation in reinforcement learning.
problem High variance and instability in off-policy policy evaluation.
method Doubly robust estimators applied to actor-critic algorithms.
result Doubly robust estimation significantly improves performance in continuous control tasks.
Decouples critic chunk length from policy to improve policy reactivity and performance.
problem Bootstrapping bias and difficulty in extracting optimal policies from chunked critics.
method Optimizes policy against a distilled critic for partial action chunks, allowing shorter chunks for policy.
result Reliably outperforms prior methods on long-horizon offline goal-conditioned tasks.
New groups found with critical exponents close to but less than max.
problem Finding discrete isometry groups with critical exponents near maximum.
method Analyzing complex hyperbolic spaces to construct groups.
result Discrete isometry groups with critical exponents arbitrarily close to max but less.
We give the classification, up to homeomorphisms, of reduced complex polynomials with 2 variables with one critical value.
Polynomials with distinct critical values have braid monodromy groups equal to braid groups.
problem Understanding the structure of braid monodromy groups of polynomials.
method Analyzing the critical values of polynomials to determine their braid monodromy groups.
result The braid monodromy group of a polynomial equals the braid group if the polynomial has distinct critical values.
Smooth manifolds have functions with exactly two critical values.
problem Characterizing manifolds with specific Reeb functions.
method Proving existence of Reeb functions with prescribed critical values.
result Characterization of manifolds in dimensions 3 and n≥5 using Reeb functions.
We describe how to compute topological objects associated to a polynomial map of several complex variables with isolated singularities. These objects are: the affine critical values, the affine Milnor numbers for all irregular fibers, the critical values at infinity, and the Milnor numbers at infinity for all irregular…
MAGE optimizes policies using action gradients from model-based learning.
problem Lack of direct gradient information from critics in actor-critic methods.
method Model-based actor-critic algorithm that learns action-value gradient.
result MAGE outperforms model-free and model-based baselines on continuous control tasks.
We give a global version of Le-Ramanujam mu-constant theorem for polynomials. Let f_t, (t in [0,1]), be a family of polynomials of n complex variables with isolated singularities, whose coefficients are polynomials in t. We consider the case where some numerical invariants are constant (the affine Milnor number, the Mi…
ESAC improves reinforcement learning by lookahead and intuition.
problem Designing efficient reinforcement learning architectures for intelligent agents.
method Integrates selector, tuner, model-learner, estimator, and lookahead into ESAC.
result ESAC outperforms other architectures in optimizing policies.
Short note on upper bounds for loop homology classes.
problem Critical values of homology classes in loop spaces of manifolds.
method Analysis of Riemannian and Finsler metrics with positive Ricci curvature.
result Upper bounds for shortest closed geodesics on manifolds.
A novel Q-learning variant reduces underestimation bias in deep actor-critic methods for reinforcement learning.
problem Underestimation bias in deep actor-critic methods for reinforcement learning.
method Introduces a parameter-free Q-learning variant that combines maximum and minimum operators to bound value estimates.
result Improves state-of-the-art performance on OpenAI Gym tasks.
WAVE improves stability in reinforcement learning by adaptively weighting critic's loss.
problem Inherent instability in actor-critic reinforcement learning algorithms.
method Wasserstein adaptive value estimation with Sinkhorn approximation.
result Achieves $\mathcal{O}\left(\frac{1}{k}
ight)$ convergence rate for critic's mean squared error.
Study magnetic geodesics on half-Lie groups, proving Hopf-Rinow theorem for energies above critical value.
problem Investigate magnetic geodesics on half-Lie groups using Riemannian and two-form structures.
method Define Mañé's critical value, prove Finsler geodesic flow equivalence, and apply Hopf-Rinow theorem.
result Hopf-Rinow theorem holds for energies above Mañé's critical value on magnetic geodesics.
In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the cr…
Study uses actor-critic method for continuous-time mean-field control with entropy regularisation.
problem Continuous-time mean-field control in reinforcement learning.
method Actor-critic approach with entropy regularisation, value function alternation, and Wasserstein space parametrisation.
result Derives exact parametrisation of actor and critic functions in linear-quadratic mean-field framework.
We use noncommutative localization to construct a chain complex which counts the critical points of a circle-valued Morse function on a manifold, generalizing the Novikov complex. As a consequence we obtain new topological lower bounds on the minimum number of critical points of a circle-valued Morse function within a …
In traditional reinforcement learning, an agent maximizes the reward collected during its interaction with the environment by approximating the optimal policy through the estimation of value functions. Typically, given a state s and action a, the corresponding value is the expected discounted sum of rewards. The optima…
We introduce a new critical value c∞(L) for Tonelli Lagrangians L on the tangent bundle of the 2-sphere without minimizing measures supported on a point. We show that c∞(L) is strictly larger than the Mañé critical value c(L), and on every energy level e∈(c(L),c∞(L)) there exist infinitely…
Formula for critical points of chi fields on manifolds.
problem Computing critical points of chi fields on general manifolds.
method Semi-analytic formula using Kac-Rice argument and Hessian matrix representation.
result Expression for expected value of critical points in high-threshold limit.
Paper proves gradient estimates for Lagrangian mean curvature equation.
problem Proving gradient estimates for Lagrangian mean curvature equation.
method Interior gradient estimates for critical and supercritical Lagrangian mean curvature equation.
result Solves Dirichlet boundary value problem for critical and supercritical Lagrangian mean curvature equation.
Some neural network modules are more critical to performance than others.
problem Understanding why some neural network architectures generalize better than others.
method Introduced module criticality, a measure based on the shape of loss valleys.
result Module criticality explains superior generalization performance of some architectures.
When f : R power n to R power p, is a surjective real analytic map with isolated critical value, we prove that the (m)-regularity condition (in a sense we define) ensures that f ||f|| is a fibration on small spheres, f induces a fibration on the tubes and both fibrations are equivalent. In particular, we make the state…
Proposes a value-based method for continuous control without an actor.
problem Computational infeasibility of evaluating Q-values in continuous action spaces.
method Structurally maximizable Q-functions, actor-free approach.
result Performance and sample efficiency comparable to actor-critic methods.
PBVFs generalize across policies using learned value functions.
problem RL algorithms forget information about old policies when updating value functions to track the learned policy.
method Introduce Parameter-Based Value Functions (PBVFs) that include policy parameters in their inputs, enabling them to generalize across different policies.
result PBVFs enable zero-shot learning of new policies that outperform any policy seen during training.
Study uncovers complex critical points in tensor decomposition.
problem Nonconvex optimization of symmetric tensor decomposition.
method Utilized symmetry to construct critical points and analyze Hessian.
result Obtained precise analytic estimates on objective function and Hessian.
We present the first provably convergent two-timescale off-policy actor-critic algorithm (COF-PAC) with function approximation. Key to COF-PAC is the introduction of a new critic, the emphasis critic, which is trained via Gradient Emphasis Learning (GEM), a novel combination of the key ideas of Gradient Temporal Differ…
We establish a new connection between value and policy based reinforcement learning (RL) based on a relationship between softmax temporal value consistency and policy optimality under entropy regularization. Specifically, we show that softmax consistent action values correspond to optimal entropy regularized policy pro…
New RL algorithm tackles complex discrete action spaces.
problem Challenges in applying on-policy RL in high-dimensional discrete action spaces.
method Action-value critic, correlated actions, gradient sparsification.
result Empirically outperforms related on-policy algorithms.
Study examines persistence diagrams in machine learning, proposing permutation tests.
problem Understanding the power and limitations of persistence diagrams in machine learning.
method Carried out experiments on graph and shape data, proposed permutation tests for persistence diagrams.
result Persistence pairing shows significant improvement in various tasks, but the most critical values are most discriminative.
The paper finds infinitely many magnetic geodesics on non-compact manifolds.
problem Existence and multiplicity of periodic orbits of magnetic flows.
method Morse theory applied to non-compact manifolds with energy levels above the Mañé critical value.
result Infinitely many noncontractible closed magnetic geodesics found.
Study magnetic geodesics on odd spheres, computing critical energy values.
problem Understanding magnetic geodesics on odd-dimensional spheres.
method Explicit computation and analysis of submanifolds and symmetries.
result Energy values determine magnetic geodesic connectivity on spheres.
The paper proves extremal black holes form at a critical point of gravitational collapse.
problem Formation of extremal black holes in gravitational collapse.
method Constructing smooth families of spherically symmetric solutions to the Einstein-Maxwell-Vlasov system.
result Extremal Reissner-Nordström black holes form at the critical collapse threshold.
Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…
This work, dealt with the classical mean value theorem and took advantage of it in the fractional calculus. The concept of a fractional critical point is introduced. Some sufficient conditions for the existence of a critical point is studied and an illustrative example rele- vant to the concept of the time dilation eff…
Solves local minima problems on smooth manifolds.
problem Local minima issues on smooth manifolds.
method Introducing valley functions and applying Morse's lemma.
result Eliminates critical points and reduces to 1D.
The z-transform technique is used to investigate the model for distribution of high-tax payers, which is proposed by two of the authors (K. Y and S. M) and others. Our analysis shows an asymptotic power-law of this model with the exponent -5/2 when a total ``mass'' has a certain critical value. Below the critical value…
This article introduces a framework to estimate the value of evidence-based decision making.
problem Lack of empirical tools to assess the value of evidence-based decision making and optimize statistical precision.
method Empirical framework using parametric and nonparametric empirical Bayes methods.
result The value of statistical evidence depends on how organizations translate it into policy decisions.
The paper defines a critical value for a magnetic system and extends solutions beyond blow-up.
problem Analyzing blow-up behavior and extending solutions for a magnetic system.
method Formulated as a magnetic geodesic equation on an infinite-dimensional Lie group, computed Mañé's critical value, established Hopf-Rinow theorem.
result Computed Mañé's critical value for the magnetic two-component Hunter-Saxton system and extended solutions beyond blow-up.
Band-limited SAC improves learning efficiency and stability in simulated environments.
problem Improving sample efficiency and stability in SAC algorithms.
method Artificially bandlimiting the target critic's spatial resolution using a convolutional filter.
result Bandlimited SAC outperforms classic twin-critic SAC in various Gym environments and is more stable.
New algorithm solves mean-field control problems using actor-critic learning with moment neural networks.
problem Solving mean-field control problems in continuous time reinforcement learning.
method Gradient-based policy and value function learning with moment neural networks on the Wasserstein space.
result Effective solution for diverse mean-field control problems, including multi-dimensional and nonlinear settings.
Given any n-tuple of complex numbers, one can canonically define a polynomial of degree n+1 that has the entries of this n-tuple as its critical points. In 2002, Beardon, Carne, and Ng studied a map θ:Cn→Cn which outputs the critical values of the canonical polynomial constructed from the…
We study the space of smooth Riemannian structures on compact three-manifolds with boundary that satisfies a critical point equation associated with a boundary value problem, for simplicity, Miao-Tam critical metrics. We provide an estimate to the area of the boundary of Miao-Tam critical metrics on compact three-manif…
We consider the estimation of the policy gradient in partially observable Markov decision processes (POMDP) with a special class of structured policies that are finite-state controllers. We show that the gradient estimation can be done in the Actor-Critic framework, by making the critic compute a "value" function that …
USAC balances pessimism and optimism in actor-critic training for better exploration and performance.
problem Excessive pessimism limits exploration, while excessive optimism leads to high-risk behaviors.
method Utility Soft Actor-Critic (USAC) dynamically adapts exploration based on critic uncertainty.
result USAC consistently outperforms state-of-the-art algorithms in continuous control tasks.