arXiv research

AAPI improves regret bound for undiscounted continuing learning in uniformly ergodic MDPs.

New algorithm learns optimal policy for average reward MDPs with sample complexity matching lower bound.

Study improves reinforcement learning for stable long-term performance.

Model-free reinforcement learning is known to be memory and computation efficient and more amendable to large scale problems. In this paper, two model-free algorithms are introduced for learning infinite-horizon average-reward Markov Decision Processes (MDPs). The first algorithm reduces the problem to the discounted-r…

The paper improves regret lower bounds for communicating MDPs.

This paper establishes that optimistic algorithms attain gap-dependent and non-asymptotic logarithmic regret for episodic MDPs. In contrast to prior work, our bounds do not suffer a dependence on diameter-like quantities or ergodicity, and smoothly interpolate between the gap dependent logarithmic-regret, and the $\wid…

Let F be a family of Borel measurable functions on a complete separable metric space. The gap (or fat-shattering) dimension of F is a combinatorial quantity that measures the extent to which functions f in F can separate finite sets of points at a predefined resolution gamma > 0. We establish a connection between the g…

New concentration inequality for U-statistics of Markov chains.

New algorithm identifies best policy in MDPs faster.

In the paper portfolio optimization over long run risk sensitive criterion is considered. It is assumed that economic factors which stimulate asset prices are ergodic but non necessarily uniformly ergodic. Solution to suitable Bellman equation using local span contraction with weighted norms is shown. The form of optim…

A new model-free algorithm achieves near-optimal regret for infinite-horizon MDPs.

The paper provides concentration inequalities for Markov chain variance estimators.

Actor-Critic method achieves optimal regret for unichain MDPs.

Unified framework for solving MDPs with stochastic mirror descent.

Paper proposes an efficient RL algorithm for discounted MDPs using feature mapping.

New algorithm reduces reinforcement learning regret to sqrt(T) without strong dynamics assumptions.

The problem of reinforcement learning in an unknown and discrete Markov Decision Process (MDP) under the average-reward criterion is considered, when the learner interacts with the system in a single stream of observations, starting from an initial state without any reset. We revisit the minimax lower bound for that pr…

New algorithm detects changes in Markov kernels with unknown post-change kernel.

Optimizes learning policies in average-reward MDPs with improved sample complexity.

In [Mas82] and [Vee78] it was proved independently that almost every interval exchange transformation is uniquely ergodic. The Birkhoff ergodic theorem implies that these maps mainly have uniformly distributed orbits. This raises the question under which conditions the orbits yield low-discrepancy sequences. The case o…

Leveraging an equivalence property in the state-space of a Markov Decision Process (MDP) has been investigated in several studies. This paper studies equivalence structure in the reinforcement learning (RL) setup, where transition distributions are no longer assumed to be known. We present a notion of similarity betwee…

The paper studies harmonic map heat flow to flat tori, proving ergodic behavior and convergence to hyperbolic measure.

The paper improves importance sampling and MCMC methods for complex distributions.

Optimizes learning policies in MDPs with weakly communicating structure.

The paper proves conditions for non-uniform expansion in partially hyperbolic systems.

New MCMC methods map high-dimensional problems to spheres for better mixing.

New method detects changes in high-dimensional Markov processes without explicit likelihood evaluation.

We show that the sets in a family with finite VC dimension can be uniformly approximated within a given error by a finite partition. Immediate corollaries include the fact that VC classes have finite bracketing numbers, satisfy uniform laws of averages under strong dependence, and exhibit uniform mixing. Our results ar…

New method extends low-rank MDPs to continuous action spaces.

We construct a counterexample for an analogue of Masur's criterion in the setting of Teichmüller space equipped with the Thurston metric. For that, we find a minimal, filling, non-uniquely ergodic lamination $λ$ on the seven-times punctured sphere with uniformly bounded annular projection distances. Then we show that a…

Study of Matsumoto maps on foliated bundles over hyperbolic manifolds.

We give a simple optimistic algorithm for which it is easy to derive regret bounds of $\tilde{O}(\sqrt{t_{\rm mix} SAT})$ after $T$ steps in uniformly ergodic Markov decision processes with $S$ states, $A$ actions, and mixing time parameter $t_{\rm mix}$. These bounds are the first regret bounds in the general, non-epi…

This paper addresses metaconsistency in Bayesian inference for metastable systems.

In Bayesian statistics, many problems can be expressed as the evaluation of the expectation of a quantity of interest with respect to the posterior distribution. Standard Monte Carlo method is often not applicable because the encountered posterior distributions cannot be sampled directly. In this case, the most popular…

New method improves long-term forecasting of stochastic dynamical systems.

Study on measurable pseudo-Anosov maps on surfaces.

This work tackles large action spaces in RL by binarizing actions.

Improved sample complexity for actor-critic algorithms in MDPs.

Adversarial online multi-task RL with task separation.

We study the problem of learning the transition matrices of a set of Markov chains from a single stream of observations on each chain. We assume that the Markov chains are ergodic but otherwise unknown. The learner can sample Markov chains sequentially to observe their states. The goal of the learner is to sequentially…

Study non-rectangular robust MDPs for average-reward, finding optimal policies and transient values.

This is a brief technical note to clarify some of the issues with applying the application of the algorithm posterior sampling for reinforcement learning (PSRL) in environments without fixed episodes. In particular, this paper aims to: - Review some of results which have been proven for finite horizon MDPs (Osband et a…

Learning optimal resource allocation policies in wireless systems can be effectively achieved by formulating finite dimensional constrained programs which depend on system configuration, as well as the adopted learning parameterization. The interest here is in cases where system models are unavailable, prompting method…

In this paper long-run risk sensitive optimisation problem is studied with dyadic impulse control applied to continuous-time Feller-Markov process. In contrast to the existing literature, focus is put on unbounded and non-uniformly ergodic case by adapting the weight norm approach. In particular, it is shown how to com…

Study counts ergodic measures in surface lamination strata.

Improved stochastic Halpern iteration for fixed-point approximation in normed spaces.

New progress on frame flow ergodicity for nearly pinched manifolds.

Recent results on ergodic theory for Riemann surface laminations and foliations.