Synthesizes computational approaches to understand neural timescales.
problem Varying definitions and measurements of neural timescales across studies.
method Reviews data analysis methods, biophysical models, and machine learning models.
result Complements experimental studies with a holistic view of neural timescales.
This study improves convergence of two-timescale SA under Markovian noise in reinforcement learning.
problem Stability and convergence of two-timescale stochastic approximations under Markovian noise.
method Introduced a new control strategy for the fast timescale parameter.
result Established almost sure convergence of TDC with eligibility traces under off-policy learning with linear function approximation.
Paper analyzes normal approximation for two-timescale stochastic algorithms, revealing interaction between fast and slow timescales.
problem Non-asymptotic bounds for accuracy of normal approximation in linear two-timescale stochastic approximation algorithms.
method Established bounds for normal approximation in terms of convex distance, focusing on last iterate and Polyak-Ruppert averaging.
result Normal approximation rate for the last iterate improves with increased timescale separation, while it decreases in the averaged setting.
We study the distributions of event-time returns and clock-time returns at different microscopic timescales using ultra-high-frequency data extracted from the limit-order books of 23 stocks traded in the Chinese stock market in 2003. We find that the returns at the one-trade timescale obey the inverse cubic law. For la…
We introduce a framework to infer lead-lag networks between the states of elements of complex systems, determined at different timescales. As such networks encode the causal structure of a system, infering lead-lag networks for many pairs of timescales provides a global picture of the mutual influence between timescale…
The report evaluates heuristics for learning timescale graphical event models.
problem Lack of heuristics for determining hyper-parameters in timescale graphical event models.
method Proposed and evaluated different heuristics for hyper-parameter determination and refined an existing distance measure.
result Conclusions about the applicability of different heuristics on synthetic data.
New analysis shows actor-critic method converges efficiently in practical settings.
problem Understanding finite-time convergence of single-timescale actor-critic methods.
method Investigated online single-timescale actor-critic algorithm with linear function approximation and Markovian sampling.
result Proved convergence to ε-approximate stationary point with sample complexity of O(ε^(-2)).
In this paper we present a framework to analyze the asymptotic behavior of two timescale stochastic approximation algorithms including those with set-valued mean fields. This paper builds on the works of Borkar and Perkins & Leslie. The framework presented herein is more general as compared to the synchronous two times…
Method detects multi-timescale consumer spending patterns from receipts.
problem Understanding and managing consumer behavior in high-dimensional data.
method Non-negative tensor factorization (NTF) to extract multi-timescale expenditure patterns.
result Consumption patterns are characterized based on spending behavior over different timescales.
Proves convergence of neural networks in a two-timescale regime.
problem Training dynamics of shallow neural networks.
method Two-timescale regime analysis of gradient flow.
result Gradient flow converges to global optimum in non-convex optimization.
Two single-timescale algorithms improve TD learning with nonlinear approximations.
problem Optimizing TD learning with nonlinear smooth function approximation.
method Proposes two single-timescale single-loop algorithms with momentum and variance reduction.
result Achieves O(ε−4) sample complexity for the first algorithm and O(ε−3) for the second. Paper analyzes Greedy-GQ for reinforcement learning with Markovian noise.
problem Analyzing Greedy-GQ for reinforcement learning with Markovian noise.
method Develops finite-sample analysis for Greedy-GQ with linear function approximation under Markovian noise.
result Provides theoretical justification for choosing stepsizes for faster convergence.
A new buffer system improves continual learning in RL agents by adapting to changing environments.
problem Improving RL agents' ability to learn from changing environments over time.
method Multi-timescale replay buffer combined with invariant risk minimization.
result The method shows improvement over baselines in continual learning settings.
Novel approach to learning models based on subjective timescales for better exploration and decision-making.
problem Learning models over multi-step timescales in environments with intermediate states.
method Developed a subjective-timescale model (STM) based on episodic memories, enabling systematic variation of temporal extent of predictions.
result STM produces more informative action-conditioned roll-outs, leading to better decision-making and exploration.
A new approach is presented to describe the change in the statistics of the log return distribution of financial data as a function of the timescale. To this purpose a measure is introduced, which quantifies the distance of a considered distribution to a reference distribution. The existence of a small timescale regime…
Single-timescale actor-critic finds globally optimal policy.
problem Finding globally optimal policy in reinforcement learning.
method Simultaneous actor and critic updates with linear or deep neural network approximations.
result Actor sequence converges to globally optimal policy at O(K−1/2) rate. Learning and memory are intertwined in our brain and their relationship is at the core of several recent neural network models. In particular, the Attention-Gated MEmory Tagging model (AuGMEnT) is a reinforcement learning network with an emphasis on biological plausibility of memory dynamics and learning. We find that …
Gradient descent-ascent converges to strict local minmax equilibria with a finite timescale separation.
problem Analyzing the convergence of gradient descent-ascent in non-convex, non-concave games with a finite timescale separation.
method Investigates the role of a finite timescale separation parameter τ on gradient descent-ascent in two-player zero-sum games, providing convergence rates and non-convergence results.
result Gradient descent-ascent converges to strict local minmax equilibria for a finite timescale separation parameter τ*.
New method disentangles sources of different timescales in planetary seismic data.
problem Unsupervised source separation of multi-scale seismic data from planetary missions.
method Wavelet scattering spectra for multi-scale clustering and variational autoencoder for source separation.
result Disentangles sources with different timescales in InSight mission seismic data.
Study Whittle index learning algorithms for restless bandits with constant stepsizes.
problem Optimizing decisions in restless multi-armed bandits with constant stepsizes.
method Developed Q-learning algorithms with constant stepsizes for index learning in restless bandits, extending to DQN and function approximations.
result The algorithms learn the Whittle index effectively.
Bayesian inference and superstatistics model financial volatility dynamics across different timescales.
problem Modeling correlated volatility in financial time series with heavy tails and long memory.
method Superstatistical dynamics, Bayesian Inference, Metropolis-Hasting sampling.
result The log-Normal model is reliable for short timescales, while inverse-Gamma is preferred for long timescales.
This paper analyzes the sample complexity of two timescale reinforcement learning algorithms.
problem Analyzing the sample complexity of two timescale reinforcement learning algorithms.
method Non-asymptotic analysis of linear and nonlinear TDC and Greedy-GQ algorithms under Markovian sampling with constant stepsize.
result The paper provides non-asymptotic convergence results for two timescale linear and nonlinear TDC and Greedy-GQ algorithms.
The paper analyzes dynamics of momentum in high dimensions with sparse updates.
problem Theoretical analysis of momentum dynamics in high-dimensional sparse settings.
method Theoretical analysis of two models: least squares with sparse inputs and logistic regression with a rare class.
result Characterization of high-dimensional limits of momentum dynamics and phase structure.
Authors improve accuracy analysis for portfolio optimization with multiple timescale factors.
problem Asymptotic accuracy of portfolio optimization approximations for general utility functions and two timescale factors.
method Construct sub- and super-solutions to fully nonlinear problem.
result Rigorous justification of accuracy for portfolio optimization with general utility functions and two timescale factors.
Single-timescale analysis improves convergence in multi-sequence stochastic approximation.
problem Finite-time convergence of nonlinear stochastic approximation with multiple coupled sequences.
method Smoothness property of fixed points and analysis of fine-grained single-timescale SA.
result Improved iteration complexity for achieving ε-accuracy in multi-sequence single-timescale SA.
Develops a new method for efficient stochastic bilevel optimization.
problem Stochastic bilevel optimization problems in machine learning applications.
method Single-Timescale stochAstic BiLevEl optimization (STABLE) method.
result Achieves the same order of sample complexity as stochastic gradient descent for single-level optimization.
Paper separates financial time series into fast and slow components.
problem Multiscale behavior in financial time series data.
method Uses variance and tail stationarity criteria as generalized eigenvalue problems.
result Identifies slow and fast components in asset returns and prices.
GRACE-C improves causal learning from time series data.
problem Misleading causal information due to mismatched timescales.
method Combines constraint programming with theoretical insights and prior information.
result Significantly faster and scalable causal learning for large datasets.
New methods improve neural connectivity analysis at submillisecond timescales.
problem Limitations of standard spike train analysis methods in terms of temporal resolution and scalability.
method Developed Monte Carlo and polynomial approximation methods for continuous-time neural spike train analysis.
result Superior accuracy and scalability compared to traditional binned GLMs, enabling precise connectivity inference.
Paper analyzes finite-time performance of SA in RL with Markovian noise.
problem Finite-time analysis of linear two-timescale stochastic approximation with Markovian noise.
method Finite-time analysis of linear two-timescale SA with Markovian noise, considering both transient and steady-state terms.
result No discrepancy in convergence rate between Markovian and martingale noise; transient term is o(1/kc) and steady-state term is O(1/k). Study on Adam-family methods for nonsmooth optimization with convergence guarantees.
problem Training nonsmooth neural networks with convergence guarantees.
method Two-timescale updating scheme and stochastic subgradient methods with gradient clipping.
result Convergence guarantees for various Adam-family methods in training nonsmooth neural networks.
Study analyzes a new algorithm for complex optimization problems.
problem Stochastic bilevel optimisation problems in continuous-time models.
method Continuous-time, two-timescale stochastic approximation algorithm.
result Obtained weak convergence rate using central limit theorem.
Study finds physical momentum portfolios in Indian stock market yield higher returns than benchmarks.
problem Determining abnormal returns for physical momentum portfolios in the Indian stock market.
method Constructed physical momentum portfolios for daily, weekly, monthly, and yearly timescales, evaluated historical returns and risk profiles.
result Daily time scale physical momentum portfolios showed the strongest reversal with a 16-fold profit.
In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such c…
HiPPO framework optimizes memory compression for sequential data.
problem Incremental representation of cumulative history in sequential data.
method Optimal polynomial projections for online function approximation.
result HiPPO-LegS achieves state-of-the-art accuracy on MNIST.
We present the first provably convergent two-timescale off-policy actor-critic algorithm (COF-PAC) with function approximation. Key to COF-PAC is the introduction of a new critic, the emphasis critic, which is trained via Gradient Emphasis Learning (GEM), a novel combination of the key ideas of Gradient Temporal Differ…
Molecular Dynamics (MD) simulation is widely used to analyze the properties of molecules and materials. Most practical applications, such as comparison with experimental measurements, designing drug molecules, or optimizing materials, rely on statistical quantities, which may be prohibitively expensive to compute from …
Two-Timescale EM Methods improve EM for nonconvex models.
problem Nonconvex latent variable models are challenging for EM.
method Two-stage stochastic updates to handle nonconvex optimization.
result Global convergence for nonconvex objective functions.
The paper analyzes an actor-critic algorithm with target networks for deep reinforcement learning.
problem Lack of theoretical understanding of target networks in actor-critic methods.
method Proposes a theoretical analysis of an online target-based actor-critic algorithm with linear function approximation.
result Establishes asymptotic convergence results and finite-time analysis for both critic and actor.
New deep learning model estimates scattering timescale of FRBs efficiently.
problem Estimating scattering timescale of fast radio bursts (FRBs) is a bottleneck.
method Multimodal Transformer Based Generic Mixture Density Network (MT-GMDN) that ingests dynamic spectrum and timeseries profile.
result Achieves 94% R2 on expected value of τ for measurable scattering. New method speeds up lead-lag detection between asynchronous time series.
problem Slow inference of lead-lag networks between long time series.
method Derive asymptotic distribution of Transfer Entropy and introduce time-shifted time series.
result Statistically validated lead-lag networks between time series.
Paper analyzes CLT for TTSA with Markovian noise, broadening its applications.
problem Analyzing asymptotic behavior of TTSA under Markovian noise.
method Central Limit Theorem applied to TTSA with Markovian noise.
result Uncovered coupled dynamics of TTSA influenced by Markov chain.
In this paper we seek to demonstrate the predictability of stock market returns and explain the nature of this return predictability. To this end, we introduce investors with different investment horizons into the news-driven, analytic, agent-based market model developed in Gusev et al. (2015). This heterogeneous frame…
We propose a method for tackling catastrophic forgetting in deep reinforcement learning that is \textit{agnostic} to the timescale of changes in the distribution of experiences, does not require knowledge of task boundaries, and can adapt in \textit{continuously} changing environments. In our \textit{policy consolidati…
In this paper, we develop an online method that leverages machine learning to obtain feasible solutions to the AC optimal power flow (OPF) problem with negligible optimality gaps on extremely fast timescales (e.g., milliseconds), bypassing solving an AC OPF altogether. This is motivated by the fact that as the power gr…
This work analyzes how neural networks learn representations in actor-critic algorithms.
problem Theoretical support for neural AC algorithms is limited to linear function approximations.
method Mean-field analysis of a two-timescale learning AC algorithm with overparameterized networks.
result Neural AC finds the globally optimal policy at a sublinear rate in the continuous-time and infinite-width limiting regime.
Deep RL improves power control and scheduling for wireless multicast systems.
problem Scalable power control and scheduling for wireless multicast networks.
method Deep reinforcement learning with function approximation using a deep neural network.
result Deep RL can learn optimal power control policies for large systems.
Agents compose pre-trained policies for complex tasks, improving zero-shot performance.
problem Challenges in long-horizon predictions and estimating visitation distributions induced by policy sequences.
method Learn predictive jumpy world models of multi-step dynamics, enhancing predictions with a consistency objective.
result Compositional planning with jumpy world models yields, on average, a 200% relative improvement over primitive actions on long-horizon tasks.