Asynchronous methods are widely used in deep learning, but have limited theoretical justification when applied to non-convex problems. We show that running stochastic gradient descent (SGD) in an asynchronous manner can be viewed as adding a momentum-like term to the SGD iteration. Our result does not assume convexity …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
29 results for “asynchrony”
Study compares Fourier estimators to mitigate asynchrony effects in finance.
problem Impact of asynchrony on instantaneous financial estimates.
method Comparison of Malliavin-Mancino and Cuchiero-Teichmann estimators.
result Malliavin-Mancino estimator produces more stable estimates under asynchrony.
Asynchronous momentum stochastic gradient descent algorithms (Async-MSGD) is one of the most popular algorithms in distributed machine learning. However, its convergence properties for these complicated nonconvex problems is still largely unknown, because of the current technical limit. Therefore, in this paper, we pro…
New model stabilizes asynchronous LTI systems, independent of synchronous stability.
problem Stability of asynchronous LTI systems under randomization and asynchrony.
method Introduced a new model for random asynchronous LTI systems and developed a method for system identification.
result Stability of random asynchronous LTI systems is independent of synchronous stability.
The performance of fully synchronized distributed systems has faced a bottleneck due to the big data trend, under which asynchronous distributed systems are becoming a major popularity due to their powerful scalability. In this paper, we study the generalization performance of stochastic gradient descent (SGD) on a dis…
Tight Convergence Rates for Online Distributed Linear Estimation with Adversarial Measurementsstat.ML
Study improves distributed linear estimation under adversarial conditions.
problem Mean estimation of a random vector with adversarial measurements and asynchrony.
method Two-timescale ℓ1-minimization algorithm with tight convergence rates.
result Unified finite-time characterization of robustness, identifiability, and statistical efficiency.
Any given classification problem can be modeled using multi-class or One-vs-All (OVA) architecture. An OVA system consists of as many OVA models as the number of classes, providing the advantage of asynchrony, where each OVA model can be re-trained independent of other models. This is particularly advantageous in setti…
We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures. Roughly, the n…
We present a method to compensate statistical errors in the calculation of correlations on asynchronous time series. The method is based on the assumption of an underlying time series. We set up a model and apply it to financial data to examine the decrease of calculated correlations towards smaller return intervals (E…
Asynchronous computation and gradient compression have emerged as two key techniques for achieving scalability in distributed optimization for large-scale machine learning. This paper presents a unified analysis framework for distributed gradient methods operating with staled and compressed gradients. Non-asymptotic bo…
Analysis and Implementation of an Asynchronous Optimization Algorithm for the Parameter Servermath.OC
This paper presents an asynchronous incremental aggregated gradient algorithm and its implementation in a parameter server framework for solving regularized optimization problems. The algorithm can handle both general convex (possibly non-smooth) regularizers and general convex constraints. When the empirical data loss…
Second-order guarantees for federated learning algorithms.
problem Non-convex optimization in federated learning with saddle-points as bottlenecks.
method Drawing on recent results on second-order optimality in centralized and decentralized settings, establish second-order guarantees for federated learning algorithms.
result Established second-order guarantees for federated learning algorithms.
We present two statistical causes for the distortion of correlations on high-frequency financial data. We demonstrate that the asynchrony of trades as well as the decimalization of stock prices has a large impact on the decline of the correlation coefficients towards smaller return intervals (Epps effect). These distor…
Pipeline parallelism (PP) when training neural networks enables larger models to be partitioned spatially, leading to both lower network communication and overall higher hardware utilization. Unfortunately, to preserve the statistical efficiency of sequential training, existing PP techniques sacrifice hardware efficien…
This work addresses the instability in asynchronous data parallel optimization. It does so by introducing a novel distributed optimizer which is able to efficiently optimize a centralized model under communication constraints. The optimizer achieves this by pushing a normalized sequence of first-order gradients to a pa…
Although distributed computing can significantly reduce the training time of deep neural networks, scaling the training process while maintaining high efficiency and final accuracy is challenging. Distributed asynchronous training enjoys near-linear speedup, but asynchrony causes gradient staleness - the main difficult…
Advances in asynchronous optimization methods for machine learning.
problem Efficiently solving large-scale optimization problems in machine learning.
method Asynchronous parallel and distributed optimization methods, accounting for information delays.
result Degree of asynchrony impacts convergence rates in stochastic optimization methods.
This paper models CSI 300 index volatility using machine learning and addresses jump prediction.
problem Volatility modeling and jump prediction for high-frequency CSI 300 index data.
method Generalized Barndorff-Nielsen and Shephard model with machine learning algorithms for parameter estimation and forecast evaluation.
result Deterministic component of stochastic volatility processes can be captured over short and longer-term windows.
Survey of determinism issues in financial AI systems.
problem Vulnerabilities in reproducibility of financial AI systems.
method Literature review and first-party experiments on public financial datasets.
result Proposed a layered evaluation framework linking modality-specific metrics to audit readiness.
Mini-batch optimization has proven to be a powerful paradigm for large-scale learning. However, the state of the art parallel mini-batch algorithms assume synchronous operation or cyclic update orders. When worker nodes are heterogeneous (due to different computational capabilities or different communication delays), s…
Recent years have witnessed the surge of asynchronous parallel (async-parallel) iterative algorithms due to problems involving very large-scale data and a large number of decision variables. Because of asynchrony, the iterates are computed with outdated information, and the age of the outdated information, which we cal…
The Epps effect helps distinguish between continuous and discrete financial tick data.
problem Determining whether financial tick data represents continuous or discrete events.
method Deriving and correcting the Epps effect, proposing experiments to discriminate between models.
result Tick data is better represented as discrete events rather than continuous Brownian diffusions.
We analyze the convergence of gradient-based optimization algorithms that base their updates on delayed stochastic gradient information. The main application of our results is to the development of gradient-based distributed optimization algorithms where a master node performs parameter updates while worker nodes compu…
Stochastic Gradient Descent (SGD) is a fundamental algorithm in machine learning, representing the optimization backbone for training several classic models, from regression to neural networks. Given the recent practical focus on distributed machine learning, significant work has been dedicated to the convergence prope…
Finding a fixed point to a nonexpansive operator, i.e., , abstracts many problems in numerical linear algebra, optimization, and other areas of scientific computing. To solve fixed-point problems, we propose ARock, an algorithmic framework in which multiple agents (machines, processors, or cores) update i…
Machine Learning (ML) solutions are nowadays distributed and are prone to various types of component failures, which can be encompassed in so-called Byzantine behavior. This paper introduces LiuBei, a Byzantine-resilient ML algorithm that does not trust any individual component in the network (neither workers nor serve…
Asynchronous distributed machine learning solutions have proven very effective so far, but always assuming perfectly functioning workers. In practice, some of the workers can however exhibit Byzantine behavior, caused by hardware failures, software bugs, corrupt data, or even malicious attacks. We introduce \emph{Karda…
New algorithm reduces distributed optimization time with stochastic delays.
problem Optimizing distributed data with stochastic delays.
method Developed ADSAGA, a variant of SAGA for distributed-data settings with stochastic delays.
result ADSAGA converges in $ ilde{O}\left(\left(n + \sqrt{m}κ
ight)\log(1/ε)
ight)$ iterations under mean delay .
New asynchronous SGD algorithms achieve optimal performance in distributed learning.
problem Asynchronous training introduces staleness, complicating optimization analysis.
method Developed rigorous framework for asynchronous first order stochastic optimization.
result Asynchronous SGD can achieve optimal time complexity, matching synchronous methods.