Ringmaster ASGD improves Asynchronous SGD's efficiency under varying worker times.
problem Suboptimal performance of Asynchronous SGD under heterogeneous worker computation times.
method Ringmaster ASGD, a novel Asynchronous SGD method with optimal time complexity.
result Ringmaster ASGD achieves optimal time complexity under arbitrary worker heterogeneity.
Improves BO efficiency by allowing asynchronous parallel computing.
problem Wasteful use of resources in batch Bayesian optimisation.
method Developed PLAyBOOK for asynchronous local penalisation.
result Asynchronous BO often outperforms synchronous BO.
Zeno++ improves robustness of asynchronous SGD in fully asynchronous settings.
problem Byzantine failures in fully asynchronous SGD.
method Estimates descent of loss after applying candidate gradient.
result Proves convergence for non-convex problems under Byzantine failures.
AEGiS optimizes expensive function evaluations asynchronously.
problem Optimizing expensive black-box functions efficiently.
method Asynchronous ε-Greedy Bayesian Optimisation combining greedy search and Thompson sampling.
result AEGiS outperforms existing asynchronous BO methods.
Secure aggregation for buffered asynchronous federated learning without TEEs.
problem Privacy and convergence in buffered asynchronous federated learning.
method Developed a new protocol (BASecAgg) that ensures privacy without TEEs by carefully designing masks.
result BASecAgg achieves similar convergence guarantees as FedBuff without TEEs.
Standard acquisition functions are sufficient for asynchronous Bayesian optimization.
problem Redundant and repeated queries in asynchronous Bayesian optimization.
method Conceptual analysis and theoretical guarantees of standard acquisitions.
result Standard acquisition functions achieve theoretical guarantees equivalent to Thompson sampling in asynchronous settings.
Paper analyzes convergence and speedup of asynchronous parallel SGD.
problem Achieving good convergence and linear speedup in asynchronous parallel SGD.
method Second-order convergence analysis of APSGD with consistent read near strictly saddle points.
result Theoretical guarantee for using at most O ( K 1 / 3 M − 1 / 3 ) O(K^{1/3}M^{-1/3}) O ( K 1/3 M − 1/3 ) workers for good convergence and linear speedup. Asynchronous method for hyperparameter and neural architecture search.
problem Efficiently searching for optimal hyperparameters and neural architectures.
method Model-based, asynchronous multi-fidelity method combining Hyperband and Gaussian process-based Bayesian optimization.
result Substantial speed-ups over current state-of-the-art methods on various benchmarks.
This work tackles resource allocation in asynchronous and stochastic systems.
problem Distributed resource allocation in asynchronous and stochastic settings.
method Approximate stochastic primal-dual approach with asynchronous updates.
result The Asynchronous stochastic Primal-Dual (Asyn-PD) algorithm converges to the saddle point solution at a rate of O ( 1 / t ) O(1/t) O ( 1/ t ) . Advances in asynchronous optimization methods for machine learning.
problem Efficiently solving large-scale optimization problems in machine learning.
method Asynchronous parallel and distributed optimization methods, accounting for information delays.
result Degree of asynchrony impacts convergence rates in stochastic optimization methods.
New Fourier transform method handles missing data and asynchronous observations.
problem Volatility inference with missing or asynchronous data.
method Spectral framework using Fourier transforms.
result Consistent volatility functional estimation with limit distributions.
New rules found to maintain neural network performance in asynchronous training.
problem Asynchronous training leads to degradation in generalization.
method Examined dynamical stability, derived rules for learning rate adjustment.
result Learning rate should be inversely proportional to delay for high delay values.
DANA mitigates gradient staleness in asynchronous distributed SGD with momentum.
problem Gradient staleness in asynchronous distributed SGD with momentum.
method DANA: a novel technique for asynchronous distributed SGD with momentum that computes the gradient on an estimated future position of the model's parameters.
result DANA fully incorporates momentum in asynchronous training with almost no ramifications to final accuracy.
As datasets continue to increase in size and multi-core computer architectures are developed, asynchronous parallel optimization algorithms become more and more essential to the field of Machine Learning. Unfortunately, conducting the theoretical analysis asynchronous methods is difficult, notably due to the introducti…
POAP and pySOT improve surrogate optimization of expensive functions.
problem Optimizing expensive functions with concurrent evaluations.
method Event-driven asynchronous framework for optimization strategies.
result Asynchronous computation offers significant speed-up advantages.
FedBuff improves federated learning scalability with asynchronous updates.
problem Limited scalability of federated learning with synchronous updates.
method Introduces asynchronous updates (staleness) in federated learning.
result Theoretical analysis shows improved convergence rate with boundedness removed.
Sparsification improves convergence in asynchronous distributed SGD, even in the presence of staleness.
problem Staleness in asynchronous distributed SGD.
method Applied sparsification to reduce communication overheads in distributed asynchronous settings.
result The ergodic convergence rate of sparsified asynchronous SGD matches that of vanilla SGD, $\mathcal{O} \left( 1/\sqrt{T}
ight)$ , even in the presence of staleness.
The asymptotic pseudo-trajectory approach to stochastic approximation of Benaim, Hofbauer and Sorin is extended for asynchronous stochastic approximations with a set-valued mean field. The asynchronicity of the process is incorporated into the mean field to produce convergence results which remain similar to those of a…
We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures. Roughly, the n…
PARyOpt optimizes functions asynchronously, reducing wall clock time.
problem Efficiently optimizing functions on distributed systems with asynchronous evaluations.
method Parallel asynchronous Bayesian optimization.
result Reduces total optimization time for various test problems.
Distributed asynchronous SGD has become widely used for deep learning in large-scale systems, but remains notorious for its instability when increasing the number of workers. In this work, we study the dynamics of distributed asynchronous SGD under the lens of Lagrangian mechanics. Using this description, we introduce …
Unified analysis of asynchronous-SGD algorithms for distributed learning.
problem Analyzing asynchronous-SGD in heterogeneous settings with varying speeds and data distributions.
method Unified convergence theory for non-convex smooth functions, including pure asynchronous SGD and its modifications.
result Unified convergence rates for various asynchronous algorithms, including novel methods.
Paper proves CLTs for Q-learning with asynchronous updates.
problem Establishing convergence rates for Q-learning algorithms.
method Polyak-Ruppert averaging, non-asymptotic and functional CLTs.
result Convergence rates in Wasserstein distance for Q-learning.
Accelerates optimization in asynchronous systems with sparse updates.
problem Optimizing finite-sum objectives in asynchronous lock-free environments.
method New accelerated SVRG variant with sparse updates.
result Achieves optimal incremental gradient complexity.
Improves asynchronous federated learning with queuing dynamics.
problem Asynchronous federated learning with varying node computational speeds.
method Proposes a non-uniform sampling scheme for the central server.
result Significant improvement over current algorithms on image classification.
We study optimization algorithms based on variance reduction for stochastic gradient descent (SGD). Remarkable recent progress has been made in this direction through development of algorithms like SAG, SVRG, SAGA. These algorithms have been shown to outperform SGD, both theoretically and empirically. However, asynchro…
Asynchronous framework speeds up model-based RL to real-time.
problem Real-time learning on real robots with model-based RL.
method Asynchronous framework for model-based reinforcement learning.
result Reduced run time to data collection time, improved sample complexity.
Ringleader ASGD optimizes SGD for diverse edge devices with varying data and computation speeds.
problem Scalable distributed optimization with heterogeneous devices and data.
method Ringleader ASGD, an asynchronous SGD algorithm.
result Achieves optimal time complexity under data heterogeneity and arbitrary computation speeds.
Combines multi-fidelity and asynchronous batch methods for faster experimental design.
problem Designing optimal experimental setups for battery performance.
method Algorithm combining multi-fidelity and asynchronous batch Bayesian Optimization.
result Algorithm outperforms single-fidelity batch and multi-fidelity sequential methods.
In many distributed learning problems, the heterogeneous loading of computing machines may harm the overall performance of synchronous strategies. In this paper, we propose an effective asynchronous distributed framework for the minimization of a sum of smooth functions, where each machine performs iterations in parall…
LSAM optimizes deep learning training with improved efficiency.
problem Inefficiency in distributed large-batch training with Sharpness-Aware Minimization (SAM).
method Integrates SAM's adversarial steps with an asynchronous distributed sampling strategy.
result Higher final accuracy compared to data-parallel SAM.
Asynchronous Gibbs sampling can accurately estimate expectations of functions of all variables under certain conditions.
problem Estimating expectations of functions of all variables in graphical models.
method Coupling synchronous and asynchronous Gibbs samplers to control expected Hamming distance, using concentration of measure results.
result The bias in estimating expectations of polynomial functions is smaller than the standard deviation of the function value in the true model.
Freya PAGE optimizes nonconvex optimization with heterogeneous, asynchronous workers.
problem Optimizing nonconvex finite-sum problems with varying worker processing times.
method Freya PAGE, a parallel method robust to stragglers and adaptive to slow computations.
result Freya PAGE offers improved time complexity guarantees compared to previous methods.
The paper gives bounds for how long it takes for gossip protocols to spread information in networks.
problem Understanding the diffusion time in asynchronous gossip protocols.
method Provides non-asymptotic bounds for the number of messages needed for consensus in asynchronous gossip protocols.
result Explicit formula and approximation for the number of messages needed for consensus in different types of graphs.
Asynchronous SGD generalizes well with enough data, improving stability and reducing error.
problem Generalization performance of asynchronous distributed SGD systems.
method Algorithm stability framework, adaptive learning rate strategy.
result Distributed asynchronous SGD generalizes well with enough data samples.
Asynchronous SGD can speed up training with a trade-off of gradient staleness.
problem Asynchronous SGD suffers from gradient staleness, affecting convergence error.
method Theoretical analysis of the error-runtime trade-off considering random straggling delays.
result A method of gradually varying synchronicity in distributed SGD is proposed and demonstrated.
Asynchronous parallel implementations of stochastic gradient (SG) have been broadly used in solving deep neural network and received many successes in practice recently. However, existing theories cannot explain their convergence and speedup properties, mainly due to the nonconvexity of most deep learning formulations …
New asynchronous algorithms improve speed in decentralized optimization networks.
problem Hard convergence analysis for asynchronous decentralized optimization.
method Continuized framework to analyze heterogeneous delays in event-driven updates.
result Achieves asynchronous speedup with convergence rate controlled by eigengap weighted by local delays.
Improved SAR in asynchronous conversations using neural models and unlabeled data.
problem Lack of labeled data for SAR in asynchronous conversations.
method Hierarchical LSTM-CRF model, semi-supervised learning with word embeddings, adversarial training.
result Adversarial training improves SAR performance by leveraging labeled data from synchronous domains.
Secure asynchronous federated learning with differential privacy for edge intelligence.
problem Privacy concerns in federated learning for edge computing.
method Differential privacy applied to secure asynchronous federated learning.
result MAPA improves model accuracy and convergence speed with sufficient privacy.
Most commonly used distributed machine learning systems are either synchronous or centralized asynchronous. Synchronous algorithms like AllReduce-SGD perform poorly in a heterogeneous environment, while asynchronous algorithms using a parameter server suffer from 1) communication bottleneck at parameter servers when wo…
New asynchronous SGD algorithms achieve optimal performance in distributed learning.
problem Asynchronous training introduces staleness, complicating optimization analysis.
method Developed rigorous framework for asynchronous first order stochastic optimization.
result Asynchronous SGD can achieve optimal time complexity, matching synchronous methods.
Asynchronous cooperative learning rules ensure all agents converge to correct hypothesis.
problem Cooperative learning in networks with unreliable communication.
method Proposed robust cooperative learning rule for weak communication networks.
result All agents' beliefs exponentially decay to the correct hypothesis.
New algorithm for multi-player bandits in decentralized, asynchronous systems.
problem Challenges in decentralized, asynchronous multi-player bandits, including coordination and player detection.
method Adaptive exploration-exploitation algorithm that reduces collisions and detects player presence.
result Achieves regret of O ( T log T + log T / Δ 2 ) \mathcal{O}(\sqrt{T \log T} + {\log T}/{Δ^2}) O ( T log T + log T / Δ 2 ) . A novel fully asynchronous scheme for distributed reinforcement learning over networks.
problem Policy evaluation in distributed reinforcement learning over networks.
method Design of a stochastic average gradient (SAG) based distributed algorithm and push-pull augmented graph approach.
result The proposed algorithm converges at a linear rate of \(\mathcal{O}(c^k)\) with \(c\in(0,1)\) and \(k\) increasing by one per node update.
CPGAs converge to locally optimal policies for coagent networks.
problem Training stochastic neural networks using reinforcement learning.
method Proved convergence of CPGAs and extended prior theory to asynchronous and recurrent networks.
result CPGAs converge to locally optimal policies for coagent networks, including asynchronous and recurrent networks.
New algorithm reduces regret in asynchronous multiplayer bandits to constant or logarithmic levels.
problem Asynchronous multiplayer bandits in cognitive radio networks.
method Cautious Greedy algorithm with O ( T log ( T ) ) \mathcal{O}(\sqrt{T\log(T)}) O ( T log ( T ) ) minimax regret. result Cautious Greedy yields constant instance-dependent regret under certain conditions.
Proposes a model for multi-horizon probabilistic forecasting of time series influenced by asynchronous events.
problem Forecasting time series influenced by asynchronous events is challenging.
method Introduces Variational Synergetic Multi-Horizon Network (VSMHN), a deep conditional generative model combining deep point processes and variational recurrent neural networks.
result Produces accurate, sharp, and realistic probabilistic forecasts.