Ringmaster ASGD improves Asynchronous SGD's efficiency under varying worker times.
problem Suboptimal performance of Asynchronous SGD under heterogeneous worker computation times.
method Ringmaster ASGD, a novel Asynchronous SGD method with optimal time complexity.
result Ringmaster ASGD achieves optimal time complexity under arbitrary worker heterogeneity.
Advances in asynchronous optimization methods for machine learning.
problem Efficiently solving large-scale optimization problems in machine learning.
method Asynchronous parallel and distributed optimization methods, accounting for information delays.
result Degree of asynchrony impacts convergence rates in stochastic optimization methods.
Asynchronous method for hyperparameter and neural architecture search.
problem Efficiently searching for optimal hyperparameters and neural architectures.
method Model-based, asynchronous multi-fidelity method combining Hyperband and Gaussian process-based Bayesian optimization.
result Substantial speed-ups over current state-of-the-art methods on various benchmarks.
Standard acquisition functions are sufficient for asynchronous Bayesian optimization.
problem Redundant and repeated queries in asynchronous Bayesian optimization.
method Conceptual analysis and theoretical guarantees of standard acquisitions.
result Standard acquisition functions achieve theoretical guarantees equivalent to Thompson sampling in asynchronous settings.
New Fourier transform method handles missing data and asynchronous observations.
problem Volatility inference with missing or asynchronous data.
method Spectral framework using Fourier transforms.
result Consistent volatility functional estimation with limit distributions.
AEGiS optimizes expensive function evaluations asynchronously.
problem Optimizing expensive black-box functions efficiently.
method Asynchronous ε-Greedy Bayesian Optimisation combining greedy search and Thompson sampling.
result AEGiS outperforms existing asynchronous BO methods.
This paper describes Plumbing for Optimization with Asynchronous Parallelism (POAP) and the Python Surrogate Optimization Toolbox (pySOT). POAP is an event-driven framework for building and combining asynchronous optimization strategies, designed for global optimization of expensive functions where concurrent function …
Accelerates optimization in asynchronous systems with sparse updates.
problem Optimizing finite-sum objectives in asynchronous lock-free environments.
method New accelerated SVRG variant with sparse updates.
result Achieves optimal incremental gradient complexity.
As datasets continue to increase in size and multi-core computer architectures are developed, asynchronous parallel optimization algorithms become more and more essential to the field of Machine Learning. Unfortunately, conducting the theoretical analysis asynchronous methods is difficult, notably due to the introducti…
Combines multi-fidelity and asynchronous batch methods for faster experimental design.
problem Designing optimal experimental setups for battery performance.
method Algorithm combining multi-fidelity and asynchronous batch Bayesian Optimization.
result Algorithm outperforms single-fidelity batch and multi-fidelity sequential methods.
Ringleader ASGD optimizes SGD for diverse edge devices with varying data and computation speeds.
problem Scalable distributed optimization with heterogeneous devices and data.
method Ringleader ASGD, an asynchronous SGD algorithm.
result Achieves optimal time complexity under data heterogeneity and arbitrary computation speeds.
We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures. Roughly, the n…
Asynchronous framework speeds up model-based RL to real-time.
problem Real-time learning on real robots with model-based RL.
method Asynchronous framework for model-based reinforcement learning.
result Reduced run time to data collection time, improved sample complexity.
Unified analysis of asynchronous-SGD algorithms for distributed learning.
problem Analyzing asynchronous-SGD in heterogeneous settings with varying speeds and data distributions.
method Unified convergence theory for non-convex smooth functions, including pure asynchronous SGD and its modifications.
result Unified convergence rates for various asynchronous algorithms, including novel methods.
We study optimization algorithms based on variance reduction for stochastic gradient descent (SGD). Remarkable recent progress has been made in this direction through development of algorithms like SAG, SVRG, SAGA. These algorithms have been shown to outperform SGD, both theoretically and empirically. However, asynchro…
New rules found to maintain neural network performance in asynchronous training.
problem Asynchronous training leads to degradation in generalization.
method Examined dynamical stability, derived rules for learning rate adjustment.
result Learning rate should be inversely proportional to delay for high delay values.
Freya PAGE optimizes nonconvex optimization with heterogeneous, asynchronous workers.
problem Optimizing nonconvex finite-sum problems with varying worker processing times.
method Freya PAGE, a parallel method robust to stragglers and adaptive to slow computations.
result Freya PAGE offers improved time complexity guarantees compared to previous methods.
Asynchronous SGD can speed up training with a trade-off of gradient staleness.
problem Asynchronous SGD suffers from gradient staleness, affecting convergence error.
method Theoretical analysis of the error-runtime trade-off considering random straggling delays.
result A method of gradually varying synchronicity in distributed SGD is proposed and demonstrated.
A new asynchronous method for vertical federated learning improves privacy and efficiency.
problem Solving vertical federated learning in an asynchronous manner with privacy and efficiency.
method A simple FL method that allows clients to run stochastic gradient algorithms asynchronously with a new perturbed local embedding technique.
result The method improves privacy and communication efficiency compared to centralized and synchronous FL methods.
Novel asynchronous SGD method resists Byzantine attacks without server storage.
problem Asynchronous distributed learning with Byzantine attacks and failures.
method Buffered Asynchronous SGD (BASGD) and its momentum variant (BASGDm).
result BASGD and BASGDm resist non-omniscient and omniscient attacks without server storage.
Improves asynchronous federated learning with queuing dynamics.
problem Asynchronous federated learning with varying node computational speeds.
method Proposes a non-uniform sampling scheme for the central server.
result Significant improvement over current algorithms on image classification.
In many distributed learning problems, the heterogeneous loading of computing machines may harm the overall performance of synchronous strategies. In this paper, we propose an effective asynchronous distributed framework for the minimization of a sum of smooth functions, where each machine performs iterations in parall…
Stochastic gradient descent~(SGD) and its variants have become more and more popular in machine learning due to their efficiency and effectiveness. To handle large-scale problems, researchers have recently proposed several parallel SGD methods for multicore systems. However, existing parallel SGD methods cannot achieve…
New asynchronous SGD algorithms achieve optimal performance in distributed learning.
problem Asynchronous training introduces staleness, complicating optimization analysis.
method Developed rigorous framework for asynchronous first order stochastic optimization.
result Asynchronous SGD can achieve optimal time complexity, matching synchronous methods.
A scalable parallel BO method for asynchronous settings.
problem Expensive-to-evaluate problems in machine learning.
method Simple and scalable Bayesian optimization method for asynchronous parallel settings.
result Demonstrated promising performance on benchmark functions and hyperparameter optimization.
Although distributed computing can significantly reduce the training time of deep neural networks, scaling the training process while maintaining high efficiency and final accuracy is challenging. Distributed asynchronous training enjoys near-linear speedup, but asynchrony causes gradient staleness - the main difficult…
New asynchronous algorithms improve speed in decentralized optimization networks.
problem Hard convergence analysis for asynchronous decentralized optimization.
method Continuized framework to analyze heterogeneous delays in event-driven updates.
result Achieves asynchronous speedup with convergence rate controlled by eigengap weighted by local delays.
Ringmaster LMO accelerates training in distributed systems by asynchronously updating neural networks.
problem Asynchronous training in distributed systems where workers compute gradients at different speeds.
method Introduces an asynchronous LMO-based momentum method for unconstrained stochastic nonconvex optimization.
result Establishes convergence guarantees and time complexity bounds for asynchronous LMO-based updates.
Recent years have witnessed exciting progress in the study of stochastic variance reduced gradient methods (e.g., SVRG, SAGA), their accelerated variants (e.g, Katyusha) and their extensions in many different settings (e.g., online, sparse, asynchronous, distributed). Among them, accelerated methods enjoy improved conv…
New method calibrates asynchronous, error-prone covariates for longitudinal data.
problem Estimation biases and slow convergence in analyzing time-varying covariates with measurement error.
method Functional calibration approach based on functional principal component analysis.
result Asymptotically unbiased and consistent estimators for time-invariant coefficients; optimal convergence rate for time-varying coefficients.
New model stabilizes asynchronous LTI systems, independent of synchronous stability.
problem Stability of asynchronous LTI systems under randomization and asynchrony.
method Introduced a new model for random asynchronous LTI systems and developed a method for system identification.
result Stability of random asynchronous LTI systems is independent of synchronous stability.
We introduce and analyze stochastic optimization methods where the input to each gradient update is perturbed by bounded noise. We show that this framework forms the basis of a unified approach to analyze asynchronous implementations of stochastic optimization algorithms.In this framework, asynchronous stochastic optim…
Markov Chain Monte Carlo (MCMC) methods such as Gibbs sampling are finding widespread use in applied statistics and machine learning. These often lead to difficult computational problems, which are increasingly being solved on parallel and distributed systems such as compute clusters. Recent work has proposed running i…
Secure aggregation for buffered asynchronous federated learning without TEEs.
problem Privacy and convergence in buffered asynchronous federated learning.
method Developed a new protocol (BASecAgg) that ensures privacy without TEEs by carefully designing masks.
result BASecAgg achieves similar convergence guarantees as FedBuff without TEEs.
Paper analyzes convergence and speedup of asynchronous parallel SGD.
problem Achieving good convergence and linear speedup in asynchronous parallel SGD.
method Second-order convergence analysis of APSGD with consistent read near strictly saddle points.
result Theoretical guarantee for using at most O(K1/3M−1/3) workers for good convergence and linear speedup. Batch Bayesian optimisation (BO) has been successfully applied to hyperparameter tuning using parallel computing, but it is wasteful of resources: workers that complete jobs ahead of others are left idle. We address this problem by developing an approach, Penalising Locally for Asynchronous Bayesian Optimisation on k…
A novel fully asynchronous scheme for distributed reinforcement learning over networks.
problem Policy evaluation in distributed reinforcement learning over networks.
method Design of a stochastic average gradient (SAG) based distributed algorithm and push-pull augmented graph approach.
result The proposed algorithm converges at a linear rate of \(\mathcal{O}(c^k)\) with \(c\in(0,1)\) and \(k\) increasing by one per node update.
This work tackles resource allocation in asynchronous and stochastic systems.
problem Distributed resource allocation in asynchronous and stochastic settings.
method Approximate stochastic primal-dual approach with asynchronous updates.
result The Asynchronous stochastic Primal-Dual (Asyn-PD) algorithm converges to the saddle point solution at a rate of O(1/t). AB dynamically scales gradients to mitigate asynchronous training delays.
problem Gradient delay in asynchronous training reduces model performance.
method Adaptive Braking (AB) dynamically scales gradients based on alignment.
result AB enables training with up to 32 update steps of delay without accuracy loss.
FAVANO improves federated learning for resource-constrained environments.
problem Asynchronous communication in federated learning leads to bias and scalability issues.
method FAVANO is a novel asynchronous federated learning framework for resource-constrained environments.
result FAVANO outperforms existing methods on standard benchmarks.
New method speeds up nuclear-norm constrained learning over multiple machines.
problem Synchronization slowdown and high communication costs in large-scale learning.
method Asynchronous Stochastic Frank-Wolfe (SFW-asyn) method.
result SFW-asyn achieves the same convergence rate as vanilla SFW but with speed-ups almost linear to the number of machines.
FedBuff improves federated learning scalability with asynchronous updates.
problem Limited scalability of federated learning with synchronous updates.
method Introduces asynchronous updates (staleness) in federated learning.
result Theoretical analysis shows improved convergence rate with boundedness removed.
A new method for efficient online federated learning reduces communication overhead.
problem Real-world limitations in online federated learning, such as heterogeneous client participation and communication delays.
method Proposes a communication-efficient asynchronous online federated learning (PAO-Fed) strategy.
result Achieves the same convergence properties as online federated stochastic gradient while reducing communication overhead by 98 percent.
Sparsification improves convergence in asynchronous distributed SGD, even in the presence of staleness.
problem Staleness in asynchronous distributed SGD.
method Applied sparsification to reduce communication overheads in distributed asynchronous settings.
result The ergodic convergence rate of sparsified asynchronous SGD matches that of vanilla SGD, $\mathcal{O} \left( 1/\sqrt{T}
ight)$, even in the presence of staleness.
GA method reduces gradient staleness in cloud computing.
problem Gradient staleness in asynchronous SGD methods.
method Gap-Aware (GA) method that penalizes stale gradients linearly to the Gap.
result GA outperforms existing methods in final test accuracy.
The asymptotic pseudo-trajectory approach to stochastic approximation of Benaim, Hofbauer and Sorin is extended for asynchronous stochastic approximations with a set-valued mean field. The asynchronicity of the process is incorporated into the mean field to produce convergence results which remain similar to those of a…
Paper introduces elastic consistency for distributed SGD, enabling convergence analysis.
problem Training large-scale machine learning models in distributed environments.
method Introduces elastic consistency as a general consistency model for distributed SGD.
result Derives convergence bounds for various distributed SGD methods.
Distributed asynchronous SGD has become widely used for deep learning in large-scale systems, but remains notorious for its instability when increasing the number of workers. In this work, we study the dynamics of distributed asynchronous SGD under the lens of Lagrangian mechanics. Using this description, we introduce …