Unified framework for online FDR control in asynchronous testing.
problem Control of false discovery rate in decentralized, asynchronous testing.
method Unified computational abstraction called 'conflict sets' for handling dependencies.
result Formal FDR guarantees under local dependence, improving on classical methods.
PARyOpt optimizes functions asynchronously, reducing wall clock time.
problem Efficiently optimizing functions on distributed systems with asynchronous evaluations.
method Parallel asynchronous Bayesian optimization.
result Reduces total optimization time for various test problems.
POAP and pySOT improve surrogate optimization of expensive functions.
problem Optimizing expensive functions with concurrent evaluations.
method Event-driven asynchronous framework for optimization strategies.
result Asynchronous computation offers significant speed-up advantages.
AB dynamically scales gradients to mitigate asynchronous training delays.
problem Gradient delay in asynchronous training reduces model performance.
method Adaptive Braking (AB) dynamically scales gradients based on alignment.
result AB enables training with up to 32 update steps of delay without accuracy loss.
New method speeds up lead-lag detection between asynchronous time series.
problem Slow inference of lead-lag networks between long time series.
method Derive asymptotic distribution of Transfer Entropy and introduce time-shifted time series.
result Statistically validated lead-lag networks between time series.
Enhances Fourier estimator performance for asynchronous event-data.
problem Improving correlation and covariance estimation on event-data.
method Implement and test NUFFT methods with different averaging kernels.
result Demonstrates improved performance and relationship between averaging scales.
New gossip algorithms improve robustness of rank-based statistics in decentralized systems.
problem Ensuring robustness in decentralized AI and edge intelligence systems, especially in the presence of corrupted or adversarial data.
method Developed asynchronous gossip algorithms for computing rank-based statistics.
result First convergence rate bound for asynchronous gossip-based rank estimation.
GA method reduces gradient staleness in cloud computing.
problem Gradient staleness in asynchronous SGD methods.
method Gap-Aware (GA) method that penalizes stale gradients linearly to the Gap.
result GA outperforms existing methods in final test accuracy.
This paper accelerates D-PSGD and AD-PSGD for large-scale deep learning tasks.
problem Decreasing spectral gap with increasing number of learners hampers convergence in D-PSGD and AD-PSGD.
method Improves spectral gap while minimizing communication cost through new techniques.
result Demonstrates faster training times and lower error rates on large-scale tasks.
PipeMare enables efficient DNN training with minimal memory and pipeline sacrifices.
problem Sacrificing hardware efficiency to maintain statistical efficiency in pipeline parallel DNN training.
method PipeMare is a simple yet robust training method that tolerates asynchronous updates during pipeline parallelism without sacrificing pipeline utilization or memory.
result PipeMare achieves up to 2.7x less memory usage or 4.3x higher pipeline utilization compared to state-of-the-art synchronous PP training techniques.
Ringmaster ASGD improves Asynchronous SGD's efficiency under varying worker times.
problem Suboptimal performance of Asynchronous SGD under heterogeneous worker computation times.
method Ringmaster ASGD, a novel Asynchronous SGD method with optimal time complexity.
result Ringmaster ASGD achieves optimal time complexity under arbitrary worker heterogeneity.
Improves BO efficiency by allowing asynchronous parallel computing.
problem Wasteful use of resources in batch Bayesian optimisation.
method Developed PLAyBOOK for asynchronous local penalisation.
result Asynchronous BO often outperforms synchronous BO.
Zeno++ improves robustness of asynchronous SGD in fully asynchronous settings.
problem Byzantine failures in fully asynchronous SGD.
method Estimates descent of loss after applying candidate gradient.
result Proves convergence for non-convex problems under Byzantine failures.
Paper proposes a more robust deep reinforcement learning agent.
problem Creating more robust reinforcement learning agents.
method Modified A3C algorithm with dual input streams.
result Significant reduction in training parameters (30%) with improved robustness.
New framework simplifies analysis of asynchronous parallel optimization algorithms.
problem Difficulty in analyzing asynchronous parallel optimization algorithms due to delay and inconsistency.
method Revisit and clarify technical issue in convergence rate proofs, propose simplified 'perturbed iterate' framework.
result Proves linear speedup for ASAGA and KROMAGNON on multi-core systems without sparsity assumptions.
New method stabilizes asynchronous SGD by maintaining energy below that of synchronous SGD.
problem Stability issues in distributed asynchronous gradient descent.
method Gradient Energy Matching (GEM) using Lagrangian mechanics.
result GEM ensures stability and speedup in asynchronous SGD, even with many workers.
AEGiS optimizes expensive function evaluations asynchronously.
problem Optimizing expensive black-box functions efficiently.
method Asynchronous ε-Greedy Bayesian Optimisation combining greedy search and Thompson sampling.
result AEGiS outperforms existing asynchronous BO methods.
Secure aggregation for buffered asynchronous federated learning without TEEs.
problem Privacy and convergence in buffered asynchronous federated learning.
method Developed a new protocol (BASecAgg) that ensures privacy without TEEs by carefully designing masks.
result BASecAgg achieves similar convergence guarantees as FedBuff without TEEs.
Standard acquisition functions are sufficient for asynchronous Bayesian optimization.
problem Redundant and repeated queries in asynchronous Bayesian optimization.
method Conceptual analysis and theoretical guarantees of standard acquisitions.
result Standard acquisition functions achieve theoretical guarantees equivalent to Thompson sampling in asynchronous settings.
Paper analyzes convergence and speedup of asynchronous parallel SGD.
problem Achieving good convergence and linear speedup in asynchronous parallel SGD.
method Second-order convergence analysis of APSGD with consistent read near strictly saddle points.
result Theoretical guarantee for using at most O ( K 1 / 3 M − 1 / 3 ) O(K^{1/3}M^{-1/3}) O ( K 1/3 M − 1/3 ) workers for good convergence and linear speedup. Asynchronous method for hyperparameter and neural architecture search.
problem Efficiently searching for optimal hyperparameters and neural architectures.
method Model-based, asynchronous multi-fidelity method combining Hyperband and Gaussian process-based Bayesian optimization.
result Substantial speed-ups over current state-of-the-art methods on various benchmarks.
This work tackles resource allocation in asynchronous and stochastic systems.
problem Distributed resource allocation in asynchronous and stochastic settings.
method Approximate stochastic primal-dual approach with asynchronous updates.
result The Asynchronous stochastic Primal-Dual (Asyn-PD) algorithm converges to the saddle point solution at a rate of O ( 1 / t ) O(1/t) O ( 1/ t ) . Advances in asynchronous optimization methods for machine learning.
problem Efficiently solving large-scale optimization problems in machine learning.
method Asynchronous parallel and distributed optimization methods, accounting for information delays.
result Degree of asynchrony impacts convergence rates in stochastic optimization methods.
New Fourier transform method handles missing data and asynchronous observations.
problem Volatility inference with missing or asynchronous data.
method Spectral framework using Fourier transforms.
result Consistent volatility functional estimation with limit distributions.
New rules found to maintain neural network performance in asynchronous training.
problem Asynchronous training leads to degradation in generalization.
method Examined dynamical stability, derived rules for learning rate adjustment.
result Learning rate should be inversely proportional to delay for high delay values.
DANA mitigates gradient staleness in asynchronous distributed SGD with momentum.
problem Gradient staleness in asynchronous distributed SGD with momentum.
method DANA: a novel technique for asynchronous distributed SGD with momentum that computes the gradient on an estimated future position of the model's parameters.
result DANA fully incorporates momentum in asynchronous training with almost no ramifications to final accuracy.
FedBuff improves federated learning scalability with asynchronous updates.
problem Limited scalability of federated learning with synchronous updates.
method Introduces asynchronous updates (staleness) in federated learning.
result Theoretical analysis shows improved convergence rate with boundedness removed.
Sparsification improves convergence in asynchronous distributed SGD, even in the presence of staleness.
problem Staleness in asynchronous distributed SGD.
method Applied sparsification to reduce communication overheads in distributed asynchronous settings.
result The ergodic convergence rate of sparsified asynchronous SGD matches that of vanilla SGD, $\mathcal{O} \left( 1/\sqrt{T}
ight)$ , even in the presence of staleness.
The asymptotic pseudo-trajectory approach to stochastic approximation of Benaim, Hofbauer and Sorin is extended for asynchronous stochastic approximations with a set-valued mean field. The asynchronicity of the process is incorporated into the mean field to produce convergence results which remain similar to those of a…
We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures. Roughly, the n…
The paper explores distributed deep learning strategies for ASR, achieving faster training times.
problem Improving ASR performance and training speed with distributed deep learning.
method Investigated and implemented various distributed strategies including SYNC, ADPSGD, and HYBRID.
result ADPSGD achieved a 7.6% WER on the Hub5-2000 Switchboard test set in 14 hours with 16 NVIDIA P100 GPUs.
Unified analysis of asynchronous-SGD algorithms for distributed learning.
problem Analyzing asynchronous-SGD in heterogeneous settings with varying speeds and data distributions.
method Unified convergence theory for non-convex smooth functions, including pure asynchronous SGD and its modifications.
result Unified convergence rates for various asynchronous algorithms, including novel methods.
Paper proves CLTs for Q-learning with asynchronous updates.
problem Establishing convergence rates for Q-learning algorithms.
method Polyak-Ruppert averaging, non-asymptotic and functional CLTs.
result Convergence rates in Wasserstein distance for Q-learning.
Accelerates optimization in asynchronous systems with sparse updates.
problem Optimizing finite-sum objectives in asynchronous lock-free environments.
method New accelerated SVRG variant with sparse updates.
result Achieves optimal incremental gradient complexity.
New stability and convergence conditions for asynchronous SAs with biased approximations.
problem Stability and convergence issues in asynchronous SAs with biased approximation errors.
method Verifiable sufficient conditions for stability and convergence of asynchronous SAs with asymptotically biased errors.
result Stability of asynchronous SAs is unaffected by asymptotically bounded biased approximation errors.
Improves asynchronous federated learning with queuing dynamics.
problem Asynchronous federated learning with varying node computational speeds.
method Proposes a non-uniform sampling scheme for the central server.
result Significant improvement over current algorithms on image classification.
AD-PSGD is an asynchronous decentralized parallel SGD that converges as fast as AllReduce-SGD but is much faster in a heterogeneous environment.
problem Designing an efficient and robust asynchronous decentralized parallel SGD algorithm.
method Proposes AD-PSGD, an asynchronous decentralized parallel SGD algorithm.
result AD-PSGD converges at the optimal O ( 1 / K ) O(1/\sqrt{K}) O ( 1/ K ) rate and has linear speedup w.r.t. number of workers. We study optimization algorithms based on variance reduction for stochastic gradient descent (SGD). Remarkable recent progress has been made in this direction through development of algorithms like SAG, SVRG, SAGA. These algorithms have been shown to outperform SGD, both theoretically and empirically. However, asynchro…
Asynchronous framework speeds up model-based RL to real-time.
problem Real-time learning on real robots with model-based RL.
method Asynchronous framework for model-based reinforcement learning.
result Reduced run time to data collection time, improved sample complexity.
Ringleader ASGD optimizes SGD for diverse edge devices with varying data and computation speeds.
problem Scalable distributed optimization with heterogeneous devices and data.
method Ringleader ASGD, an asynchronous SGD algorithm.
result Achieves optimal time complexity under data heterogeneity and arbitrary computation speeds.
Combines multi-fidelity and asynchronous batch methods for faster experimental design.
problem Designing optimal experimental setups for battery performance.
method Algorithm combining multi-fidelity and asynchronous batch Bayesian Optimization.
result Algorithm outperforms single-fidelity batch and multi-fidelity sequential methods.
In many distributed learning problems, the heterogeneous loading of computing machines may harm the overall performance of synchronous strategies. In this paper, we propose an effective asynchronous distributed framework for the minimization of a sum of smooth functions, where each machine performs iterations in parall…
LSAM optimizes deep learning training with improved efficiency.
problem Inefficiency in distributed large-batch training with Sharpness-Aware Minimization (SAM).
method Integrates SAM's adversarial steps with an asynchronous distributed sampling strategy.
result Higher final accuracy compared to data-parallel SAM.
Asynchronous Gibbs sampling can accurately estimate expectations of functions of all variables under certain conditions.
problem Estimating expectations of functions of all variables in graphical models.
method Coupling synchronous and asynchronous Gibbs samplers to control expected Hamming distance, using concentration of measure results.
result The bias in estimating expectations of polynomial functions is smaller than the standard deviation of the function value in the true model.
This work analyzes trade-offs between stragglers and gradient staleness in asynchronous distributed SGD.
problem Asynchronous distributed SGD suffers from gradient staleness that can affect convergence.
method Theoretical analysis of trade-offs between error and runtime, considering random straggler delays.
result Design of distributed SGD algorithms that balance stragglers and staleness, and a new learning rate schedule.
Freya PAGE optimizes nonconvex optimization with heterogeneous, asynchronous workers.
problem Optimizing nonconvex finite-sum problems with varying worker processing times.
method Freya PAGE, a parallel method robust to stragglers and adaptive to slow computations.
result Freya PAGE offers improved time complexity guarantees compared to previous methods.
The paper gives bounds for how long it takes for gossip protocols to spread information in networks.
problem Understanding the diffusion time in asynchronous gossip protocols.
method Provides non-asymptotic bounds for the number of messages needed for consensus in asynchronous gossip protocols.
result Explicit formula and approximation for the number of messages needed for consensus in different types of graphs.
A new simple algorithm reduces variance for fast convergence.
problem Improving convergence rates for stochastic variance reduced algorithms.
method Introducing a simple stochastic variance reduced algorithm (MiG) with fast convergence rates.
result MiG achieves best-known convergence rates for both strongly and non-strongly convex problems.