Asynchronous framework improves distributed learning performance.
problem Heterogeneous computing machines hinder synchronous learning strategies.
method Asynchronous distributed framework with parameter exchanges.
result Convergence of consistency in distributed asynchronous methods for gradient iterations.
New method stabilizes asynchronous SGD by maintaining energy below that of synchronous SGD.
problem Stability issues in distributed asynchronous gradient descent.
method Gradient Energy Matching (GEM) using Lagrangian mechanics.
result GEM ensures stability and speedup in asynchronous SGD, even with many workers.
Sparsification improves convergence in asynchronous distributed SGD, even in the presence of staleness.
problem Staleness in asynchronous distributed SGD.
method Applied sparsification to reduce communication overheads in distributed asynchronous settings.
result The ergodic convergence rate of sparsified asynchronous SGD matches that of vanilla SGD, $\mathcal{O} \left( 1/\sqrt{T}
ight)$ , even in the presence of staleness.
DANA mitigates gradient staleness in asynchronous distributed SGD with momentum.
problem Gradient staleness in asynchronous distributed SGD with momentum.
method DANA: a novel technique for asynchronous distributed SGD with momentum that computes the gradient on an estimated future position of the model's parameters.
result DANA fully incorporates momentum in asynchronous training with almost no ramifications to final accuracy.
This work tackles resource allocation in asynchronous and stochastic systems.
problem Distributed resource allocation in asynchronous and stochastic settings.
method Approximate stochastic primal-dual approach with asynchronous updates.
result The Asynchronous stochastic Primal-Dual (Asyn-PD) algorithm converges to the saddle point solution at a rate of O ( 1 / t ) O(1/t) O ( 1/ t ) . Ringmaster ASGD improves Asynchronous SGD's efficiency under varying worker times.
problem Suboptimal performance of Asynchronous SGD under heterogeneous worker computation times.
method Ringmaster ASGD, a novel Asynchronous SGD method with optimal time complexity.
result Ringmaster ASGD achieves optimal time complexity under arbitrary worker heterogeneity.
Advances in asynchronous optimization methods for machine learning.
problem Efficiently solving large-scale optimization problems in machine learning.
method Asynchronous parallel and distributed optimization methods, accounting for information delays.
result Degree of asynchrony impacts convergence rates in stochastic optimization methods.
Asynchronous SGD generalizes well with enough data, improving stability and reducing error.
problem Generalization performance of asynchronous distributed SGD systems.
method Algorithm stability framework, adaptive learning rate strategy.
result Distributed asynchronous SGD generalizes well with enough data samples.
LSAM optimizes deep learning training with improved efficiency.
problem Inefficiency in distributed large-batch training with Sharpness-Aware Minimization (SAM).
method Integrates SAM's adversarial steps with an asynchronous distributed sampling strategy.
result Higher final accuracy compared to data-parallel SAM.
This work analyzes trade-offs between stragglers and gradient staleness in asynchronous distributed SGD.
problem Asynchronous distributed SGD suffers from gradient staleness that can affect convergence.
method Theoretical analysis of trade-offs between error and runtime, considering random straggler delays.
result Design of distributed SGD algorithms that balance stragglers and staleness, and a new learning rate schedule.
Unified analysis of asynchronous-SGD algorithms for distributed learning.
problem Analyzing asynchronous-SGD in heterogeneous settings with varying speeds and data distributions.
method Unified convergence theory for non-convex smooth functions, including pure asynchronous SGD and its modifications.
result Unified convergence rates for various asynchronous algorithms, including novel methods.
Asynchronous MCMC speeds up sampling with elastic coupling.
problem Parallel asynchronous sampling for stochastic gradient MCMC.
method Stochastic gradient Hamiltonian Monte Carlo with elastic coupling.
result Significantly speeds up exploration of target distribution.
A novel fully asynchronous scheme for distributed reinforcement learning over networks.
problem Policy evaluation in distributed reinforcement learning over networks.
method Design of a stochastic average gradient (SAG) based distributed algorithm and push-pull augmented graph approach.
result The proposed algorithm converges at a linear rate of \(\mathcal{O}(c^k)\) with \(c\in(0,1)\) and \(k\) increasing by one per node update.
Ringleader ASGD optimizes SGD for diverse edge devices with varying data and computation speeds.
problem Scalable distributed optimization with heterogeneous devices and data.
method Ringleader ASGD, an asynchronous SGD algorithm.
result Achieves optimal time complexity under data heterogeneity and arbitrary computation speeds.
Paper introduces elastic consistency for distributed SGD, enabling convergence analysis.
problem Training large-scale machine learning models in distributed environments.
method Introduces elastic consistency as a general consistency model for distributed SGD.
result Derives convergence bounds for various distributed SGD methods.
PARyOpt optimizes functions asynchronously, reducing wall clock time.
problem Efficiently optimizing functions on distributed systems with asynchronous evaluations.
method Parallel asynchronous Bayesian optimization.
result Reduces total optimization time for various test problems.
New algorithm reduces distributed optimization time with stochastic delays.
problem Optimizing distributed data with stochastic delays.
method Developed ADSAGA, a variant of SAGA for distributed-data settings with stochastic delays.
result ADSAGA converges in $ ilde{O}\left(\left(n + \sqrt{m}κ
ight)\log(1/ε)
ight)$ iterations under mean delay m m m . Develops an asynchronous distributed algorithm for convex optimization.
problem Distributed convex optimization with varying communication costs and delays.
method A flexible proximal gradient algorithm that adapts to different levels of communication and delays.
result The algorithm converges linearly in the strongly convex case and provides convergence guarantees for non-strongly convex functions.
A new method for asynchronous eigenspace computation on the Grassmannian.
problem Asynchronous optimization for finite-sum eigenspace computation in distributed systems.
method Grassmannian incremental aggregation method that refreshes only arriving components and reuses cached gradients.
result Two-phase linear convergence with constants controlled by component spectral spreads.
Asynchronous SGD can speed up training with a trade-off of gradient staleness.
problem Asynchronous SGD suffers from gradient staleness, affecting convergence error.
method Theoretical analysis of the error-runtime trade-off considering random straggling delays.
result A method of gradually varying synchronicity in distributed SGD is proposed and demonstrated.
New asynchronous SGD algorithms achieve optimal performance in distributed learning.
problem Asynchronous training introduces staleness, complicating optimization analysis.
method Developed rigorous framework for asynchronous first order stochastic optimization.
result Asynchronous SGD can achieve optimal time complexity, matching synchronous methods.
Asynchronous method for hyperparameter and neural architecture search.
problem Efficiently searching for optimal hyperparameters and neural architectures.
method Model-based, asynchronous multi-fidelity method combining Hyperband and Gaussian process-based Bayesian optimization.
result Substantial speed-ups over current state-of-the-art methods on various benchmarks.
DSCOVR improves distributed optimization for big data with less communication and synchronization.
problem Efficiently optimizing large linear models with convex loss functions over distributed systems.
method Randomized primal-dual block coordinate algorithms with doubly stochastic coordinate optimization and variance reduction.
result DSCOVR algorithms require less overall computation and communication compared to other first-order distributed algorithms.
Paper extends LfC to distributed learning with asynchronous optimization.
problem Learning from constraints in a distributed, nonconvex setting.
method Applying asynchronous Method of Multipliers to Learning from Constraints.
result Demonstrates privacy-preserving distributed learning with local constraints.
New Fourier transform method handles missing data and asynchronous observations.
problem Volatility inference with missing or asynchronous data.
method Spectral framework using Fourier transforms.
result Consistent volatility functional estimation with limit distributions.
ADVGP scales up Gaussian process regression to large datasets efficiently.
problem Expensive computational cost of traditional GP inference for large datasets.
method Asynchronous Distributed Variational Gaussian Process (ADVGP) with weight space augmentation and asynchronous proximal gradient optimization.
result ADVGP achieves superior prediction accuracy for large-scale regression tasks.
Novel asynchronous SGD method resists Byzantine attacks without server storage.
problem Asynchronous distributed learning with Byzantine attacks and failures.
method Buffered Asynchronous SGD (BASGD) and its momentum variant (BASGDm).
result BASGD and BASGDm resist non-omniscient and omniscient attacks without server storage.
Paper analyzes convergence and speedup of asynchronous parallel SGD.
problem Achieving good convergence and linear speedup in asynchronous parallel SGD.
method Second-order convergence analysis of APSGD with consistent read near strictly saddle points.
result Theoretical guarantee for using at most O ( K 1 / 3 M − 1 / 3 ) O(K^{1/3}M^{-1/3}) O ( K 1/3 M − 1/3 ) workers for good convergence and linear speedup. Asynchronous federated modeling improves spatial data sharing without centralizing raw data.
problem Privacy and bandwidth constraints in distributed spatial data.
method Asynchronous federated modeling using low-rank Gaussian process approximations with block-wise optimization and adaptive strategies.
result Asynchronous federated modeling achieves synchronous performance and outperforms it in heterogeneous settings.
Distributed computing offers a high degree of flexibility to accommodate modern learning constraints and the ever increasing size of datasets involved in massive data issues. Drawing inspiration from the theory of distributed computation models developed in the context of gradient-type optimization algorithms, we prese…
The paper gives bounds for how long it takes for gossip protocols to spread information in networks.
problem Understanding the diffusion time in asynchronous gossip protocols.
method Provides non-asymptotic bounds for the number of messages needed for consensus in asynchronous gossip protocols.
result Explicit formula and approximation for the number of messages needed for consensus in different types of graphs.
Asynchronous L-BFGS speeds up non-convex optimization.
problem Non-convex optimization challenges in machine learning.
method Asynchronous stochastic L-BFGS algorithm for non-convex optimization.
result Achieves an ergodic convergence rate of O ( 1 / N ) {\cal O}(1/\sqrt{N}) O ( 1/ N ) and linear speedup. New stability and convergence conditions for asynchronous SAs with biased approximations.
problem Stability and convergence issues in asynchronous SAs with biased approximation errors.
method Verifiable sufficient conditions for stability and convergence of asynchronous SAs with asymptotically biased errors.
result Stability of asynchronous SAs is unaffected by asymptotically bounded biased approximation errors.
Ringmaster LMO accelerates training in distributed systems by asynchronously updating neural networks.
problem Asynchronous training in distributed systems where workers compute gradients at different speeds.
method Introduces an asynchronous LMO-based momentum method for unconstrained stochastic nonconvex optimization.
result Establishes convergence guarantees and time complexity bounds for asynchronous LMO-based updates.
A distributed algorithm learns patterns in large images and signals.
problem High-dimensional optimization in large images and signals.
method Distributed asynchronous algorithm with locally greedy coordinate descent.
result Patterns can be learned on large scales images from the Hubble Space Telescope.
New models predict future events with uncertainty.
problem Predicting future events in asynchronous sequences with uncertainty.
method Two new architectures, WGP-LN and FD-Dir, modeling time-dependent distributions.
result Models outperform other approaches in various datasets.
New method improves convergence speed of asynchronous SGD.
problem Stale gradients in asynchronous SGD cause convergence issues.
method Quantifies staleness using moving averages of gradient statistics.
result Outperforms previous methods in convergence speed and scalability.
Asynchronous cooperative learning rules ensure all agents converge to correct hypothesis.
problem Cooperative learning in networks with unreliable communication.
method Proposed robust cooperative learning rule for weak communication networks.
result All agents' beliefs exponentially decay to the correct hypothesis.
Asynchronous algorithms reduce privacy costs in distributed machine learning.
problem Privacy concerns in training machine learning models on scattered private data.
method Differentially-private asynchronous algorithms for collaborative training.
result Cost of privacy is inversely proportional to dataset size and privacy budgets.
A new simple algorithm reduces variance for fast convergence.
problem Improving convergence rates for stochastic variance reduced algorithms.
method Introducing a simple stochastic variance reduced algorithm (MiG) with fast convergence rates.
result MiG achieves best-known convergence rates for both strongly and non-strongly convex problems.
GoSGD speeds up deep learning training with gossip exchange.
problem Speeding up the training of deep learning models.
method Distributed optimization using stochastic gradient descent with gossip algorithms.
result GoSGD achieves fully asynchronous and decentralized training.
Paper tackles reinforcement learning for asynchronous events.
problem Designing online interventions for complex asynchronous environments.
method Deep reinforcement learning of marked temporal point processes.
result Method can find effective interventions for diverse applications.
New parallel Bayesian optimization method outperforms existing techniques.
problem Expensive function evaluations in parallel settings.
method Asynchronous Thompson Sampling (TS) for Bayesian optimization.
result Asynchronous TS achieves asymptotically lower regret under time constraints.
AD-PSGD is an asynchronous decentralized parallel SGD that converges as fast as AllReduce-SGD but is much faster in a heterogeneous environment.
problem Designing an efficient and robust asynchronous decentralized parallel SGD algorithm.
method Proposes AD-PSGD, an asynchronous decentralized parallel SGD algorithm.
result AD-PSGD converges at the optimal O ( 1 / K ) O(1/\sqrt{K}) O ( 1/ K ) rate and has linear speedup w.r.t. number of workers. Freya PAGE optimizes nonconvex optimization with heterogeneous, asynchronous workers.
problem Optimizing nonconvex finite-sum problems with varying worker processing times.
method Freya PAGE, a parallel method robust to stragglers and adaptive to slow computations.
result Freya PAGE offers improved time complexity guarantees compared to previous methods.
New theory helps MCMC algorithms work well in parallel systems.
problem Proving convergence of asynchronous MCMC methods.
method Developed new convergence theory for asynchronous MCMC algorithms.
result Understood and provided guidelines for making asynchronous MCMC algorithms converge.
Unified analysis for distributed learning with compressed gradients.
problem Scalability in distributed optimization for large-scale machine learning.
method Unified analysis framework for distributed gradient methods with compressed and stale gradients.
result Explicit expressions for step-sizes and communication complexity guarantees.
Improved SAR in asynchronous conversations using neural models and unlabeled data.
problem Lack of labeled data for SAR in asynchronous conversations.
method Hierarchical LSTM-CRF model, semi-supervised learning with word embeddings, adversarial training.
result Adversarial training improves SAR performance by leveraging labeled data from synchronous domains.