LSGD improves deep learning training efficiency by synchronizing and decentralizing SGD.
problem Asynchronous SGD's accuracy issues and synchronous SGD's communication inefficiency.
method LSGD divides nodes into subgroups with centralized communication and decentralized computation.
result LSGD achieves better accuracy and efficiency than synchronous and asynchronous SGD.
CSER improves SGD efficiency by resetting errors and partial synchronization.
problem Limited scalability of Distributed Stochastic Gradient Descent (SGD) due to communication bottlenecks.
method Introduces 'error reset' technique and partial synchronization for gradients and models.
result Proves convergence for smooth non-convex problems and accelerates distributed training significantly.
New algorithm reduces communication in distributed SGD, improving efficiency.
problem Slow communication rounds bottleneck synchronous mini-batch SGD convergence.
method Proposes non-asymptotic error analysis for Local-SGD, comparing to averaging methods.
result Local-SGD reduces communication by a factor of O(√T/P^(3/2)) for large step sizes.
Most commonly used distributed machine learning systems are either synchronous or centralized asynchronous. Synchronous algorithms like AllReduce-SGD perform poorly in a heterogeneous environment, while asynchronous algorithms using a parameter server suffer from 1) communication bottleneck at parameter servers when wo…
Local SGD with periodic averaging achieves faster convergence with less communication.
problem Communication overhead in distributed optimization.
method Local SGD with periodic averaging, Polyak-Łojasiewicz condition, adaptive synchronization.
result Local SGD can achieve linear speed up with fewer communication rounds, especially for non-strongly convex functions.
Asynchronous SGD can speed up training with a trade-off of gradient staleness.
problem Asynchronous SGD suffers from gradient staleness, affecting convergence error.
method Theoretical analysis of the error-runtime trade-off considering random straggling delays.
result A method of gradually varying synchronicity in distributed SGD is proposed and demonstrated.
SlowMo improves distributed SGD by periodically synchronizing and using momentum, leading to better accuracy.
problem Reducing communication overhead in distributed training while maintaining model accuracy.
method Periodically synchronizing workers and using momentum after multiple iterations of a base optimization algorithm.
result SlowMo consistently yields improvements in optimization and generalization performance relative to the base optimizer.
New asynchronous SGD algorithms achieve optimal performance in distributed learning.
problem Asynchronous training introduces staleness, complicating optimization analysis.
method Developed rigorous framework for asynchronous first order stochastic optimization.
result Asynchronous SGD can achieve optimal time complexity, matching synchronous methods.
Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies rem…
We propose three new robust aggregation rules for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilie…
HPSGD speeds up DNN training by paralleling data sync with local training.
problem Low cluster utilization in distributed deep neural network training.
method Hierarchical Parallel SGD (HPSGD) with improved model updating for stale gradients.
result Significantly boosts distributed DNN training and reduces stale gradients.
In this paper, we focus on approaches to parallelizing stochastic gradient descent (SGD) wherein data is farmed out to a set of workers, the results of which, after a number of updates, are then combined at a central master node. Although such synchronized SGD approaches parallelize well in idealized computing environm…
Asynchronous decentralized SGD with quantized and local updates converges in gossip model.
problem Scalable distributed machine learning challenges in decentralized optimization.
method Asynchronous decentralized SGD with quantization and local steps in gossip model.
result Decentralized optimization with quantization and local steps converges in asynchronous gossip model.
ADPSGD optimizes SGD convergence by varying averaging periods based on model variance.
problem High communication costs in parallel SGD.
method Varying averaging period based on model variance.
result ADPSGD achieves better convergence and smaller communication compared to CPSGD.
Proposes DC-S3GD for efficient large-scale decentralized neural network training.
problem Training large-scale decentralized neural networks efficiently.
method Decentralized stale-synchronous version of DC-ASGD with gradient correction.
result Achieves state-of-the-art results in training Convolutional Neural Networks.
Recently, new defense techniques have been developed to tolerate Byzantine failures for distributed machine learning. The Byzantine model captures workers that behave arbitrarily, including malicious and compromised workers. In this paper, we break two prevailing Byzantine-tolerant techniques. Specifically we show robu…
Distributed training of deep nets is an important technique to address some of the present day computing challenges like memory consumption and computational demands. Classical distributed approaches, synchronous or asynchronous, are based on the parameter server architecture, i.e., worker nodes compute gradients which…
Overlap-Local-SGD improves distributed SGD by overlapping communication and computation.
problem High communication delay and node slowdown in distributed SGD.
method Adding an anchor model to synchronize local updates and pull them towards the anchor model.
result Overlap-Local-SGD speeds up distributed training and mitigates straggler effects.
Qsparse-local-SGD reduces communication in large-scale learning models.
problem Communication bottleneck in distributed optimization of large-scale models.
method Combines sparsification, quantization, and local computation with error compensation.
result Converges at the same rate as vanilla distributed SGD for many sparsifiers and quantizers.
Distributed Stochastic Gradient Descent (SGD) when run in a synchronous manner, suffers from delays in waiting for the slowest learners (stragglers). Asynchronous methods can alleviate stragglers, but cause gradient staleness that can adversely affect convergence. In this work we present a novel theoretical characteriz…
New convergence bounds for shuffling-based SGD methods in distributed learning.
problem Analyzing the performance of shuffling-based variants of SGD in distributed learning.
method Study of minibatch and local Random Reshuffling methods, proving convergence bounds and lower bounds.
result Shuffling-based variants converge faster than with-replacement sampling methods, and the bounds are tight.
A new algorithm speeds up deep learning training by decoupling computation and communication.
problem High communication cost limits the speedup of distributed SGD.
method CoCoD-SGD: runs computation and communication in parallel.
result Linear time speedup with respect to hardware resources.
Unified analysis for decentralized SGD across various topologies and updates.
problem Analysis of decentralized SGD methods with changing topologies and local updates.
method Unified convergence analysis covering local SGD updates and adaptive network topology.
result Universal convergence rates for smooth problems, interpolating between heterogeneous and iid-data settings.
Distributed asynchronous SGD has become widely used for deep learning in large-scale systems, but remains notorious for its instability when increasing the number of workers. In this work, we study the dynamics of distributed asynchronous SGD under the lens of Lagrangian mechanics. Using this description, we introduce …
LAGS-SGD optimizes deep learning training by sparsifying gradients layer-wise.
problem Reduces long training times in large deep neural networks with distributed S-SGD.
method Layer-wise adaptive gradient sparsification combined with S-SGD.
result LAGS-SGD achieves convergence guarantees and outperforms vanilla S-SGD.
Asynchronous SGD generalizes well with enough data, improving stability and reducing error.
problem Generalization performance of asynchronous distributed SGD systems.
method Algorithm stability framework, adaptive learning rate strategy.
result Distributed asynchronous SGD generalizes well with enough data samples.
Mixed-precision CA-SGD for generalized linear models on GPUs
problem SGD communication bottleneck
method Mixed-precision CA-SGD
result Matches FP32 SGD loss within 0.5% on various problems
Unified analysis of asynchronous-SGD algorithms for distributed learning.
problem Analyzing asynchronous-SGD in heterogeneous settings with varying speeds and data distributions.
method Unified convergence theory for non-convex smooth functions, including pure asynchronous SGD and its modifications.
result Unified convergence rates for various asynchronous algorithms, including novel methods.
We propose a novel robust aggregation rule for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilience…
GA method reduces gradient staleness in cloud computing.
problem Gradient staleness in asynchronous SGD methods.
method Gap-Aware (GA) method that penalizes stale gradients linearly to the Gap.
result GA outperforms existing methods in final test accuracy.
Hyperparameter tuning is one of the most time-consuming workloads in deep learning. State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable. Recently researchers have shown renewed interest in simpler methods like momentum SGD …
Variance reduction (VR) methods boost the performance of stochastic gradient descent (SGD) by enabling the use of larger, constant stepsizes and preserving linear convergence rates. However, current variance reduced SGD methods require either high memory usage or an exact gradient computation (using the entire dataset)…
AB dynamically scales gradients to mitigate asynchronous training delays.
problem Gradient delay in asynchronous training reduces model performance.
method Adaptive Braking (AB) dynamically scales gradients based on alignment.
result AB enables training with up to 32 update steps of delay without accuracy loss.
A(DP)2SGD improves federated learning privacy and efficiency.
problem Privacy and efficiency in federated learning with asynchronous decentralized parallel SGD.
method Differentially private asynchronous decentralized parallel SGD (A(DP)2SGD) using R{é}nyi differential privacy. result Achieves optimal convergence rate and comparable model accuracy to SSGD but faster.
FedAc accelerates Federated Averaging for distributed optimization.
problem Efficiently optimizing distributed machine learning models.
method Federated Accelerated Stochastic Gradient Descent (FedAc) using a potential-based perturbed iterate analysis.
result FedAc achieves faster convergence and lower communication costs than previous methods.
A new method reduces communication costs in distributed learning.
problem Reduces communication bottlenecks in distributed learning.
method Local SGD with communication-computation overlap and delay-corrected sparse model averaging.
result Theoretical convergence guarantees for smooth non-convex objectives.
Adaptive batch sizes improve local gradient methods in distributed training.
problem Communication bottlenecks in distributed deep learning.
method Adaptive batch size strategies for local gradient methods.
result Adaptive batch sizes reduce minibatch gradient variance and improve training efficiency.
Asynchronous federated learning for vertically partitioned data improves efficiency and privacy.
problem Efficiently train models on vertically partitioned data without a trusted third party.
method Proposed AFSGD-VP and its SVRG and SAGA variants for asynchronous federated learning.
result AFSGD-VP and its variants achieve higher efficiency than synchronous algorithms.
New method reduces communication in deep learning training.
problem Communication overhead in distributed deep learning training.
method Random-block sparsification to reduce gradients communicated.
result Performance close to standard SGD with reduced communication.
Reducing communication in training large-scale machine learning applications on distributed platform is still a big challenge. To address this issue, we propose a distributed hierarchical averaging stochastic gradient descent (Hier-AVG) algorithm with infrequent global reduction by introducing local reduction. As a gen…
A new hybrid-ordered SGD method reduces communication and complexity for non-convex optimization.
problem Balancing communication, computational complexity, and convergence rate in distributed non-convex optimization.
method Hybrid-ordered distributed SGD with pre-shared scalers and periodic vector communication.
result Order-wise faster convergence compared to existing methods.
Single global merging boosts decentralized learning performance.
problem Limited communication in decentralized learning hinders performance.
method Scheduled communication, focusing on final step with global merging.
result Single global merging improves global test performance.
Large-scale machine learning training, in particular distributed stochastic gradient descent, needs to be robust to inherent system variability such as node straggling and random communication delays. This work considers a distributed training framework where each worker node is allowed to perform local model updates a…
Ringmaster LMO accelerates training in distributed systems by asynchronously updating neural networks.
problem Asynchronous training in distributed systems where workers compute gradients at different speeds.
method Introduces an asynchronous LMO-based momentum method for unconstrained stochastic nonconvex optimization.
result Establishes convergence guarantees and time complexity bounds for asynchronous LMO-based updates.
DBS dynamically adjusts batch sizes to improve cluster utilization.
problem Inefficient use of high-performance workers in synchronous distributed training.
method Dynamic Batch Size (DBS) strategy that adjusts batch size and dataset partition based on worker performance.
result Significantly improves cluster utilization and reduces training time.
Novel asynchronous SGD method resists Byzantine attacks without server storage.
problem Asynchronous distributed learning with Byzantine attacks and failures.
method Buffered Asynchronous SGD (BASGD) and its momentum variant (BASGDm).
result BASGD and BASGDm resist non-omniscient and omniscient attacks without server storage.
A new method reduces communication in Federated Learning by pulling less often.
problem Reducing communication overhead in Federated Learning.
method Pulling Reduction with Local Compensation (PRLC) for SGD.
result PRLC achieves lower pulling frequency and maintains the same convergence rate as synchronous SGD.
Nonlinear conjugate gradient (NLCG) based optimizers have shown superior loss convergence properties compared to gradient descent based optimizers for traditional optimization problems. However, in Deep Neural Network (DNN) training, the dominant optimization algorithm of choice is still Stochastic Gradient Descent (SG…