Gradient coding improves distributed learning by tolerating failures.
problem Mitigating stragglers in distributed learning.
method Carefully replicating data blocks and coding across gradients.
result Shows tolerance to failures and stragglers for Synchronous Gradient Descent.
K-AVG improves convergence for nonconvex optimization problems.
problem Improving the convergence of ASGD for nonconvex optimization.
method K-step averaging stochastic gradient descent (K-AVG) for nonconvex objectives.
result K-AVG converges faster and achieves better accuracies than ASGD.
Proposes DC-S3GD for efficient large-scale decentralized neural network training.
problem Training large-scale decentralized neural networks efficiently.
method Decentralized stale-synchronous version of DC-ASGD with gradient correction.
result Achieves state-of-the-art results in training Convolutional Neural Networks.
New algorithm boosts deep learning training speed.
problem Efficiently train deep neural networks on large clusters.
method Synchronous distributed SGD with amortized inference model.
result Dynamic cutoff improves convergence and training time.
Gradient descent with delayed updates converges faster with noise, even when delays are significant.
problem Analyzing convergence of gradient descent with delayed gradients and stochastic noise.
method Novel technique using generating functions for convergence analysis.
result Convergence bounds show that stochastic noise mitigates the negative effects of delays, improving performance.
GA method reduces gradient staleness in cloud computing.
problem Gradient staleness in asynchronous SGD methods.
method Gap-Aware (GA) method that penalizes stale gradients linearly to the Gap.
result GA outperforms existing methods in final test accuracy.
CSER improves SGD efficiency by resetting errors and partial synchronization.
problem Limited scalability of Distributed Stochastic Gradient Descent (SGD) due to communication bottlenecks.
method Introduces 'error reset' technique and partial synchronization for gradients and models.
result Proves convergence for smooth non-convex problems and accelerates distributed training significantly.
Phocas improves robustness in SGD against Byzantine failures.
problem Byzantine failures in distributed SGD.
method Proposes a novel aggregation rule for Byzantine-resilient SGD.
result Proves Byzantine resilience of the proposed aggregation rules.
LSGD improves deep learning training efficiency by synchronizing and decentralizing SGD.
problem Asynchronous SGD's accuracy issues and synchronous SGD's communication inefficiency.
method LSGD divides nodes into subgroups with centralized communication and decentralized computation.
result LSGD achieves better accuracy and efficiency than synchronous and asynchronous SGD.
New method stabilizes asynchronous SGD by maintaining energy below that of synchronous SGD.
problem Stability issues in distributed asynchronous gradient descent.
method Gradient Energy Matching (GEM) using Lagrangian mechanics.
result GEM ensures stability and speedup in asynchronous SGD, even with many workers.
This paper presents Rudra, a parameter server based distributed computing framework tuned for training large-scale deep neural networks. Using variants of the asynchronous stochastic gradient descent algorithm we study the impact of synchronization protocol, stale gradient updates, minibatch size, learning rates, and n…
NLCG optimizes DNN training, especially with large mini-batches.
problem Improving convergence speed in large-scale DNN training.
method Stochastic Preconditioned Nonlinear Conjugate Gradient (SP-NLCG) algorithm.
result NLCG improves DNN training accuracy by over 10 percentage points at large mini-batch sizes.
DORE reduces communication costs in distributed learning by 95%.
problem High communication costs in distributed learning for large-scale models.
method DORE: Double Residual Compression Algorithm for stochastic gradient descent.
result DORE reduces communication costs by over 95%.
Variance reduction (VR) methods boost the performance of stochastic gradient descent (SGD) by enabling the use of larger, constant stepsizes and preserving linear convergence rates. However, current variance reduced SGD methods require either high memory usage or an exact gradient computation (using the entire dataset)…
FedAc accelerates Federated Averaging for distributed optimization.
problem Efficiently optimizing distributed machine learning models.
method Federated Accelerated Stochastic Gradient Descent (FedAc) using a potential-based perturbed iterate analysis.
result FedAc achieves faster convergence and lower communication costs than previous methods.
Proposes a new method for parallelizing SGD that combines partial results from all workers.
problem Slow workers (stragglers) cause convergence issues in synchronous SGD.
method Fixes worker computation time and combines partial results from all workers.
result Improves convergence significantly compared to existing methods.
Asynchronous SGD can speed up training with a trade-off of gradient staleness.
problem Asynchronous SGD suffers from gradient staleness, affecting convergence error.
method Theoretical analysis of the error-runtime trade-off considering random straggling delays.
result A method of gradually varying synchronicity in distributed SGD is proposed and demonstrated.
A new method reduces data movement in neural network training.
problem Large data movement during neural network training.
method Streaming batch principal component analysis for low-rank updates.
result Effective training of convolutional neural networks with low overhead.
New robust aggregation rules for SGD in the presence of Byzantine failures.
problem Byzantine failures in distributed SGD.
method Three new aggregation rules for robust SGD under a general Byzantine failure model.
result Proven Byzantine resilience of the proposed aggregation rules.
Swarming improves convergence speed in distributed stochastic optimization.
problem Faster convergence in distributed stochastic optimization.
method Inspired by swarming, each thread performs a stochastic gradient descent algorithm with a swarming potential, achieving better performance than a centralized algorithm.
result The swarming-based approach converges faster than a centralized algorithm, with a monotone decreasing error bound in network size and connectivity.
A new algorithm speeds up deep learning training by decoupling computation and communication.
problem High communication cost limits the speedup of distributed SGD.
method CoCoD-SGD: runs computation and communication in parallel.
result Linear time speedup with respect to hardware resources.
This paper surveys distributed training techniques for deep learning models.
problem Substantial compute needed for training deep learning models.
method Exploration of various algorithms and techniques for distributed training.
result Recent advancements have reduced training time from weeks to minutes.
New method reduces communication in deep learning training.
problem Communication overhead in distributed deep learning training.
method Random-block sparsification to reduce gradients communicated.
result Performance close to standard SGD with reduced communication.
NS-RGS improves orthogonal group synchronization with faster convergence.
problem Orthogonal group synchronization from pairwise measurements.
method Newton-Schulz iteration for Riemannian gradient optimization.
result NS-RGS achieves linear convergence and near-optimal accuracy.
This work analyzes trade-offs between stragglers and gradient staleness in asynchronous distributed SGD.
problem Asynchronous distributed SGD suffers from gradient staleness that can affect convergence.
method Theoretical analysis of trade-offs between error and runtime, considering random straggler delays.
result Design of distributed SGD algorithms that balance stragglers and staleness, and a new learning rate schedule.
This work investigates the impact of staleness in distributed ML systems and offers insights into convergence.
problem The effects of staleness on the convergence of distributed machine learning algorithms are inconclusive and challenging to monitor.
method Extensive experiments with various ML models and algorithms under delayed updates.
result The empirical findings reveal the diverse effects of staleness on ML algorithm convergence and match the best-known convergence rate.
AD-PSGD is an asynchronous decentralized parallel SGD that converges as fast as AllReduce-SGD but is much faster in a heterogeneous environment.
problem Designing an efficient and robust asynchronous decentralized parallel SGD algorithm.
method Proposes AD-PSGD, an asynchronous decentralized parallel SGD algorithm.
result AD-PSGD converges at the optimal O ( 1 / K ) O(1/\sqrt{K}) O ( 1/ K ) rate and has linear speedup w.r.t. number of workers. New method reduces synchronization costs in parallel optimization.
problem Inhibiting scalability of optimization methods due to communication costs.
method Extending Communication-Avoiding Krylov subspace methods to first-order, block coordinate descent methods.
result SA-variants attain large speedups of up to 5.1x on a supercomputer.
NHGD solves bilevel optimization problems with reduced computational time.
problem Solving bilevel optimization problems with high computational cost.
method Exploits statistical structure of inner optimization to use empirical Fisher matrix as Hessian surrogate, enabling parallel optimization and approximation.
result NHGD achieves error bounds and sample complexity guarantees matching state-of-the-art methods, with significantly reduced computational time.
New algorithm uses PSO to optimize DNN training parameters in distributed systems.
problem Reducing synchronization frequency in DNN training leads to poor convergence.
method Integrates PSO into distributed training to automatically compute new parameters.
result Proposed algorithm outperforms synchronous methods in distributed DNN training.
AB dynamically scales gradients to mitigate asynchronous training delays.
problem Gradient delay in asynchronous training reduces model performance.
method Adaptive Braking (AB) dynamically scales gradients based on alignment.
result AB enables training with up to 32 update steps of delay without accuracy loss.
Nonconvex optimization improves low-rank matrix factorization with statistical models.
problem Improving low-rank matrix factorization via nonconvex optimization.
method Two-stage algorithms and global landscape analysis.
result Provable accuracy and efficiency of nonconvex optimization methods.
Adaptive batch sizes improve local gradient methods in distributed training.
problem Communication bottlenecks in distributed deep learning.
method Adaptive batch size strategies for local gradient methods.
result Adaptive batch sizes reduce minibatch gradient variance and improve training efficiency.
SplitNN enables deep learning model training without sharing patient data.
problem Training deep learning models without exposing sensitive patient data.
method Distributed deep learning method called SplitNN.
result SplitNN outperforms other distributed learning methods in performance and resource efficiency.
AdaBatch speeds up SGD for sparse problems with minimal code changes.
problem Efficiently aggregating gradients in stochastic optimization for sparse problems.
method AdaBatch: a new aggregation operator for mini-batches.
result AdaBatch achieves similar sample efficiency to standard SGD but with larger batch sizes.
Efficient deep learning on exascale supercomputers solves materials imaging inverse problems.
problem Solving scientific inverse problems in materials imaging using deep learning.
method Novel communication strategies in synchronous distributed deep learning, including decentralized gradient reduction and computational graph-aware grouping.
result Achieved near-linear scaling of distributed training up to 27,600 GPUs on Summit, reaching 2.15(4) EFLOPS 16 _{16} 16 . Paper addresses group synchronization with incomplete measurements and proves linear convergence of GPM.
problem Orthogonal group synchronization with incomplete measurements and additive noise.
method Generalized power method (GPM) with local error bound analysis.
result Linear convergence of GPM to a global maximizer under general additive noise model.
Asynchronous SGD generalizes well with enough data, improving stability and reducing error.
problem Generalization performance of asynchronous distributed SGD systems.
method Algorithm stability framework, adaptive learning rate strategy.
result Distributed asynchronous SGD generalizes well with enough data samples.
Gradient descent and noisy gradient descent explored on simple functions.
problem Behavior of gradient descent and noisy gradient descent on simple functions.
method Computer experiments with gradient descent and noisy gradient descent on simple functions.
result Noise affects the trajectory of gradient descent on simple functions.
Blind Descent avoids gradient issues, using a different learning approach.
problem Gradient issues like exploding and vanishing gradients.
method Does not use gradients to guide learning; instead, it is a more fundamental learning process.
result Gradient descent is a specific case of Blind Descent.
HPSGD speeds up DNN training by paralleling data sync with local training.
problem Low cluster utilization in distributed deep neural network training.
method Hierarchical Parallel SGD (HPSGD) with improved model updating for stale gradients.
result Significantly boosts distributed DNN training and reduces stale gradients.
LAGS-SGD optimizes deep learning training by sparsifying gradients layer-wise.
problem Reduces long training times in large deep neural networks with distributed S-SGD.
method Layer-wise adaptive gradient sparsification combined with S-SGD.
result LAGS-SGD achieves convergence guarantees and outperforms vanilla S-SGD.
Study on noisy gradient descent in higher-dimensional minima.
problem Behavior of gradient descent in higher codimension.
method Computer experiments with noisy gradient descent.
result Effects of noise on gradient descent trajectories in higher codimension.
Sideways trains video models by overwriting activations as new frames arrive, potentially improving generalization.
problem Training deep video models synchronously slows down and requires storing activations, limiting parallelism.
method Sideways trains video models by overwriting activations as new frames arrive, breaking the precise correspondence between gradients and activations.
result Sideways training can converge and potentially generalize better than standard synchronized backpropagation.
New algorithm reduces communication in distributed SGD, improving efficiency.
problem Slow communication rounds bottleneck synchronous mini-batch SGD convergence.
method Proposes non-asymptotic error analysis for Local-SGD, comparing to averaging methods.
result Local-SGD reduces communication by a factor of O(√T/P^(3/2)) for large step sizes.
Stochastic gradient descent on manifolds improves low-rank approximation.
problem Efficiently approximate large matrices with lower rank.
method Stochastic gradient descent on a manifold.
result Algorithm outperforms Euclidean space methods on Netflix Prize data.
CodedReduce combines tree topology and gradient coding for efficient and resilient gradient aggregation.
problem Efficient and robust gradient aggregation in distributed learning.
method CodedReduce combines tree topology and gradient coding to overcome bandwidth bottlenecks and straggler delays.
result CodedReduce achieves up to 27.2x speedup over benchmarks GC and RAR.
Adaptive learning rate improves gradient descent for deep models.
problem Difficulty in tuning fixed learning rates for deep models.
method Proposes adaptive learning rates using either first-order or second-order methods.
result Optimized gradient descent leads to better convergence for machine learning algorithms.