Enhances parallelism in decentralized learning for larger networks.
problem Scalability limitations in decentralized learning with increasing number of machines.
method Proposes Decentralized Anytime SGD, a novel algorithm that extends parallelism threshold.
result Establishes a theoretical upper bound on parallelism surpassing current state-of-the-art.
Most commonly used distributed machine learning systems are either synchronous or centralized asynchronous. Synchronous algorithms like AllReduce-SGD perform poorly in a heterogeneous environment, while asynchronous algorithms using a parameter server suffer from 1) communication bottleneck at parameter servers when wo…
Single global merging boosts decentralized learning performance.
problem Limited communication in decentralized learning hinders performance.
method Scheduled communication, focusing on final step with global merging.
result Single global merging improves global test performance.
This paper accelerates D-PSGD and AD-PSGD for large-scale deep learning tasks.
problem Decreasing spectral gap with increasing number of learners hampers convergence in D-PSGD and AD-PSGD.
method Improves spectral gap while minimizing communication cost through new techniques.
result Demonstrates faster training times and lower error rates on large-scale tasks.
A(DP)2SGD improves federated learning privacy and efficiency.
problem Privacy and efficiency in federated learning with asynchronous decentralized parallel SGD.
method Differentially private asynchronous decentralized parallel SGD (A(DP)2SGD) using R{é}nyi differential privacy. result Achieves optimal convergence rate and comparable model accuracy to SSGD but faster.
There is significant recent interest to parallelize deep learning algorithms in order to handle the enormous growth in data and model sizes. While most advances focus on model parallelization and engaging multiple computing agents via using a central parameter server, aspect of data parallelization along with decentral…
This paper studies the problem of error-runtime trade-off, typically encountered in decentralized training based on stochastic gradient descent (SGD) using a given network. While a denser (sparser) network topology results in faster (slower) error convergence in terms of iterations, it incurs more (less) communication …
New proof shows D-SGD and SAM are equivalent, revealing advantages of decentralization.
problem The generalization benefits of decentralized learning.
method Proved D-SGD implicitly minimizes SAM's loss function.
result Decentralized SGD and Average-direction SAM are asymptotically equivalent.
SQuARM-SGD improves decentralized SGD efficiency with momentum.
problem Efficient decentralized training of large-scale models over networks.
method Fixed local SGD steps with Nesterov's momentum, sparsified and quantized updates, locally computed triggering criterion.
result Convergence rate matches vanilla SGD, momentum improves test performance.
Moniqua improves SGD convergence with quantized communication.
problem Efficiently communicating in decentralized SGD with limited bandwidth.
method Modulo quantized communication in decentralized SGD.
result Moniqua converges at the same rate as full-precision communication with less bits.
Data parallelism has become the de facto standard for training Deep Neural Network on multiple processing units. In this work we propose DC-S3GD, a decentralized (without Parameter Server) stale-synchronous version of the Delay-Compensated Asynchronous Stochastic Gradient Descent (DC-ASGD) algorithm. In our approach, w…
Unified analysis for decentralized SGD across various topologies and updates.
problem Analysis of decentralized SGD methods with changing topologies and local updates.
method Unified convergence analysis covering local SGD updates and adaptive network topology.
result Universal convergence rates for smooth problems, interpolating between heterogeneous and iid-data settings.
The paper provides Gaussian approximations for decentralized Federated Learning.
problem Lack of asymptotic statistical guarantees for local SGD in Federated Learning.
method Two generalized Gaussian approximation results for local SGD trajectories.
result Valid multiplier bootstrap procedures and Gaussian bootstrap-based tests for detecting adversarial attacks.
Recently, the technique of local updates is a powerful tool in centralized settings to improve communication efficiency via periodical communication. For decentralized settings, it is still unclear how to efficiently combine local updates and decentralized communication. In this work, we propose an algorithm named as L…
DE-SGD shows heavy-tailed behavior in decentralized settings.
problem Heavy-tailed behavior in decentralized SGD.
method Analyzes the emergence of heavy-tails in DE-SGD, considering both quadratic and twice continuously differentiable strongly convex loss functions.
result DE-SGD exhibits heavier tails than centralized SGD, and tail behavior depends on network parameters.
The paper analyzes stability and generalization of decentralized SGD.
problem Stability and generalization of decentralized stochastic gradient descent.
method Novel formulation of decentralized stochastic gradient descent combined with non/convex optimization theory.
result First stability and generalization guarantees for decentralized stochastic gradient descent.
Novel periodic momentum SGD method for decentralized training with linear speedup.
problem Lack of effective momentum schema in decentralized training methods.
method Proposes a novel periodic decentralized momentum SGD method.
result Achieves linear speedup in decentralized training.
New analysis shows D-SGD can generalize well regardless of graph connectivity.
problem Improving generalization of D-SGD in decentralized settings.
method Algorithmic stability analysis and optimization-dependent generalization bounds.
result D-SGD can achieve generalization bounds similar to classical SGD, independent of graph connectivity.
Paper analyzes D-SGD convergence with heterogeneous data and proposes topology learning.
problem Efficiently dealing with data heterogeneity in decentralized learning.
method Revisits D-SGD analysis, introduces neighborhood heterogeneity, and proposes topology learning.
result Formulates topology learning as a tractable optimization problem and demonstrates its effectiveness.
MixML unifies analysis of weakly consistent parallel learning.
problem Lack of insight into how communication structure affects convergence in parallel learning.
method Proposes MixML framework for analyzing convergence of weakly consistent parallel machine learning.
result Shows dependency of convergence on mixing time tmix.
A new topology improves decentralized learning efficiency and accuracy.
problem Finding efficient decentralized learning topologies with fast consensus and low maximum degree.
method Proposed the Base-(k+1) Graph topology for decentralized learning. result The Base-(k+1) Graph enables faster convergence and better communication efficiency than the exponential graph. We address the issue of speeding up the training of convolutional networks. Here we study a distributed method adapted to stochastic gradient descent (SGD). The parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way to share information bet…
Stochastic Gradient Descent (SGD) is the most popular algorithm for training deep neural networks (DNNs). As larger networks and datasets cause longer training times, training on distributed systems is common and distributed SGD variants, mainly asynchronous and synchronous SGD, are widely used. Asynchronous SGD is com…
Stochastic gradient descent (SGD) is a popular stochastic optimization method in machine learning. Traditional parallel SGD algorithms, e.g., SimuParallel SGD, often require all nodes to have the same performance or to consume equal quantities of data. However, these requirements are difficult to satisfy when the paral…
MindFlayer SGD improves parallel SGD for heterogeneous, random compute times.
problem Minimizing nonconvex functions with heterogeneous, random compute times.
method MindFlayer SGD, designed for stochastic and heterogeneous delays.
result MindFlayer SGD outperforms existing methods in environments with heavy-tailed noise.
The paper proves D-SGD's stability and generalization bound, highlighting the importance of communication topology.
problem The stability and generalization of decentralized stochastic gradient descent (D-SGD).
method Theoretical analysis of D-SGD's stability and generalization bound, considering spectral gap and communication topology.
result D-SGD's generalization bound is positively correlated with the spectral gap of the communication topology.
Stochastic gradient descent~(SGD) and its variants have become more and more popular in machine learning due to their efficiency and effectiveness. To handle large-scale problems, researchers have recently proposed several parallel SGD methods for multicore systems. However, existing parallel SGD methods cannot achieve…
In this paper, we propose and analyze SPARQ-SGD, which is an event-triggered and compressed algorithm for decentralized training of large-scale machine learning models. Each node can locally compute a condition (event) which triggers a communication where quantized and sparsified local model parameters are sent. In SPA…
Improves decentralized learning by teleporting active nodes for better convergence.
problem Decentralized learning's convergence rate degrades with large node numbers.
method Activates a subset of nodes, fetches parameters from previous active nodes, updates, and performs gossip averaging on a small topology.
result Teleportation completely alleviates convergence rate degradation with proper node activation.
Decentralized optimization is emerging as a viable alternative for scalable distributed machine learning, but also introduces new challenges in terms of synchronization costs. To this end, several communication-reduction techniques, such as non-blocking communication, quantization, and local steps, have been explored i…
Improved SGD bounds for machine learning models with Markovian noise.
problem Uniform high-probability bounds for SGD under PL condition with Markovian noise.
method Combining Poisson equation for Markovian noise and probabilistic induction for almost-sure bounds.
result Matching 1/k decay rate for expected suboptimality. HybridSGD improves SGD performance by balancing computation and communication.
problem Limited scalability and performance of SGD due to communication costs.
method 2D parallel SGD method (HybridSGD) that trades off between 1D s-step SGD and 1D Federated SGD (FedAvg). result HybridSGD achieves better convergence than FedAvg at similar processor scales and up to 121x speedup over FedAvg.
Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies rem…
Proposes a method to reduce parallel complexity of MLMC in SGD.
problem Poor scalability of MLMC in SGD on parallel platforms.
method Proposes a delayed MLMC gradient estimator to reduce parallel complexity.
result Proves reduction in average parallel complexity per iteration at the cost of slightly worse convergence rate.
Adaptive quantization improves SGD accuracy in data-parallel settings.
problem Fixed gradient quantization schemes lead to suboptimal performance in deep learning.
method Developed adaptive quantization schemes ALQ and AMQ that update compression schemes based on gradient statistics.
result Improved validation accuracy on CIFAR-10 and ImageNet datasets by 2% and 1% respectively.
Decor protects decentralized learning models from curious users.
problem Privacy violation in decentralized learning.
method Decor uses correlated Gaussian noises to protect local models in decentralized SGD with differential privacy guarantees.
result Decor matches central DP optimal privacy-utility trade-off for arbitrary connected graphs.
While machine learning has achieved remarkable results in a wide variety of domains, the training of models often requires large datasets that may need to be collected from different individuals. As sensitive information may be contained in the individual's dataset, sharing training data may lead to severe privacy conc…
While training a machine learning model using multiple workers, each of which collects data from their own data sources, it would be most useful when the data collected from different workers can be {\em unique} and {\em different}. Ironically, recent analysis of decentralized parallel stochastic gradient descent (D-PS…
New method amplifies privacy in decentralized learning without centralized communication.
problem Privacy amplification in decentralized federated learning.
method Random check-in protocol for DP-SGD in FL.
result Privacy/accuracy trade-offs similar to subsampling/shuffling, but without server-initiated communication.
Improved convergence analysis for decentralized non-convex optimization.
problem Minimizing a sum of smooth non-convex functions over a network.
method Gradient tracking in decentralized stochastic gradient descent (GT-DSGD).
result GT-DSGD achieves network-independent performances matching centralized SGD under certain conditions.
New analysis shows Local SGD can achieve error scaling with only fixed number of communications.
problem Speeding up SGD by parallelizing across multiple workers with reduced communication overhead.
method Proposed and analyzed Local SGD method with a fixed number of communications independent of the number of steps.
result Achieves an error scaling as 1/(nT) with only a fixed number of communications (Ω(n)).
This work characterizes the benefits of averaging schemes widely used in conjunction with stochastic gradient descent (SGD). In particular, this work provides a sharp analysis of: (1) mini-batching, a method of averaging many samples of a stochastic gradient to both reduce the variance of the stochastic gradient estima…
Stochastic gradient descent (SGD) is a well known method for regression and classification tasks. However, it is an inherently sequential algorithm at each step, the processing of the current example depends on the parameters learned from the previous examples. Prior approaches to parallelizing linear learners using SG…
Improved time complexity for parallel stochastic optimization in heterogeneous systems.
problem Time complexity in parallel stochastic optimization for large-scale machine learning models.
method Proposes Rennala MVR, a variance-reduced extension of Rennala SGD based on momentum-based variance reduction.
result Variance reduction improves time complexity in relevant parameter regimes for parallel stochastic optimization in heterogeneous systems.
Paper tightens lower bounds on decentralized training complexity.
problem Understanding and optimizing iteration complexity in decentralized training.
method Proved a tight lower bound on iteration complexity and proposed DeTAG algorithm.
result DeTAG achieves the theoretical lower bound with only a logarithmic gap.
Stochastic Gradient Descent (SGD) has become one of the most popular optimization methods for training machine learning models on massive datasets. However, SGD suffers from two main drawbacks: (i) The noisy gradient updates have high variance, which slows down convergence as the iterates approach the optimum, and (ii)…
HPSGD speeds up DNN training by paralleling data sync with local training.
problem Low cluster utilization in distributed deep neural network training.
method Hierarchical Parallel SGD (HPSGD) with improved model updating for stale gradients.
result Significantly boosts distributed DNN training and reduces stale gradients.
This work bounds the run-time of nonconvex optimization with early stopping.
problem Bounding the expected run-time of nonconvex optimization with early stopping.
method Derives conditions for well-defined early stopping based on validation function norms and bounds the expected number of iterations and gradient evaluations.
result Guarantees the validity of early stopping and provides bounds on the expected run-time for various optimization algorithms.