Paper introduces an asynchronous optimization algorithm for parameter servers.
problem Solving optimization problems with asynchrony and general regularizers.
method Asynchronous incremental aggregated gradient algorithm in a parameter server framework.
result Established linear convergence rate and step-size choices for strong convex data loss.
Asynch-SGBDT speeds up GBDT training on Parameter Server.
problem Training GBDT on Parameter Server is slow due to computational resources and time.
method Developed asynch-SGBDT, an asynchronous parallel method for GBDT training.
result Asynch-SGBDT achieves linear speedup in asynchronous parallel training.
AD-PSGD is an asynchronous decentralized parallel SGD that converges as fast as AllReduce-SGD but is much faster in a heterogeneous environment.
problem Designing an efficient and robust asynchronous decentralized parallel SGD algorithm.
method Proposes AD-PSGD, an asynchronous decentralized parallel SGD algorithm.
result AD-PSGD converges at the optimal O ( 1 / K ) O(1/\sqrt{K}) O ( 1/ K ) rate and has linear speedup w.r.t. number of workers. DSCOVR improves distributed optimization for big data with less communication and synchronization.
problem Efficiently optimizing large linear models with convex loss functions over distributed systems.
method Randomized primal-dual block coordinate algorithms with doubly stochastic coordinate optimization and variance reduction.
result DSCOVR algorithms require less overall computation and communication compared to other first-order distributed algorithms.
Enhanced federated learning reduces communication costs and improves model accuracy.
problem Reducing communication costs in federated learning.
method Asynchronous model update and temporally weighted aggregation.
result The proposed algorithm outperforms baseline in terms of communication cost and model accuracy.
New theory helps MCMC algorithms work well in parallel systems.
problem Proving convergence of asynchronous MCMC methods.
method Developed new convergence theory for asynchronous MCMC algorithms.
result Understood and provided guidelines for making asynchronous MCMC algorithms converge.
APS-LDA scales topic modeling to web-scale data using Spark.
problem Scaling topic models to web-scale data.
method Integrates Spark with LDA using an asynchronous parameter server.
result APS-LDA processes up to 135x more data and 10x more topics than Spark LDA.
This work tackles resource allocation in asynchronous and stochastic systems.
problem Distributed resource allocation in asynchronous and stochastic settings.
method Approximate stochastic primal-dual approach with asynchronous updates.
result The Asynchronous stochastic Primal-Dual (Asyn-PD) algorithm converges to the saddle point solution at a rate of O ( 1 / t ) O(1/t) O ( 1/ t ) . New algorithm solves distributed ML with no trusted central server.
problem Tackles Byzantine failures in distributed machine learning.
method ByzSGD algorithm using Scatter/Gather, DMC, and MDA.
result Solves general Byzantine-resilient distributed ML problem.
New algorithm reduces distributed optimization time with stochastic delays.
problem Optimizing distributed data with stochastic delays.
method Developed ADSAGA, a variant of SAGA for distributed-data settings with stochastic delays.
result ADSAGA converges in $ ilde{O}\left(\left(n + \sqrt{m}κ
ight)\log(1/ε)
ight)$ iterations under mean delay m m m . Asynchronous distributed stochastic gradient descent methods have trouble converging because of stale gradients. A gradient update sent to a parameter server by a client is stale if the parameters used to calculate that gradient have since been updated on the server. Approaches have been proposed to circumvent this pro…
A(DP) 2 ^2 2 SGD improves federated learning privacy and efficiency.
problem Privacy and efficiency in federated learning with asynchronous decentralized parallel SGD.
method Differentially private asynchronous decentralized parallel SGD (A(DP) 2 ^2 2 SGD) using R{é}nyi differential privacy. result Achieves optimal convergence rate and comparable model accuracy to SSGD but faster.
Novel optimizer mitigates instability in asynchronous data parallel optimization.
problem Instability in asynchronous data parallel optimization.
method Pushes a normalized sequence of first-order gradients to a parameter server.
result Achieves better convergence rate compared to other optimizers.
Improves asynchronous federated learning with queuing dynamics.
problem Asynchronous federated learning with varying node computational speeds.
method Proposes a non-uniform sampling scheme for the central server.
result Significant improvement over current algorithms on image classification.
Pipe-SGD improves deep net training time by 5.4x with pipelined AllReduce.
problem Efficiently training deep neural networks on distributed systems.
method Pipelined SGD with two worker nodes, considering network latency and bandwidth.
result Up to 5.4x improvement in wall-clock training time.
In distributed ML applications, shared parameters are usually replicated among computing nodes to minimize network overhead. Therefore, proper consistency model must be carefully chosen to ensure algorithm's correctness and provide high throughput. Existing consistency models used in general-purpose databases and moder…
Novel asynchronous SGD method resists Byzantine attacks without server storage.
problem Asynchronous distributed learning with Byzantine attacks and failures.
method Buffered Asynchronous SGD (BASGD) and its momentum variant (BASGDm).
result BASGD and BASGDm resist non-omniscient and omniscient attacks without server storage.
Asynchronous SGD generalizes well with enough data, improving stability and reducing error.
problem Generalization performance of asynchronous distributed SGD systems.
method Algorithm stability framework, adaptive learning rate strategy.
result Distributed asynchronous SGD generalizes well with enough data samples.
A new privacy-preserving ML framework using random sampling.
problem Achieving differential privacy in client-distributed ML.
method Draw and Discard: random model sampling for scalability and privacy.
result Demonstrated improved model quality and privacy guarantees.
Zeno++ improves robustness of asynchronous SGD in fully asynchronous settings.
problem Byzantine failures in fully asynchronous SGD.
method Estimates descent of loss after applying candidate gradient.
result Proves convergence for non-convex problems under Byzantine failures.
A simple algorithm reduces federated contextual linear bandits' regret efficiently.
problem Solving federated contextual linear bandits with asynchronous agents.
method Proposed a simple algorithm exttt{FedLinUCB} based on optimism principle.
result Proved exttt{FedLinUCB} has bounded regret i l d e O ( d ∑ m = 1 M T m ) ilde{O}(d\sqrt{\sum_{m=1}^M T_m}) i l d e O ( d ∑ m = 1 M T m ) and communication complexity i l d e O ( d M 2 ) ilde{O}(dM^2) i l d e O ( d M 2 ) . Triadic-OCD detects changes in data streams robustly and optimally, even in asynchronous settings.
problem Online change detection in data streams with practical constraints.
method Triadic-OCD framework for asynchronous online change detection with provable robustness, optimality, and convergence.
result The proposed triadic-OCD algorithm achieves optimal performance and convergence in asynchronous settings.
New algorithm reduces federated learning rounds and improves privacy.
problem Inefficient synchronous federated learning causing scalability issues.
method Asynchronous federated learning with reduced communication and differential privacy via Gaussian noise.
result The algorithm reduces waiting times and network communication, making federated learning more scalable and private.
AsyB-ProxSGD parallelizes model updates and stochastic gradient descent for large models and data.
problem Efficiently training large models and handling large datasets in parallel.
method AsyB-ProxSGD: model parallel proximal stochastic gradient algorithm for asynchronous systems.
result Achieves linear speedup with O ( K 1 / 4 ) O(K^{1/4}) O ( K 1/4 ) number of workers for nonconvex problems. We study the problem of stochastic optimization for deep learning in the parallel computing environment under communication constraints. A new algorithm is proposed in this setting where the communication and coordination of work among concurrent processes (local workers), is based on an elastic force which links the p…
This paper makes two contributions to Bayesian machine learning algorithms. Firstly, we propose stochastic natural gradient expectation propagation (SNEP), a novel alternative to expectation propagation (EP), a popular variational inference algorithm. SNEP is a black box variational algorithm, in that it does not requi…
Unified analysis of asynchronous-SGD algorithms for distributed learning.
problem Analyzing asynchronous-SGD in heterogeneous settings with varying speeds and data distributions.
method Unified convergence theory for non-convex smooth functions, including pure asynchronous SGD and its modifications.
result Unified convergence rates for various asynchronous algorithms, including novel methods.
DASA speeds up SA with delayed agents, achieving N-fold speedup.
problem Speeding up Stochastic Approximation with asynchronous delays.
method DASA: Delay-Adaptive Multi-Agent Stochastic Approximation algorithm.
result First algorithm with convergence rate dependent on mixing time and average delay.
Lapse improves parameter servers by dynamically allocating parameters, achieving near-linear scaling.
problem Efficiently managing distributed training with reduced communication overhead.
method Integrate dynamic parameter allocation into parameter servers, proposing Lapse.
result Lapse provides near-linear scaling and can be orders of magnitude faster than existing parameter servers.
Proposes DC-S3GD for efficient large-scale decentralized neural network training.
problem Training large-scale decentralized neural networks efficiently.
method Decentralized stale-synchronous version of DC-ASGD with gradient correction.
result Achieves state-of-the-art results in training Convolutional Neural Networks.
DSSP improves deep learning training speed by dynamically adjusting staleness thresholds.
problem Time-consuming deep learning training on large datasets.
method Dynamic Stale Synchronous Parallel (DSSP) framework that adapts staleness threshold at runtime.
result DSSP converges faster and achieves higher accuracy than other paradigms.
Paper proposes double quantization to reduce communication in distributed machine learning.
problem High communication overhead in synchronizing stochastic gradients and model parameters in distributed training.
method Proposes double quantization for model parameters and gradients, and three communication-efficient algorithms.
result Established performance guarantees and demonstrated effective bit reduction without performance degradation.
Asynchronous framework improves distributed learning performance.
problem Heterogeneous computing machines hinder synchronous learning strategies.
method Asynchronous distributed framework with parameter exchanges.
result Convergence of consistency in distributed asynchronous methods for gradient iterations.
A distributed algorithm learns patterns in large images and signals.
problem High-dimensional optimization in large images and signals.
method Distributed asynchronous algorithm with locally greedy coordinate descent.
result Patterns can be learned on large scales images from the Hubble Space Telescope.
Distributed Collaborative Hashing improves recommendation efficiency in big data.
problem Efficiency in offline model training and online recommendation for collaborative filtering.
method Distributed Learning Framework + Hashing Technique.
result DCH model achieves comparable recommendation accuracy with fast convergence and real-time efficiency.
This paper presents Rudra, a parameter server based distributed computing framework tuned for training large-scale deep neural networks. Using variants of the asynchronous stochastic gradient descent algorithm we study the impact of synchronization protocol, stale gradient updates, minibatch size, learning rates, and n…
Asynchronous algorithms reduce privacy costs in distributed machine learning.
problem Privacy concerns in training machine learning models on scattered private data.
method Differentially-private asynchronous algorithms for collaborative training.
result Cost of privacy is inversely proportional to dataset size and privacy budgets.
OL4EL optimizes edge learning on resource-constrained servers.
problem Resource constraints on edge servers hinder effective distributed machine learning.
method Online Learning for EL (OL4EL) framework using budget-limited multi-armed bandit model.
result OL4EL significantly improves learning performance while conserving resources.
New robust aggregation rules for SGD in the presence of Byzantine failures.
problem Byzantine failures in distributed SGD.
method Three new aggregation rules for robust SGD under a general Byzantine failure model.
result Proven Byzantine resilience of the proposed aggregation rules.
Rescaled ASGD optimizes distributed learning under heterogeneous data.
problem Vanilla ASGD biases towards a frequency-weighted average of local objectives.
method Rescale worker stepsizes by their computation times.
result Rescaled ASGD converges to the correct global objective in fixed-computation model.
Paper proposes a GPU-based system for training massive deep learning models in ads systems.
problem Training massive deep learning models with terabyte-scale parameters in ads systems.
method Hierarchical GPU parameter server with 3-layer storage (GPU High-Bandwidth Memory, CPU main memory, SSD).
result 4-node hierarchical GPU parameter server trains a model 2X faster than a 150-node in-memory system.
Stanza separates convolutional and fully connected layers for faster deep learning training.
problem Heavy data transfer between workers and servers in distributed deep learning.
method Layer separation: most nodes train convolutional layers, others train fully connected layers only.
result Significant acceleration of training time (1.34x--13.9x) over current systems.
DANA mitigates gradient staleness in asynchronous distributed SGD with momentum.
problem Gradient staleness in asynchronous distributed SGD with momentum.
method DANA: a novel technique for asynchronous distributed SGD with momentum that computes the gradient on an estimated future position of the model's parameters.
result DANA fully incorporates momentum in asynchronous training with almost no ramifications to final accuracy.
Paper tackles fault tolerance in distributed linear regression.
problem Fault tolerance in distributed linear regression with Byzantine faulty agents.
method Robustified distributed gradient descent using norm-based filters.
result Server can determine linear relationship deterministically in a log-linear computation cost.
This work provides bounds on generalization error and privacy leakage in federated learning.
problem Bounding generalization error and privacy leakage in federated learning.
method Information-theoretic framework for classical, distributed, and federated learning.
result Upper and lower bounds on generalization error and privacy leakage.
LiuBei is a resilient ML algorithm that tolerates Byzantine workers and servers without trusting any component.
problem Byzantine failures in distributed ML solutions.
method Byzantine-resilient ML algorithm that aggregates gradients and replicates parameter servers, using a filtering mechanism and scatter/gather protocol.
result LiuBei achieves Byzantine resilience to both servers and workers and guarantees convergence, with an accuracy loss of around 5% and a 24% convergence overhead.
Study improves distributed linear estimation under adversarial conditions.
problem Mean estimation of a random vector with adversarial measurements and asynchrony.
method Two-timescale ℓ1-minimization algorithm with tight convergence rates.
result Unified finite-time characterization of robustness, identifiability, and statistical efficiency.
DFedAvgM is a decentralized FedAvg with momentum for privacy and communication efficiency.
problem Efficiently train models with privacy and communication efficiency in federated learning.
method Decentralized Federated Averaging with Momentum (DFedAvgM) on clients connected by an undirected graph, using stochastic gradient descent with momentum and quantization.
result DFedAvgM converges under trivial assumptions and can be improved with the PŁ property, numerically verified.