Elastic Gossip distributes neural network training using gossip-like protocols.
problem Distributing neural network training across heterogeneous environments.
method Pairwise-communication using Gossip-like protocols, building on Elastic Averaging SGD.
result Elastic Gossip performs better than Gossiping SGD in experiments, but hyper-parameter search may yield better configurations.
Paper proposes robust gossip algorithms for mean and trimmed mean estimation.
problem Vulnerability of mean-based gossip algorithms to malicious nodes.
method Developed extsc{GoRank} for rank estimation and extsc{GoTrim} for trimmed mean estimation.
result Established convergence rates for rank and trimmed mean estimation.
AsylADMM improves gossip-based learning for non-smooth objectives.
problem Efficient and robust decentralized learning on edge devices.
method Asynchronous gossip algorithm for non-smooth optimization.
result AsylADMM converges faster on non-smooth problems.
Improved gossip algorithm for networks of given dimension using Jacobi polynomial iterations.
problem Efficiently estimating the average of values in a network of agents.
method A gossip algorithm that depends only on the spectral dimension of the network, using Jacobi polynomial iterations.
result Significantly faster convergence compared to previous methods in the non-asymptotic regime.
Floating Gossip improves continuous machine learning in a decentralized manner.
problem Continuous learning in infrastructure-less environments.
method Mean field approach to analyze the impact of communication and computing in Floating Gossip.
result Floating Gossip can effectively incorporate large amounts of data into machine learning models.
GADGET SVM uses gossip-based distributed learning for scalable SVMs.
problem Scalability issues in traditional SVM algorithms for large datasets.
method Gossip-based distributed learning for the primal SVM formulation.
result Performance comparable to centralized and online SVM algorithms.
Novel gossip algorithms reduce communication in decentralized stochastic optimization with compressed updates.
problem Decentralized stochastic optimization with limited communication.
method Proposed CHOCO-SGD and CHOCO-GOSSIP algorithms for unbiased and biased compression, achieving linear convergence.
result Both algorithms outperform state-of-the-art, reducing communication by at least two orders of magnitude.
Continuized Nesterov acceleration accelerates stochastic gradient descent and gossip algorithms.
problem Improving the convergence rate of stochastic gradient descent and gossip algorithms.
method Introducing a continuized variant of Nesterov acceleration, which mixes variables continuously and takes gradient steps at random times.
result The continuized Nesterov acceleration achieves convergence rates similar to Nesterov's original acceleration but with random parameters.
The paper gives bounds for how long it takes for gossip protocols to spread information in networks.
problem Understanding the diffusion time in asynchronous gossip protocols.
method Provides non-asymptotic bounds for the number of messages needed for consensus in asynchronous gossip protocols.
result Explicit formula and approximation for the number of messages needed for consensus in different types of graphs.
GoSGD speeds up deep learning training with gossip exchange.
problem Speeding up the training of deep learning models.
method Distributed optimization using stochastic gradient descent with gossip algorithms.
result GoSGD achieves fully asynchronous and decentralized training.
New gossip algorithms improve robustness of rank-based statistics in decentralized systems.
problem Ensuring robustness in decentralized AI and edge intelligence systems, especially in the presence of corrupted or adversarial data.
method Developed asynchronous gossip algorithms for computing rank-based statistics.
result First convergence rate bound for asynchronous gossip-based rank estimation.
This paper proposes a gossip-based algorithm for distributed bilevel optimization over networks.
problem Distributed bilevel optimization over networks with limited communication.
method Gossip-based distributed bilevel learning algorithm.
result Achieves optimal sample complexities for general and strongly convex objectives.
Multiple gossip steps improve decentralized optimization convergence.
problem Efficiently optimizing large-scale machine learning models with limited communication.
method Integrates multiple gossip steps between gradient descent iterations in compressed decentralized optimization.
result Convergence to within ε of the optimal value for smooth non-convex objectives.
Gossip-based actor-learner architectures improve deep reinforcement learning efficiency and scalability.
problem Stabilizing and scaling deep reinforcement learning through multi-simulator training.
method Gossip-based communication among actor-learners in a peer-to-peer topology, reducing synchronization.
result Gossip-based architectures outperform A2C in terms of robustness and sample efficiency.
Decentralized ranking consensus via gossip for robust and scalable systems.
problem Achieving reliable and resilient consensus on collective rankings in a decentralized setting.
method Random gossip communication for decentralized computation of global rankings.
result Robust and scalable consensus on collective rankings achieved through decentralized, local interactions.
A new decentralized federated learning approach tackles network capacity challenges.
problem Efficiently utilizing network capacities between nodes in federated learning.
method Proposes a segmented gossip approach for decentralized federated learning.
result Demonstrates significant reduction in training time compared to centralized federated learning.
Algorithm reduces regret in multi-agent bandits through gossiping.
problem Minimizing cumulative regret in decentralized multi-agent bandits.
method Insert-Eliminate algorithm with gossip communication.
result Regret significantly reduced with minimal collaboration.
Unified framework for Byzantine robust gossip algorithms with guaranteed performance.
problem Vulnerability of decentralized machine learning to misbehaving devices.
method Introduces F-RG framework and CS+ robust aggregation rule for Byzantine resilience.
result CS+-RG has near-optimal breakdown tolerance and outperforms existing methods.
In decentralized networks (of sensors, connected objects, etc.), there is an important need for efficient algorithms to optimize a global cost function, for instance to learn a global model from the local data collected by each computing unit. In this paper, we address the problem of decentralized minimization of pairw…
We address the issue of speeding up the training of convolutional networks. Here we study a distributed method adapted to stochastic gradient descent (SGD). The parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way to share information bet…
Asynchronous decentralized SGD with quantized and local updates converges in gossip model.
problem Scalable distributed machine learning challenges in decentralized optimization.
method Asynchronous decentralized SGD with quantization and local steps in gossip model.
result Decentralized optimization with quantization and local steps converges in asynchronous gossip model.
Paper tightens lower bounds on decentralized training complexity.
problem Understanding and optimizing iteration complexity in decentralized training.
method Proved a tight lower bound on iteration complexity and proposed DeTAG algorithm.
result DeTAG achieves the theoretical lower bound with only a logarithmic gap.
In this paper, we determine the optimal convergence rates for strongly convex and smooth distributed optimization in two settings: centralized and decentralized communications over a network. For centralized (i.e. master/slave) algorithms, we show that distributing Nesterov's accelerated gradient descent is optimal and…
A novel algorithm reduces regret in multi-agent multi-armed bandits through gossip-based communication.
problem Minimizing average cumulative regret in a large number of agents solving the same MAB problem.
method A gossip-based protocol for decentralized decision-making and communication among agents.
result Achieves significant reduction in per-agent regret and communication complexity.
Novel algorithm for decentralized optimization in time-varying networks with delays.
problem Decentralized optimization in networks with communication delays.
method DT-GO algorithm, applicable to general directed graphs, converges to same complexity as centralized SGD.
result Algorithm DT-GO achieves convergence rates for convex and non-convex objectives, similar to centralized SGD.
This paper examines ADMM for network averaging, revealing its convergence rates and network topology impacts.
problem Efficiently averaging local information over a network using ADMM.
method Comparative analysis of ADMM and other algorithms on a canonical distributed averaging problem.
result Characterization of ADMM convergence and optimal parameter tuning based on network spectral properties.
GOAT learns multiple node representations from graph structure alone.
problem Context-free graph representation learning limits model performance.
method Inspired by gossip and mutual attention, GOAT learns multiple node representations.
result GOAT outperforms 12 SOTA baselines on link prediction and clustering tasks.
Efficient and robust algorithms for decentralized estimation in networks are essential to many distributed systems. Whereas distributed estimation of sample mean statistics has been the subject of a good deal of attention, computation of U-statistics, relying on more expensive averaging over pairs of observations, is…
A new approach corrects bias in federated learning due to varying communication links.
problem Bias in federated learning due to non-uniform and time-varying communication links.
method Proposes Federated Postponed Broadcast (FedPBC) to correct bias in Federated Average (FedAvg).
result FedPBC converges to a stationary point of the global objective, overcoming bias caused by varying communication links.
RelaySum improves decentralized deep learning by uniformly distributing data across workers.
problem Handling data heterogeneity in decentralized deep learning.
method RelaySum uses spanning trees to distribute information exactly uniformly across all workers with finite delays.
result RelaySum is independent of data heterogeneity and scales to many workers, enabling highly accurate decentralized deep learning.
This paper develops efficient federated learning and unlearning methods in Bayesian models.
problem Managing epistemic uncertainty and legal right to be forgotten in decentralized networks.
method Develops federated variational inference solutions based on decentralized local free energy minimization.
result Demonstrates efficient unlearning mechanisms in federated learning and unlearning.
Stochastic gradient descent is a simple approach to find the local minima of a cost function whose evaluations are corrupted by noise. In this paper, we develop a procedure extending stochastic gradient descent algorithms to the case where the function is defined on a Riemannian manifold. We prove that, as in the Eucli…
A novel fully asynchronous scheme for distributed reinforcement learning over networks.
problem Policy evaluation in distributed reinforcement learning over networks.
method Design of a stochastic average gradient (SAG) based distributed algorithm and push-pull augmented graph approach.
result The proposed algorithm converges at a linear rate of \(\mathcal{O}(c^k)\) with \(c\in(0,1)\) and \(k\) increasing by one per node update.
Paper develops Byzantine-resilient algorithms for decentralized learning.
problem Vulnerability of distributed learning to Byzantine attacks.
method Dual approach for decentralized optimization.
result Convergence guarantees and experimental validation of the proposed algorithm.
Unified analysis for decentralized SGD across various topologies and updates.
problem Analysis of decentralized SGD methods with changing topologies and local updates.
method Unified convergence analysis covering local SGD updates and adaptive network topology.
result Universal convergence rates for smooth problems, interpolating between heterogeneous and iid-data settings.
SUCAG accelerates convergence in distributed optimization with unbiased curvature information.
problem Efficiently solving finite sum optimization problems in distributed settings.
method Stochastic Unbiased Curvature-aided Gradient (SUCAG) method using Hessian information.
result Establishes linear convergence for strongly convex problems under asynchronous model.
We consider a set of learning agents in a collaborative peer-to-peer network, where each agent learns a personalized model according to its own learning objective. The question addressed in this paper is: how can agents improve upon their locally trained model by communicating with other agents that have similar object…
Paper proposes DSGT and GSGT for distributed optimization over networks.
problem Distributed optimization over networks with local cost functions.
method Distributed stochastic gradient tracking methods (DSGT) and gossip-like stochastic gradient tracking methods (GSGT).
result Iterates converge to a neighborhood of the optimal solution, with error bounds decreasing with network size.
New asynchronous algorithms improve speed in decentralized optimization networks.
problem Hard convergence analysis for asynchronous decentralized optimization.
method Continuized framework to analyze heterogeneous delays in event-driven updates.
result Achieves asynchronous speedup with convergence rate controlled by eigengap weighted by local delays.
SGP combines PushSum with stochastic gradient updates for robust distributed deep learning.
problem Synchronization issues in distributed deep learning.
method Stochastic Gradient Push (SGP) using PushSum for approximate distributed averaging.
result SGP converges to a stationary point at the same rate as SGD and achieves consensus.
Bayesian algorithms minimize cumulative regret in decentralized multi-agent bandits.
problem Minimizing cumulative regret in a decentralized multi-agent multi-armed bandit problem.
method Proposed decentralized Bayesian multi-armed bandit framework, including Thompson Sampling and Bayes-UCB algorithms.
result Regret scales logarithmically with constants matching those of an optimal centralized agent.
In this paper we present an optimization-based view of distributed parameter estimation and observational social learning in networks. Agents receive a sequence of random, independent and identically distributed (i.i.d.) signals, each of which individually may not be informative about the underlying true state, but the…
Distributed Gradient Descent achieves optimal rates in non-parametric regression with linear speed-up.
problem Optimal statistical rates in decentralized non-parametric regression.
method Distributed Gradient Descent with i.i.d. samples and linear speed-up.
result Achieves optimal statistical rates with linear speed-up in the big data regime.
Improves decentralized learning by teleporting active nodes for better convergence.
problem Decentralized learning's convergence rate degrades with large node numbers.
method Activates a subset of nodes, fetches parameters from previous active nodes, updates, and performs gossip averaging on a small topology.
result Teleportation completely alleviates convergence rate degradation with proper node activation.
DESTRESS optimizes decentralized nonconvex optimization with optimal IFO complexity and efficient communication.
problem Decentralized nonconvex finite-sum optimization in multi-agent systems.
method DESTRESS uses stochastic recursive gradient updates, gradient tracking, and careful hyper-parameter choices to achieve optimal IFO complexity with efficient communication.
result DESTRESS matches the optimal IFO complexity of centralized algorithms while maintaining communication efficiency.
SlowMo improves distributed SGD by periodically synchronizing and using momentum, leading to better accuracy.
problem Reducing communication overhead in distributed training while maintaining model accuracy.
method Periodically synchronizing workers and using momentum after multiple iterations of a base optimization algorithm.
result SlowMo consistently yields improvements in optimization and generalization performance relative to the base optimizer.
Stochastic gradient descent achieves polynomial convergence rates for noiseless linear models.
problem Convergence analysis of stochastic gradient descent in noiseless linear models.
method Fixed step-size stochastic gradient descent on least-square risk.
result Polynomial convergence rates depend on the regularities of the optimum and feature vectors.
New algorithm for decentralized online convex optimization with unknown delays.
problem Decentralized online convex optimization with unknown, time-varying feedback delays.
method Proposes a novel algorithm that incorporates adaptive learning rates and gossip-based delay estimation.
result Achieves improved regret bounds of O(N √d tot + N √T (1-σ^2)^(1/4)) and O(N δmax ln T α) for different settings.