A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Paper proposes robust gossip algorithms for mean and trimmed mean estimation.
New gossip algorithms improve robustness of rank-based statistics in decentralized systems.
AsylADMM improves gossip-based learning for non-smooth objectives.
Continuized Nesterov acceleration accelerates stochastic gradient descent and gossip algorithms.
Distributing Neural Network training is of particular interest for several reasons including scaling using computing clusters, training at data sources such as IOT devices and edge servers, utilizing underutilized resources across heterogeneous environments, and so on. Most contemporary approaches primarily address sca…
We consider decentralized stochastic optimization with the objective function (e.g. data samples for machine learning task) being distributed over machines that can only communicate to their neighbors on a fixed communication graph. To reduce the communication bottleneck, the nodes compress (e.g. quantize or sparsi…
Floating Gossip improves continuous machine learning in a decentralized manner.
Multiple gossip steps improve decentralized optimization convergence.
Decentralized ranking consensus via gossip for robust and scalable systems.
This paper proposes a gossip-based algorithm for distributed bilevel optimization over networks.
Unified framework for Byzantine robust gossip algorithms with guaranteed performance.
Consider a network of agents connected by communication links, where each agent holds a real value. The gossip problem consists in estimating the average of the values diffused in the network in a distributed manner. We develop a method solving the gossip problem that depends only on the spectral dimension of the netwo…
The paper gives bounds for how long it takes for gossip protocols to spread information in networks.
In the era of big data, an important weapon in a machine learning researcher's arsenal is a scalable Support Vector Machine (SVM) algorithm. SVMs are extensively used for solving classification problems. Traditional algorithms for learning SVMs often scale super linearly with training set size which becomes infeasible …
We address the issue of speeding up the training of convolutional neural networks by studying a distributed method adapted to stochastic gradient descent. Our parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way of sharing information bet…
We address the issue of speeding up the training of convolutional networks. Here we study a distributed method adapted to stochastic gradient descent (SGD). The parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way to share information bet…
We consider a decentralized multi-agent Multi Armed Bandit (MAB) setup consisting of agents, solving the same MAB instance to minimize individual cumulative regret. In our model, agents collaborate by exchanging messages through pairwise gossip style communications on an arbitrary connected graph. We develop two no…
Paper tightens lower bounds on decentralized training complexity.
Efficient and robust algorithms for decentralized estimation in networks are essential to many distributed systems. Whereas distributed estimation of sample mean statistics has been the subject of a good deal of attention, computation of -statistics, relying on more expensive averaging over pairs of observations, is…
In decentralized networks (of sensors, connected objects, etc.), there is an important need for efficient algorithms to optimize a global cost function, for instance to learn a global model from the local data collected by each computing unit. In this paper, we address the problem of decentralized minimization of pairw…
Novel algorithm for decentralized optimization in time-varying networks with delays.
Multi-simulator training has contributed to the recent success of Deep Reinforcement Learning by stabilizing learning and allowing for higher training throughputs. We propose Gossip-based Actor-Learner Architectures (GALA) where several actor-learners (such as A2C agents) are organized in a peer-to-peer communication t…
In this paper, we determine the optimal convergence rates for strongly convex and smooth distributed optimization in two settings: centralized and decentralized communications over a network. For centralized (i.e. master/slave) algorithms, we show that distributing Nesterov's accelerated gradient descent is optimal and…
GOAT learns multiple node representations from graph structure alone.
This paper examines ADMM for network averaging, revealing its convergence rates and network topology impacts.
Stochastic gradient descent is a simple approach to find the local minima of a cost function whose evaluations are corrupted by noise. In this paper, we develop a procedure extending stochastic gradient descent algorithms to the case where the function is defined on a Riemannian manifold. We prove that, as in the Eucli…
A new approach corrects bias in federated learning due to varying communication links.
Paper develops Byzantine-resilient algorithms for decentralized learning.
The emerging concern about data privacy and security has motivated the proposal of federated learning, which allows nodes to only synchronize the locally-trained models instead their own original data. Conventional federated learning architecture, inherited from the parameter server design, relies on highly centralized…
Distributed data-parallel algorithms aim to accelerate the training of deep neural networks by parallelizing the computation of large mini-batch gradient updates across multiple nodes. Approaches that synchronize nodes using exact distributed averaging (e.g., via AllReduce) are sensitive to stragglers and communication…
We consider a set of learning agents in a collaborative peer-to-peer network, where each agent learns a personalized model according to its own learning objective. The question addressed in this paper is: how can agents improve upon their locally trained model by communicating with other agents that have similar object…
RelaySum improves decentralized deep learning by uniformly distributing data across workers.
This paper develops efficient federated learning and unlearning methods in Bayesian models.
DESTRESS optimizes decentralized nonconvex optimization with optimal IFO complexity and efficient communication.
New asynchronous algorithms improve speed in decentralized optimization networks.
Distributed optimization is essential for training large models on large datasets. Multiple approaches have been proposed to reduce the communication overhead in distributed training, such as synchronizing only after performing multiple local SGD steps, and decentralized methods (e.g., using gossip algorithms) to decou…
In this paper, we study the problem of distributed multi-agent optimization over a network, where each agent possesses a local cost function that is smooth and strongly convex. The global objective is to find a common solution that minimizes the average of all cost functions. Assuming agents only have access to unbiase…
Bayesian algorithms minimize cumulative regret in decentralized multi-agent bandits.
A novel fully asynchronous scheme for distributed reinforcement learning over networks.
We propose and analyze a new stochastic gradient method, which we call Stochastic Unbiased Curvature-aided Gradient (SUCAG), for finite sum optimization problems. SUCAG constitutes an unbiased total gradient tracking technique that uses Hessian information to accelerate con- vergence. We analyze our method under the ge…
Unified analysis for decentralized SGD across various topologies and updates.
In this paper we present an optimization-based view of distributed parameter estimation and observational social learning in networks. Agents receive a sequence of random, independent and identically distributed (i.i.d.) signals, each of which individually may not be informative about the underlying true state, but the…
Stochastic gradient descent achieves polynomial convergence rates for noiseless linear models.
Decentralized optimization is emerging as a viable alternative for scalable distributed machine learning, but also introduces new challenges in terms of synchronization costs. To this end, several communication-reduction techniques, such as non-blocking communication, quantization, and local steps, have been explored i…
In this paper, we introduce a distributed version of the classical stochastic Multi-Arm Bandit (MAB) problem. Our setting consists of a large number of agents that collaboratively and simultaneously solve the same instance of armed MAB to minimize the average cumulative regret over all agents. The agents can co…
New algorithm for decentralized online convex optimization with unknown delays.
We analyse the learning performance of Distributed Gradient Descent in the context of multi-agent decentralised non-parametric regression with the square loss function when i.i.d. samples are assigned to agents. We show that if agents hold sufficiently many samples with respect to the network size, then Distributed Gra…
Improves decentralized learning by teleporting active nodes for better convergence.