A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
The paper gives bounds for how long it takes for gossip protocols to spread information in networks.
AsylADMM improves gossip-based learning for non-smooth objectives.
New gossip algorithms improve robustness of rank-based statistics in decentralized systems.
Multi-simulator training has contributed to the recent success of Deep Reinforcement Learning by stabilizing learning and allowing for higher training throughputs. We propose Gossip-based Actor-Learner Architectures (GALA) where several actor-learners (such as A2C agents) are organized in a peer-to-peer communication t…
Asynchronous decentralized SGD with quantized and local updates converges in gossip model.
We address the issue of speeding up the training of convolutional neural networks by studying a distributed method adapted to stochastic gradient descent. Our parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way of sharing information bet…
We address the issue of speeding up the training of convolutional networks. Here we study a distributed method adapted to stochastic gradient descent (SGD). The parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way to share information bet…
Continuized Nesterov acceleration accelerates stochastic gradient descent and gossip algorithms.
Efficient and robust algorithms for decentralized estimation in networks are essential to many distributed systems. Whereas distributed estimation of sample mean statistics has been the subject of a good deal of attention, computation of -statistics, relying on more expensive averaging over pairs of observations, is…
A novel fully asynchronous scheme for distributed reinforcement learning over networks.
New asynchronous algorithms improve speed in decentralized optimization networks.
In decentralized networks (of sensors, connected objects, etc.), there is an important need for efficient algorithms to optimize a global cost function, for instance to learn a global model from the local data collected by each computing unit. In this paper, we address the problem of decentralized minimization of pairw…
Distributing Neural Network training is of particular interest for several reasons including scaling using computing clusters, training at data sources such as IOT devices and edge servers, utilizing underutilized resources across heterogeneous environments, and so on. Most contemporary approaches primarily address sca…
Floating Gossip improves continuous machine learning in a decentralized manner.
We propose and analyze a new stochastic gradient method, which we call Stochastic Unbiased Curvature-aided Gradient (SUCAG), for finite sum optimization problems. SUCAG constitutes an unbiased total gradient tracking technique that uses Hessian information to accelerate con- vergence. We analyze our method under the ge…
Paper proposes robust gossip algorithms for mean and trimmed mean estimation.
A novel algorithm reduces regret in multi-agent multi-armed bandits through gossip-based communication.
We consider a set of learning agents in a collaborative peer-to-peer network, where each agent learns a personalized model according to its own learning objective. The question addressed in this paper is: how can agents improve upon their locally trained model by communicating with other agents that have similar object…
Multiple gossip steps improve decentralized optimization convergence.
Decentralized ranking consensus via gossip for robust and scalable systems.
We consider decentralized stochastic optimization with the objective function (e.g. data samples for machine learning task) being distributed over machines that can only communicate to their neighbors on a fixed communication graph. To reduce the communication bottleneck, the nodes compress (e.g. quantize or sparsi…
Algorithm reduces regret in multi-agent bandits through gossiping.
Consider a network of agents connected by communication links, where each agent holds a real value. The gossip problem consists in estimating the average of the values diffused in the network in a distributed manner. We develop a method solving the gossip problem that depends only on the spectral dimension of the netwo…
This paper proposes a gossip-based algorithm for distributed bilevel optimization over networks.
Unified framework for Byzantine robust gossip algorithms with guaranteed performance.
In the era of big data, an important weapon in a machine learning researcher's arsenal is a scalable Support Vector Machine (SVM) algorithm. SVMs are extensively used for solving classification problems. Traditional algorithms for learning SVMs often scale super linearly with training set size which becomes infeasible …
Paper tightens lower bounds on decentralized training complexity.
Ringmaster ASGD improves Asynchronous SGD's efficiency under varying worker times.
AEGiS optimizes expensive function evaluations asynchronously.
Secure aggregation for buffered asynchronous federated learning without TEEs.
The emerging concern about data privacy and security has motivated the proposal of federated learning, which allows nodes to only synchronize the locally-trained models instead their own original data. Conventional federated learning architecture, inherited from the parameter server design, relies on highly centralized…
Standard acquisition functions are sufficient for asynchronous Bayesian optimization.
Paper analyzes convergence and speedup of asynchronous parallel SGD.
Batch Bayesian optimisation (BO) has been successfully applied to hyperparameter tuning using parallel computing, but it is wasteful of resources: workers that complete jobs ahead of others are left idle. We address this problem by developing an approach, Penalising Locally for Asynchronous Bayesian Optimisation on …
Asynchronous method for hyperparameter and neural architecture search.
This work tackles resource allocation in asynchronous and stochastic systems.
Advances in asynchronous optimization methods for machine learning.
New Fourier transform method handles missing data and asynchronous observations.
GOAT learns multiple node representations from graph structure alone.
As datasets continue to increase in size and multi-core computer architectures are developed, asynchronous parallel optimization algorithms become more and more essential to the field of Machine Learning. Unfortunately, conducting the theoretical analysis asynchronous methods is difficult, notably due to the introducti…
FedBuff improves federated learning scalability with asynchronous updates.
Sparsification improves convergence in asynchronous distributed SGD, even in the presence of staleness.
The asymptotic pseudo-trajectory approach to stochastic approximation of Benaim, Hofbauer and Sorin is extended for asynchronous stochastic approximations with a set-valued mean field. The asynchronicity of the process is incorporated into the mean field to produce convergence results which remain similar to those of a…
We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures. Roughly, the n…
Distributed asynchronous SGD has become widely used for deep learning in large-scale systems, but remains notorious for its instability when increasing the number of workers. In this work, we study the dynamics of distributed asynchronous SGD under the lens of Lagrangian mechanics. Using this description, we introduce …
This paper describes Plumbing for Optimization with Asynchronous Parallelism (POAP) and the Python Surrogate Optimization Toolbox (pySOT). POAP is an event-driven framework for building and combining asynchronous optimization strategies, designed for global optimization of expensive functions where concurrent function …
Unified analysis of asynchronous-SGD algorithms for distributed learning.
Paper proves CLTs for Q-learning with asynchronous updates.