arXiv research

New algorithm for multi-player bandits in decentralized, asynchronous systems.

Ringmaster ASGD improves Asynchronous SGD's efficiency under varying worker times.

Asynchronous method for hyperparameter and neural architecture search.

Standard acquisition functions are sufficient for asynchronous Bayesian optimization.

Sparsification improves convergence in asynchronous distributed SGD, even in the presence of staleness.

Advances in asynchronous optimization methods for machine learning.

Secure aggregation for buffered asynchronous federated learning without TEEs.

Although distributed computing can significantly reduce the training time of deep neural networks, scaling the training process while maintaining high efficiency and final accuracy is challenging. Distributed asynchronous training enjoys near-linear speedup, but asynchrony causes gradient staleness - the main difficult…

PARyOpt is a python based implementation of the Bayesian optimization routine designed for remote and asynchronous function evaluations. Bayesian optimization is especially attractive for computational optimization due to its low cost function footprint as well as the ability to account for uncertainties in data. A key…

New method speeds up nuclear-norm constrained learning over multiple machines.

New algorithm reduces regret in asynchronous multiplayer bandits to constant or logarithmic levels.

New asynchronous SGD algorithms achieve optimal performance in distributed learning.

New method calibrates asynchronous, error-prone covariates for longitudinal data.

AsylADMM improves gossip-based learning for non-smooth objectives.

Asynchronous algorithms reduce privacy costs in distributed machine learning.

New algorithms solve nonconvex federated learning problems efficiently.

Asynchronous decentralized SGD with quantized and local updates converges in gossip model.

AEGiS optimizes expensive function evaluations asynchronously.

Unified framework for high-dimensional online learning with non-divergent error bounds and adaptive gains.

The analysis of the intraday dynamics of correlations among high-frequency returns is challenging due to the presence of asynchronous trading and market microstructure noise. Both effects may lead to significant data reduction and may severely underestimate correlations if traditional methods for low-frequency data are…

Paper analyzes convergence and speedup of asynchronous parallel SGD.

Batch Bayesian optimisation (BO) has been successfully applied to hyperparameter tuning using parallel computing, but it is wasteful of resources: workers that complete jobs ahead of others are left idle. We address this problem by developing an approach, Penalising Locally for Asynchronous Bayesian Optimisation on $k$…

This work tackles resource allocation in asynchronous and stochastic systems.

New Fourier transform method handles missing data and asynchronous observations.

As datasets continue to increase in size and multi-core computer architectures are developed, asynchronous parallel optimization algorithms become more and more essential to the field of Machine Learning. Unfortunately, conducting the theoretical analysis asynchronous methods is difficult, notably due to the introducti…

FedBuff improves federated learning scalability with asynchronous updates.

Federated learning obtains a central model on the server by aggregating models trained locally on clients. As a result, federated learning does not require clients to upload their data to the server, thereby preserving the data privacy of the clients. One challenge in federated learning is to reduce the client-server c…

New Q-learning method achieves optimal sample complexity for average-reward problems.

The asymptotic pseudo-trajectory approach to stochastic approximation of Benaim, Hofbauer and Sorin is extended for asynchronous stochastic approximations with a set-valued mean field. The asynchronicity of the process is incorporated into the mean field to produce convergence results which remain similar to those of a…

We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures. Roughly, the n…

Distributed asynchronous SGD has become widely used for deep learning in large-scale systems, but remains notorious for its instability when increasing the number of workers. In this work, we study the dynamics of distributed asynchronous SGD under the lens of Lagrangian mechanics. Using this description, we introduce …

This paper describes Plumbing for Optimization with Asynchronous Parallelism (POAP) and the Python Surrogate Optimization Toolbox (pySOT). POAP is an event-driven framework for building and combining asynchronous optimization strategies, designed for global optimization of expensive functions where concurrent function …

This paper presents a methodology for creating streaming, distributed inference algorithms for Bayesian nonparametric (BNP) models. In the proposed framework, processing nodes receive a sequence of data minibatches, compute a variational posterior for each, and make asynchronous streaming updates to a central model. In…

Unified analysis of asynchronous-SGD algorithms for distributed learning.

Paper proves CLTs for Q-learning with asynchronous updates.

Accelerates optimization in asynchronous systems with sparse updates.

Improves asynchronous federated learning with queuing dynamics.

We study optimization algorithms based on variance reduction for stochastic gradient descent (SGD). Remarkable recent progress has been made in this direction through development of algorithms like SAG, SVRG, SAGA. These algorithms have been shown to outperform SGD, both theoretically and empirically. However, asynchro…

This paper quantifies and mitigates a bias in the Hayashi-Yoshida estimator causing data loss.

Asynchronous framework speeds up model-based RL to real-time.

Background: Recent developments have made it possible to accelerate neural networks training significantly using large batch sizes and data parallelism. Training in an asynchronous fashion, where delay occurs, can make training even more scalable. However, asynchronous training has its pitfalls, mainly a degradation in…

Ringleader ASGD optimizes SGD for diverse edge devices with varying data and computation speeds.

Combines multi-fidelity and asynchronous batch methods for faster experimental design.

In many distributed learning problems, the heterogeneous loading of computing machines may harm the overall performance of synchronous strategies. In this paper, we propose an effective asynchronous distributed framework for the minimization of a sum of smooth functions, where each machine performs iterations in parall…

LSAM optimizes deep learning training with improved efficiency.

Distributed training of deep nets is an important technique to address some of the present day computing challenges like memory consumption and computational demands. Classical distributed approaches, synchronous or asynchronous, are based on the parameter server architecture, i.e., worker nodes compute gradients which…

Freya PAGE optimizes nonconvex optimization with heterogeneous, asynchronous workers.

The paper gives bounds for how long it takes for gossip protocols to spread information in networks.