A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Ringmaster ASGD improves Asynchronous SGD's efficiency under varying worker times.
Advances in asynchronous optimization methods for machine learning.
Standard acquisition functions are sufficient for asynchronous Bayesian optimization.
We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures. Roughly, the n…
This paper describes Plumbing for Optimization with Asynchronous Parallelism (POAP) and the Python Surrogate Optimization Toolbox (pySOT). POAP is an event-driven framework for building and combining asynchronous optimization strategies, designed for global optimization of expensive functions where concurrent function …
Accelerates optimization in asynchronous systems with sparse updates.
Asynchronous method for hyperparameter and neural architecture search.
Ringleader ASGD optimizes SGD for diverse edge devices with varying data and computation speeds.
Combines multi-fidelity and asynchronous batch methods for faster experimental design.
As datasets continue to increase in size and multi-core computer architectures are developed, asynchronous parallel optimization algorithms become more and more essential to the field of Machine Learning. Unfortunately, conducting the theoretical analysis asynchronous methods is difficult, notably due to the introducti…
PARyOpt is a python based implementation of the Bayesian optimization routine designed for remote and asynchronous function evaluations. Bayesian optimization is especially attractive for computational optimization due to its low cost function footprint as well as the ability to account for uncertainties in data. A key…
Freya PAGE optimizes nonconvex optimization with heterogeneous, asynchronous workers.
LSAM optimizes deep learning training with improved efficiency.
New asynchronous SGD algorithms achieve optimal performance in distributed learning.
Distributed asynchronous SGD has become widely used for deep learning in large-scale systems, but remains notorious for its instability when increasing the number of workers. In this work, we study the dynamics of distributed asynchronous SGD under the lens of Lagrangian mechanics. Using this description, we introduce …
AEGiS optimizes expensive function evaluations asynchronously.
Triadic-OCD detects changes in data streams robustly and optimally, even in asynchronous settings.
New asynchronous algorithms improve speed in decentralized optimization networks.
We introduce and analyze stochastic optimization methods where the input to each gradient update is perturbed by bounded noise. We show that this framework forms the basis of a unified approach to analyze asynchronous implementations of stochastic optimization algorithms.In this framework, asynchronous stochastic optim…
A scalable parallel BO method for asynchronous settings.
AB dynamically scales gradients to mitigate asynchronous training delays.
New algorithm reduces regret in asynchronous multiplayer bandits to constant or logarithmic levels.
This work tackles resource allocation in asynchronous and stochastic systems.
We study optimization algorithms based on variance reduction for stochastic gradient descent (SGD). Remarkable recent progress has been made in this direction through development of algorithms like SAG, SVRG, SAGA. These algorithms have been shown to outperform SGD, both theoretically and empirically. However, asynchro…
A new method for efficient optimization of expensive simulations on HPC.
Ringmaster LMO accelerates training in distributed systems by asynchronously updating neural networks.
A new estimator for asynchronous tick data shows better correlation estimates.
In machine learning, asynchronous parallel stochastic gradient descent (APSGD) is broadly used to speed up the training process through multi-workers. Meanwhile, the time delay of stale gradients in asynchronous algorithms is generally proportional to the total number of workers, which brings additional deviation from …
We study the volatility functional inference by Fourier transforms. This spectral framework is advantageous in that it harnesses the power of harmonic analysis to handle missing data and asynchronous observations without any artificial time alignment nor data imputation. Under conditions, this spectral approach is cons…
Asynchronous federated modeling improves spatial data sharing without centralizing raw data.
New algorithm reduces distributed optimization time with stochastic delays.
Asynchronous parallel implementations of stochastic gradient (SG) have been broadly used in solving deep neural network and received many successes in practice recently. However, existing theories cannot explain their convergence and speedup properties, mainly due to the nonconvexity of most deep learning formulations …
Large scale machine learning is increasingly relying on distributed optimization, whereby several machines contribute to the training process of a statistical model. In this work we study the performance of asynchronous, distributed settings, when applying sparsification, a technique used to reduce communication overhe…
Asynchronous cooperative learning rules ensure all agents converge to correct hypothesis.
Enhances SGLD for log-concave posteriors with asynchronous computation.
We describe ASAGA, an asynchronous parallel version of the incremental gradient algorithm SAGA that enjoys fast linear convergence rates. Through a novel perspective, we revisit and clarify a subtle but important technical issue present in a large fraction of the recent convergence rate proofs for asynchronous parallel…
Machine learning with big data often involves large optimization models. For distributed optimization over a cluster of machines, frequent communication and synchronization of all model parameters (optimization variables) can be very costly. A promising solution is to use parameter servers to store different subsets of…
This paper considers the multi-armed thresholding bandit problem -- identifying all arms whose expected rewards are above a predefined threshold via as few pulls (or rounds) as possible -- proposed by Locatelli et al. [2016] recently. Although the proposed algorithm in Locatelli et al. [2016] achieves the optimal round…
A simple algorithm reduces federated contextual linear bandits' regret efficiently.
We propose Zeno++, a new robust asynchronous Stochastic Gradient Descent~(SGD) procedure which tolerates Byzantine failures of the workers. In contrast to previous work, Zeno++ removes some unrealistic restrictions on worker-server communications, allowing for fully asynchronous updates from anonymous workers, arbitrar…
Unified proof for scalable personalized federated learning.
We present CYCLADES, a general framework for parallelizing stochastic optimization algorithms in a shared memory setting. CYCLADES is asynchronous during shared model updates, and requires no memory locking mechanisms, similar to HOGWILD!-type algorithms. Unlike HOGWILD!, CYCLADES introduces no conflicts during the par…
Most commonly used distributed machine learning systems are either synchronous or centralized asynchronous. Synchronous algorithms like AllReduce-SGD perform poorly in a heterogeneous environment, while asynchronous algorithms using a parameter server suffer from 1) communication bottleneck at parameter servers when wo…
Asynchronous parallel optimization algorithms for solving large-scale machine learning problems have drawn significant attention from academia to industry recently. This paper proposes a novel algorithm, decoupled asynchronous proximal stochastic gradient descent (DAP-SGD), to minimize an objective function that is the…
New algorithms solve nonconvex federated learning problems efficiently.
Coagent policy gradient algorithms (CPGAs) are reinforcement learning algorithms for training a class of stochastic neural networks called coagent networks. In this work, we prove that CPGAs converge to locally optimal policies. Additionally, we extend prior theory to encompass asynchronous and recurrent coagent networ…
ODC protocol improves learning in asynchronous multi-agent bandits.
We propose a stochastic modified equations (SME) for modeling the asynchronous stochastic gradient descent (ASGD) algorithms. The resulting SME of Langevin type extracts more information about the ASGD dynamics and elucidates the relationship between different types of stochastic gradient algorithms. We show the conver…