arXiv research

A scalable parallel BO method for asynchronous settings.

PHS optimizes hyperparameters in parallel for expensive computations.

We suggest a general oracle-based framework that captures different parallel stochastic optimization settings described by a dependency graph, and derive generic lower bounds in terms of this graph. We then use the framework and derive lower bounds for several specific parallel optimization settings, including delayed …

A single policy suffices for near-optimal parallel exploration in RL.

Proposes a new PBO method with theoretical guarantees.

In many applications of black-box optimization, one can evaluate multiple points simultaneously, e.g. when evaluating the performances of several different neural network architectures in a parallel computing environment. In this paper, we develop a novel batch Bayesian optimization algorithm --- the parallel knowledge…

OptEx accelerates first-order optimization with parallelized iterations.

Efficiently optimizes orthogonal and Stiefel matrices on parallel units.

TensorOpt finds optimal parallelization strategies for DNN training.

Enhances parallelism in decentralized learning for larger networks.

New BO methods exploit parallel experiments, reducing search time and improving solution quality.

Optimized parallel algorithms for identifying strong ties in data.

Parallel Bayesian optimization tackles noisy multi-objective problems.

Parallel BO method for multi-objective optimization with constraints.

Predictability enables efficient parallelization of nonlinear models.

Study the effects of data parallelism and sparsity on neural network training.

Can one parallelize complex exploration exploitation tradeoffs? As an example, consider the problem of optimal high-throughput experimental design, where we wish to sequentially design batches of experiments in order to simultaneously learn a surrogate function mapping stimulus to response and identify the maximum of t…

A scalable portfolio approach speeds up Bayesian optimization for noisy functions.

In this paper we study the limitations of parallelization in convex optimization. A convenient approach to study parallelization is through the prism of \emph{adaptivity} which is an information theoretic measure of the parallel runtime of an algorithm [BS18]. Informally, adaptivity is the number of sequential rounds a…

Many machine learning algorithms minimize a regularized risk, and stochastic optimization is widely used for this task. When working with massive data, it is desirable to perform stochastic optimization in parallel. Unfortunately, many existing stochastic optimization algorithms cannot be parallelized efficiently. In t…

We study the question of whether parallelization in the exploration of the feasible set can be used to speed up convex optimization, in the local oracle model of computation. We show that the answer is negative for both deterministic and randomized algorithms applied to essentially any of the interesting geometries and…

A new algorithm for parallel transport on shape spaces is presented and compared to existing methods.

Bayesian optimization has emerged as a strong candidate tool for global optimization of functions with expensive evaluation costs. However, due to the dynamic nature of research in Bayesian approaches, and the evolution of computing technology, using Bayesian optimization in a parallel computing environment remains a c…

A new method for efficient optimization of expensive simulations on HPC.

Classical optimization algorithms in machine learning often take a long time to compute when applied to a multi-dimensional problem and require a huge amount of CPU and GPU resource. Quantum parallelism has a potential to speed up machine learning algorithms. We describe a generic mathematical model to leverage quantum…

Method predicts how probability distributions evolve over time.

This research optimizes energy consumption forecasting in Puno using parallel computing and ARIMA models.

Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed …

Clapping reduces memory usage in distributed optimization by reusing data samples.

We present Acquisition Thompson Sampling (ATS), a novel technique for batch Bayesian Optimization (BO) based on the idea of sampling multiple acquisition functions from a stochastic process. We define this process through the dependency of the acquisition functions on a set of model hyper-parameters. ATS is conceptuall…

MUSE provides unbiased stopping estimates for optimal problems.

Improves parallel deep model performance by restructuring and pruning.

Non-negative Matrix Factorization (NMF) is a key kernel for unsupervised dimension reduction used in a wide range of applications, including topic modeling, recommender systems and bioinformatics. Due to the compute-intensive nature of applications that must perform repeated NMF, several parallel implementations have b…

Randomized block-diagonal preconditioning improves parallel learning convergence.

Parallel surrogate optimization algorithms have proven to be efficient methods for solving expensive noisy optimization problems. In this work we develop a new parallel surrogate optimization algorithm (ProSRS), using a novel tree-based "zoom strategy" to improve the efficiency of the algorithm. We prove that if ProSRS…

Adaptive batch size schedules improve language model training efficiency and generalization.

We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures. Roughly, the n…

TSSM splits neural networks for parallel training with minimal accuracy loss.

We develop parallel predictive entropy search (PPES), a novel algorithm for Bayesian optimization of expensive black-box objective functions. At each iteration, PPES aims to select a batch of points which will maximize the information gain about the global maximizer of the objective. Well known strategies exist for sug…

A new parallel BO method with exact gradients for multi-objective optimization.

Safe RL for autonomous vehicles using PCPO with trust regions and parallel learners.

We propose HAMSI (Hessian Approximated Multiple Subsets Iteration), which is a provably convergent, second order incremental algorithm for solving large-scale partially separable optimization problems. The algorithm is based on a local quadratic approximation, and hence, allows incorporating curvature information to sp…

As datasets continue to increase in size and multi-core computer architectures are developed, asynchronous parallel optimization algorithms become more and more essential to the field of Machine Learning. Unfortunately, conducting the theoretical analysis asynchronous methods is difficult, notably due to the introducti…

Monte Carlo (MC) methods are widely used for Bayesian inference and optimization in statistics, signal processing and machine learning. A well-known class of MC methods are Markov Chain Monte Carlo (MCMC) algorithms. In order to foster better exploration of the state space, specially in high-dimensional applications, s…

SOBER optimizes and quadrates efficiently in parallel for diverse tasks.

Recurrent neural networks (RNN) have been successfully applied to various sequential decision-making tasks, natural language processing applications, and time-series predictions. Such networks are usually trained through back-propagation through time (BPTT) which is prohibitively expensive, especially when the length o…

In this paper we analyze, evaluate, and improve the performance of training generalized linear models on modern CPUs. We start with a state-of-the-art asynchronous parallel training algorithm, identify system-level performance bottlenecks, and apply optimizations that improve data parallelism, cache line locality, and …

In [dLMu05], DeLellis and Müller proved a quantitative version of Codazzi's theorem, namely for a smooth embedded surface $\ Σ\subseteq \mathbb{R}^3\$ with area normalized to $\ {\cal H}^2(Σ) = 4 π$, it was shown that $\ \parallel A_Σ- id \parallel_{L^2(Σ)} \leq C \parallel A^0_Σ\parallel_{L^2(Σ)}\$, and building on…