Parallelizes MCTS for continuous domains using leaf and root parallelization.
problem Solving challenging tasks in continuous domains using MCTS.
method Extends existing parallelization strategies to continuous domains, focusing on leaf and root parallelization.
result Proposes two final selection strategies for continuous states in root parallelization.
Chemical space is so large that brute force searches for new interesting molecules are infeasible. High-throughput virtual screening via computer cluster simulations can speed up the discovery process by collecting very large amounts of data in parallel, e.g., up to hundreds or thousands of parallel measurements. Bayes…
PHS optimizes hyperparameters in parallel for expensive computations.
problem Optimizing hyperparameters in computationally expensive tasks.
method Bayesian optimization for parallel hyperparameter search.
result Efficient hyperparameter optimization on multiple instances.
Optimal transport kernels improve neural architecture search efficiency.
problem Comparing complex neural architectures similarity using Euclidean metric fails.
method Developed a novel discrepancy using tree-Wasserstein (TW) for neural architectures.
result TW-based approaches outperform other methods in sequential and parallel NAS.
We introduce a new sampling method for large language models that balances diversity and parallelism.
problem Balancing diversity and parallelism in decoding for large language models.
method Arithmetic sampling framework compatible with various sampling variations.
result Improves estimation of expected BLEU score reward and reduces the gap with beam search.
This paper studies effective parallelization of MCTS for computer games.
problem Computational expense of MCTS due to many rollouts.
method Examined parallel MCTS performance, identified necessary conditions for desirable parallelization, and proposed BU-UCT algorithm.
result Theoretical foundation for effective parallel MCTS, with BU-UCT outperforming baselines.
Enhances RL performance with a population-guided parallel learning scheme.
problem Improving off-policy reinforcement learning performance.
method Population-guided parallel learning scheme with shared experience replay buffer and soft policy update.
result Monotone improvement of the expected cumulative return proved theoretically and demonstrated in practice.
AgEBO-Tabular combines NAS and hyperparameter tuning for fast, high-performing tabular models.
problem Developing high-performing predictive models for large tabular data sets is challenging.
method Combines aging evolution NAS and asynchronous Bayesian optimization for hyperparameter tuning in data-parallel training.
result Automatically discovered neural network models outperform state-of-the-art AutoML ensembles in inference speed by two orders of magnitude.
WU-UCT parallelizes MCTS with linear speedup and limited performance loss.
problem Challenges in parallelizing Monte Carlo Tree Search (MCTS) due to its sequential nature.
method Introduces unobserved samples to track incomplete simulations and modify UCT tree policy.
result Achieves linear speedup and only limited performance loss with increasing parallel workers.
In this paper, we propose a convergent parallel best-response algorithm with the exact line search for the nondifferentiable nonconvex sparsity-regularized rank minimization problem. On the one hand, it exhibits a faster convergence than subgradient algorithms and block coordinate descent algorithms. On the other hand,…
New reshaping method improves hyperparameter search over random search.
problem Improving hyperparameter search efficiency.
method Introducing reshaping techniques to optimize search distributions.
result Substantial gains over random search in various experiments.
This paper speeds up speech recognition by vectorizing hypotheses and speech.
problem Slowness in beam search during speech recognition.
method Vectorization of hypotheses and speech, parallelism, batch processing, shallow fusion.
result 3.7x speedup achieved by vectorizing hypotheses, 10.5x by GPU.
New BO methods exploit parallel experiments, reducing search time and improving solution quality.
problem Limitation of Bayesian optimization in exploiting parallel experiments.
method Propose new parallel BO paradigms that exploit the structure of the system to partition the design space.
result Significantly reduce search time and increase probability of finding global solutions.
Mango automates hyperparameter tuning for large-scale ML training.
problem Manual hyperparameter tuning is tedious and inefficient for large-scale machine learning.
method Parallel hyperparameter tuning with intelligent search strategies and flexible abstractions.
result Mango achieves comparable performance to Hyperopt while supporting distributed computing.
We develop parallel predictive entropy search (PPES), a novel algorithm for Bayesian optimization of expensive black-box objective functions. At each iteration, PPES aims to select a batch of points which will maximize the information gain about the global maximizer of the objective. Well known strategies exist for sug…
Parallel BO method for multi-objective optimization with constraints.
problem Optimizing multiple objectives under constraints with expensive evaluations.
method PPESMOC, a batch method for simultaneous optimization of black-box functions.
result Empirical evidence shows PPESMOC is effective for multi-objective optimization with constraints.
A scalable parallel BO method for asynchronous settings.
problem Expensive-to-evaluate problems in machine learning.
method Simple and scalable Bayesian optimization method for asynchronous parallel settings.
result Demonstrated promising performance on benchmark functions and hyperparameter optimization.
New vector quantization method reduces relevance of parallel components in database points.
problem Scaling maximum inner product search to massive databases.
method Developed anisotropic vector quantization loss functions.
result Achieves state-of-the-art results on public benchmarks.
A new method for faster expensive function evaluations using multi-fidelity Bayesian optimization.
problem Expensive function evaluations in Bayesian optimization.
method Max-value entropy search (MES) for multi-fidelity Bayesian optimization (MFBO), with parallelization.
result Significant reduction in computational complexity for multi-fidelity Bayesian optimization.
A new framework improves graph construction for semi-supervised learning.
problem Improving graph construction for better semi-supervised classification accuracy.
method Parallel hyperparameter search with adaptive resource allocation for gradient-based optimization of edge weights.
result Significantly outperforms existing graph construction schemes in accuracy and scalability.
Proposes learning default hyperparameters from empirical results.
problem Selecting optimal hyperparameters for machine learning algorithms.
method Learning a set of complementary default values from a database of prior empirical results.
result Learning default values improves performance and is more efficient than random search and Bayesian Optimization.
This paper provides a block coordinate descent algorithm to solve unconstrained optimization problems. In our algorithm, computation of function values or gradients is not required. Instead, pairwise comparison of function values is used. Our algorithm consists of two steps; one is the direction estimate step and the o…
Neural architecture search (NAS) has been proposed to automatically tune deep neural networks, but existing search algorithms, e.g., NASNet, PNAS, usually suffer from expensive computational cost. Network morphism, which keeps the functionality of a neural network while changing its neural architecture, could be helpfu…
NEST optimizes deep learning training by placing devices efficiently across networks and memory.
problem Inefficient device placement in distributed deep learning leads to high communication and memory overhead.
method NEST uses network-, compute-, and memory-aware dynamic programming to optimize device placement.
result NEST achieves up to 2.43 times higher throughput and better memory efficiency.
Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated. Examples include grid search, uniform random search, low discrepancy sequences, and other sampl…
Study uses supercomputers to improve financial predictions.
problem Improving financial predictions through better exploration of data.
method Refactored and ran algorithm on Fugaku supercomputer, exploring more rules.
result Increasing the number of explored rules improves predictive performance.
Training examples are not all equally informative. Active learning strategies leverage this observation in order to massively reduce the number of examples that need to be labeled. We leverage the same observation to build a generic strategy for parallelizing learning algorithms. This strategy is effective because the …
New deep learning method solves TSP faster and more efficiently.
problem Approximately solving the Travelling Salesman Problem on 2D Euclidean graphs.
method Uses Graph Convolutional Networks for efficient TSP graph representations and non-autoregressive beam search.
result Significantly reduces optimality gap for large problem instances.
RAAL optimizes black box function optimization with multifidelity models.
problem Time-consuming and unfeasible black box optimization.
method Resource Aware Multifidelity Active Learning (RAAL) for efficient optimization.
result RAAL optimizes black box function optimization with multifidelity models.
TensorOpt finds optimal parallelization strategies for DNN training.
problem Finding efficient parallelization strategies for DNN training.
method TensorOpt uses an algorithm (FT) to search for an optimal set of parallelization strategies considering multiple objectives.
result TensorOpt provides accurate runtime cost estimation and adapts to resource availability.
This work proposes an efficient autoregressive model for text generation.
problem The challenge of generating high-quality text with autoregressive models.
method Introduces a cascaded decoding approach using Markov transformers to achieve sub-linear parallel time generation.
result Shows competitive accuracy/speed tradeoff compared to existing methods on five machine translation datasets.
New algorithms speed up EMD computation by four orders of magnitude.
problem High computational complexity of EMD for discrete probability distributions.
method Data-parallel approximation algorithms for linear time complexity.
result Four orders of magnitude faster than existing methods.
MORBO improves multi-objective BO for high-dimensional problems.
problem Optimizing multiple objectives in high-dimensional spaces with expensive evaluations.
method Parallel local BO in multiple regions with coordinated strategy.
result Significant improvement in sample efficiency for high-dimensional problems.
Machine learning applications often require hyperparameter tuning. The hyperparameters usually drive both the efficiency of the model training process and the resulting model quality. For hyperparameter tuning, machine learning algorithms are complex black-boxes. This creates a class of challenging optimization problem…
New algorithm reconstructs sparse networks in subquadratic time.
problem Reconstructing sparse networks from limited data.
method Stochastic second neighbor search to bypass quadratic complexity.
result Subquadratic time complexity, up to O(N3/2logN). MemNet optimizes neural architectures for memory efficiency.
problem Memory constraints in mobile devices limit the use of large neural networks.
method Augment-trim learning with memory consumption ranking score.
result MemNet finds architectures with 24.17% less memory usage compared to state-of-the-art methods.
We reduce the computational cost of Neural AutoML with transfer learning. AutoML relieves human effort by automating the design of ML algorithms. Neural AutoML has become popular for the design of deep learning architectures, however, this method has a high computation cost. To address this we propose Transfer Neural A…
In this paper, we discuss software design issues related to the development of parallel computational intelligence algorithms on multi-core CPUs, using the new Java 8 functional programming features. In particular, we focus on probabilistic graphical models (PGMs) and present the parallelisation of a collection of algo…
New algorithms allow multiple robots to search efficiently without central coordination.
problem Efficiently locating targets in an unknown environment with multiple robots.
method SPATS and LATSI algorithms for asynchronous multi-agent active search.
result Multiple robots can independently make data-collection decisions without a central coordinator.
MACH reduces memory usage for extreme classification by hashing.
problem Expensive training of deep models with large softmax layers.
method Merged-Average Classifiers via Hashing (MACH) using count-min sketch.
result Significant memory reduction and training speedup.
This paper reviews hyperparameter optimization methods and best practices.
problem Finding optimal hyperparameters for machine learning models.
method Various hyperparameter optimization methods are reviewed, including grid search, random search, evolutionary algorithms, Bayesian optimization, Hyperband, and racing.
result Practical recommendations for conducting hyperparameter optimization are provided.
Paper proposes CNN-LSTM model for multi-speaker speech separation.
problem Multi-speaker source separation using deep learning.
method Parallel CNN-LSTM architecture with Bayesian hyperparameter optimization.
result Parallel CNN-LSTM model outperforms LSTM-only and CNN-only models.
A new algorithm speeds up solving complex spin system problems.
problem Finding the lowest energy state in Ising models, which is hard and important.
method Mean-field Annealing from a Random State (MARS) algorithm, combining SA and MFA.
result MARS solves large Ising spin systems and maximum cut problems efficiently.
Deep neuroevolution, that is evolutionary policy search methods based on deep neural networks, have recently emerged as a competitor to deep reinforcement learning algorithms due to their better parallelization capabilities. However, these methods still suffer from a far worse sample efficiency. In this paper we invest…
rpForests uses random projections to speed up kNN search.
problem Efficient kNN search in high-dimensional data.
method Random projection forests aggregating results from multiple random projection trees.
result rpForests achieves fast decay in missing rate and kNN distance discrepancy.
The classification of MRI images according to the anatomical field of view is a necessary task to solve when faced with the increasing quantity of medical images. In parallel, advances in deep learning makes it a suitable tool for computer vision problems. Using a common architecture (such as AlexNet) provides quite go…
Bayesian optimization is a sample-efficient approach to solving global optimization problems. Along with a surrogate model, this approach relies on theoretically motivated value heuristics (acquisition functions) to guide the search process. Maximizing acquisition functions yields the best performance; unfortunately, t…
This work proposes splitting deep neural networks into smaller sub-networks for faster and more efficient distillation.
problem Challenges in training deep neural networks, including local optima, gradient issues, and computational demands.
method Proposes a non-end-to-end distillation approach by splitting networks into smaller, independent sub-networks (neighbourhoods).
result Independent training of smaller sub-networks can speed up distillation and improve efficiency in various applications.