PHS optimizes hyperparameters in parallel for expensive computations.
problem Optimizing hyperparameters in computationally expensive tasks.
method Bayesian optimization for parallel hyperparameter search.
result Efficient hyperparameter optimization on multiple instances.
Mango automates hyperparameter tuning for large-scale ML training.
problem Manual hyperparameter tuning is tedious and inefficient for large-scale machine learning.
method Parallel hyperparameter tuning with intelligent search strategies and flexible abstractions.
result Mango achieves comparable performance to Hyperopt while supporting distributed computing.
New reshaping method improves hyperparameter search over random search.
problem Improving hyperparameter search efficiency.
method Introducing reshaping techniques to optimize search distributions.
result Substantial gains over random search in various experiments.
AgEBO-Tabular combines NAS and hyperparameter tuning for fast, high-performing tabular models.
problem Developing high-performing predictive models for large tabular data sets is challenging.
method Combines aging evolution NAS and asynchronous Bayesian optimization for hyperparameter tuning in data-parallel training.
result Automatically discovered neural network models outperform state-of-the-art AutoML ensembles in inference speed by two orders of magnitude.
Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated. Examples include grid search, uniform random search, low discrepancy sequences, and other sampl…
Machine learning applications often require hyperparameter tuning. The hyperparameters usually drive both the efficiency of the model training process and the resulting model quality. For hyperparameter tuning, machine learning algorithms are complex black-boxes. This creates a class of challenging optimization problem…
A scalable parallel BO method for asynchronous settings.
problem Expensive-to-evaluate problems in machine learning.
method Simple and scalable Bayesian optimization method for asynchronous parallel settings.
result Demonstrated promising performance on benchmark functions and hyperparameter optimization.
This paper reviews hyperparameter optimization methods and best practices.
problem Finding optimal hyperparameters for machine learning models.
method Various hyperparameter optimization methods are reviewed, including grid search, random search, evolutionary algorithms, Bayesian optimization, Hyperband, and racing.
result Practical recommendations for conducting hyperparameter optimization are provided.
A new framework improves graph construction for semi-supervised learning.
problem Improving graph construction for better semi-supervised classification accuracy.
method Parallel hyperparameter search with adaptive resource allocation for gradient-based optimization of edge weights.
result Significantly outperforms existing graph construction schemes in accuracy and scalability.
Survey of techniques for automating hyperparameter optimization in machine learning.
problem Finding optimal hyperparameters for machine learning algorithms.
method Review of various hyperparameter optimization techniques.
result Unified treatment of hyperparameter optimization techniques.
SLM Lab is a framework for reproducible RL research with modular algorithms.
problem Reproducibility in deep reinforcement learning.
method Modular software framework for RL algorithms, synchronous/asynchronous execution, hyperparameter search, result analysis.
result Comprehensive benchmark and novel RL algorithms (e.g., discrete-AC variant, hybrid training method).
Empirical comparison of 18 hyperparameter tuning algorithms for SVM.
problem Tuning hyperparameters C and γ for SVM with RBF kernel. method Compared 18 search algorithms on 115 real-life data sets.
result Trees of Parzen estimators and particle swarm optimization perform similarly to grid search.
The performance of modern machine learning methods highly depends on their hyperparameter configurations. One simple way of selecting a configuration is to use default settings, often proposed along with the publication and implementation of a new algorithm. Those default values are usually chosen in an ad-hoc manner t…
Paper proposes HCDC to improve hyperparameter search efficiency.
problem Poor generalizability of dataset condensation across different hyperparameters.
method HCDC algorithm that matches hyperparameter gradients for synthetic validation dataset.
result HCDC effectively maintains validation-performance rankings of models.
Paper proposes an efficient bandit-based algorithm for hyperparameter optimization.
problem Efficiently evaluating hyperparameters in deep learning models with large search spaces.
method Sub-Sampling (SS) algorithm combined with Bayesian Optimization (BOSS).
result Theoretical proof of optimality and empirical validation of superior performance.
WRS improves CNN hyperparameter optimization.
problem Finding optimal hyperparameters for CNNs efficiently.
method Combination of Random Search and probabilistic greedy heuristic.
result WRS outperforms other methods in CNN hyperparameter optimization.
Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed …
Asynchronous method for hyperparameter and neural architecture search.
problem Efficiently searching for optimal hyperparameters and neural architectures.
method Model-based, asynchronous multi-fidelity method combining Hyperband and Gaussian process-based Bayesian optimization.
result Substantial speed-ups over current state-of-the-art methods on various benchmarks.
Improved Random Search for hyperparameter optimization.
problem Optimizing machine learning hyperparameters efficiently.
method Weighted Random Search with probabilistic hyperparameter updates.
result Our method outperforms standard Random Search within the same budget.
We introduce the hyperparameter search problem in the field of machine learning and discuss its main challenges from an optimization perspective. Machine learning methods attempt to build models that capture some element of interest based on given data. Most common learning algorithms feature a set of hyperparameters t…
New method optimizes hyperparameters for non-smooth problems efficiently.
problem Efficiently tuning hyperparameters for non-smooth cost functions.
method Combines hyperparameter search with proximal gradient updates.
result Method converges to local optimum of LOO validation error.
New tools reveal simple structure in complex hyperparameter loss surfaces near optima.
problem Understanding the behavior of hyperparameter loss surfaces near model optima.
method Developed a novel technique based on random search to uncover asymptotic features of the loss surface.
result Random search within the asymptotic regime yields a new distribution with parameters defining the loss surface.
SMAC method optimizes tree-boosting hyperparameters best.
problem Optimizing hyperparameters for tree-boosting to improve model accuracy.
method Compared and evaluated various hyperparameter optimization methods.
result SMAC method outperforms other methods for hyperparameter tuning.
This paper compares Grid Search, Random Search, and Genetic Algorithm for NAS.
problem Hyperparameter optimization for neural architecture search.
method Comparison of Grid Search, Random Search, and Genetic Algorithm.
result Genetic Algorithm outperforms Grid Search and Random Search in terms of accuracy and execution time.
A new Randomized-Hyperopt method improves XGBoost hyperparameter tuning.
problem Improving the performance of XGBoost through hyperparameter optimization.
method Proposes Randomized-Hyperopt for XGBoost hyperparameter tuning.
result Randomized-Hyperopt outperforms other methods in terms of accuracy and execution time.
While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal. Likewise, we demonstrate that the common practice of using very few …
Paper studies kernel hyperparameters for clustering, proposing an efficient search method.
problem Challenges in tuning kernel parameters for clustering, especially for RBF kernels.
method Derives a lower bound for RBF kernel parameters, proposes an efficient hyperparameter search algorithm.
result Proposes an efficient algorithm for hyperparameter search in kernel clustering, improving upon grid search.
We propose a new algorithm for hyperparameter selection in machine learning algorithms. The algorithm is a novel modification of Harmonica, a spectral hyperparameter selection approach using sparse recovery methods. In particular, we show that a special encoding of hyperparameter space enables a natural group-sparse re…
Differentially private hyperparameter tuning improves privacy in machine learning.
problem Hyperparameter tuning leaks private information through selected configurations.
method Local Bayesian optimization using Gaussian Process surrogate for private gradient approximation.
result DP-GIBO converges to locally optimal hyperparameters with polynomial dimensional dependence.
In this paper, we propose the use of a black-box optimization method called deterministic Mesh Adaptive Direct Search (MADS) algorithm with orthogonal directions (Ortho-MADS) for the selection of hyperparameters of Support Vector Machines with a Gaussian kernel. Different from most of the methods in the literature that…
Improves neural architecture search methods to be more stable and efficient.
problem Neural architecture search methods are unstable and sensitive to hyperparameters.
method Discusses practical considerations to improve stability and efficiency.
result Improves overall performance of neural architecture search methods.
Modern machine learning algorithms are increasingly computationally demanding, requiring specialized hardware and distributed computation to achieve high performance in a reasonable time frame. Many hyperparameter search algorithms have been proposed for improving the efficiency of model selection, however their adapta…
Convolutional Neural Network is known as ConvNet have been extensively used in many complex machine learning tasks. However, hyperparameters optimization is one of a crucial step in developing ConvNet architectures, since the accuracy and performance are reliant on the hyperparameters. This multilayered architecture pa…
In deep learning, performance is strongly affected by the choice of architecture and hyperparameters. While there has been extensive work on automatic hyperparameter optimization for simple spaces, complex spaces such as the space of deep architectures remain largely unexplored. As a result, the choice of architecture …
Automatically searching for optimal hyperparameter configurations is of crucial importance for applying deep learning algorithms in practice. Recently, Bayesian optimization has been proposed for optimizing hyperparameters of various machine learning algorithms. Those methods adopt probabilistic surrogate models like G…
Hyperparameter optimization aims to find the optimal hyperparameter configuration of a machine learning model, which provides the best performance on a validation dataset. Manual search usually leads to get stuck in a local hyperparameter configuration, and heavily depends on human intuition and experience. A simple al…
Parallelizes MCTS for continuous domains using leaf and root parallelization.
problem Solving challenging tasks in continuous domains using MCTS.
method Extends existing parallelization strategies to continuous domains, focusing on leaf and root parallelization.
result Proposes two final selection strategies for continuous states in root parallelization.
Proposes a gradient-based bilevel optimization method for efficient hyperparameter tuning.
problem Efficiently tuning hyperparameters in machine learning models.
method Gradient-based bilevel optimization approach.
result The proposed method is multiple times faster than existing techniques.
Proposes baselines for joint NAS and HPO optimization.
problem Joint optimization of neural architecture and hyperparameters for multiple objectives.
method Extends existing methods to jointly optimize with multiple objectives.
result Serves as simple baselines for future multi-objective joint NAS + HPO research.
In order to find hyperparameters for a machine learning model, algorithms such as grid search or random search are used over the space of possible values of the models hyperparameters. These search algorithms opt the solution that minimizes a specific cost function. In language models, perplexity is one of the most pop…
HASSO improves SO algorithms by dynamically tuning hyperparameters.
problem Inefficiency of hyperparameter tuning for SO algorithms.
method HASSO is a self-adjusting SO algorithm that dynamically tunes its own hyperparameters.
result HASSO enhances the performance of various SO algorithms across different test problems.
ExperienceThinking optimizes hyperparameters quickly with smart pruning and knowledge use.
problem Efficiently optimizing hyperparameters in machine learning with limited evaluations.
method Two novel methods: search space pruning and knowledge utilization.
result ExperienceThinking outperforms classical algorithms in few evaluations.
Autodock is a widely used molecular modeling tool which predicts how small molecules bind to a receptor of known 3D structure. The current version of AutoDock uses meta-heuristic algorithms in combination with local search methods for doing the conformation search. Appropriate settings of hyperparameters in these algor…
Auptimizer simplifies hyperparameter tuning for machine learning models.
problem Difficulty and time-consuming hyperparameter tuning for machine learning models.
method General HPO framework that distributes computing resources and integrates various HPO techniques.
result Simplified model tuning and bookkeeping for data scientists.
Approximate nearest neighbor algorithms are used to speed up nearest neighbor search in a wide array of applications. However, current indexing methods feature several hyperparameters that need to be tuned to reach an acceptable accuracy--speed trade-off. A grid search in the parameter space is often impractically slow…
This paper explores BDL hyperparameters for robust polynomial mapping with noise.
problem Designing BDL hyperparameters for robust function mapping with uncertainty quantification.
method Mapping Bayesian connectionist representations to polynomials of varying orders and noise types.
result Optimal network depth and ensemble size for prediction and uncertainty quantification.
Efficiently identifies promising hyperparameters for online learning models.
problem Expensive hyperparameter search for non-stationary model training.
method Two-stage approach: efficient configuration identification followed by full training.
result Up to 10x reduction in hyperparameter search cost on public benchmark.
New method optimizes BO by learning search spaces from past evaluations.
problem Optimizing expensive black-box functions efficiently.
method Automatically designs BO search space from historical data.
result Significant boost in BO performance by reducing search space size.