Hyperparameters are critical in machine learning, as different hyperparameters often result in models with significantly different performance. Hyperparameters may be deemed confidential because of their commercial value and the confidentiality of the proprietary algorithms that the learner uses to learn them. In this …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “hyperparameter learning”
Survey of techniques for automating hyperparameter optimization in machine learning.
problem Finding optimal hyperparameters for machine learning algorithms.
method Review of various hyperparameter optimization techniques.
result Unified treatment of hyperparameter optimization techniques.
Efficiently tunes hyperparameters with dynamic accuracy method.
problem Optimizing machine learning hyperparameters with inexact evaluations.
method Dynamic accuracy derivative-free optimization for hyperparameter tuning.
result Demonstrates robust and efficient hyperparameter tuning compared to fixed accuracy methods.
Deep learning models are full of hyperparameters, which are set manually before the learning process can start. To find the best configuration for these hyperparameters in such a high dimensional space, with time-consuming and expensive model training / validation, is not a trivial challenge. Bayesian optimization is a…
Hyperparameter tuning is an omnipresent problem in machine learning as it is an integral aspect of obtaining the state-of-the-art performance for any model. Most often, hyperparameters are optimized just by training a model on a grid of possible hyperparameter values and taking the one that performs best on a validatio…
New optimizer improves privacy-protected hyperparameter tuning.
problem No practical methods for differentially private hyperparameter selection.
method Study honest hyperparameter selection under DP, show adaptive optimizers like DPAdam have an advantage.
result DPAdam optimizes hyperparameters more efficiently under DP constraints.
Sherpa automates hyperparameter tuning for machine learning models.
problem Hyperparameter optimization for computationally expensive models.
method Powerful and interchangeable algorithms, interactive dashboard.
result Automates tedious aspects of model tuning.
Method analyzes hyperparameters using HSIC for better neural network performance.
problem Complex hyperparameter spaces in deep learning.
method Goal-oriented sensitivity analysis using HSIC.
result Robust analysis index quantifies hyperparameters' impact.
Paper defines hyperparameter importance for efficient tuning.
problem Computational inefficiency in tuning all hyperparameters.
method Defines hyperparameter importance via subsampling procedures.
result Proposed importance consistent with full data under weak conditions.
One-pass optimisation for high-dimensional hyperparameters.
problem Efficient optimisation of hyperparameters in machine learning models.
method Approximate hypergradient-based optimisation for any continuous hyperparameter, requiring only one training episode.
result Competitive performance on various datasets without hyperparameter restarts.
PHS optimizes hyperparameters in parallel for expensive computations.
problem Optimizing hyperparameters in computationally expensive tasks.
method Bayesian optimization for parallel hyperparameter search.
result Efficient hyperparameter optimization on multiple instances.
Improves hyperparameter learning in GP models with non-conjugate likelihoods.
problem Hyperparameter learning entangled with approximate inference in GP models.
method Hybrid training procedure combining VI for inference and EP-like marginal likelihood approximation for hyperparameter learning.
result Empirically demonstrates the effectiveness of the proposed training procedure across various data sets.
Study shows leaving some hyperparameters at default can outperform tuning them.
problem Importance of tuning hyperparameters in machine learning performance.
method Non-inferiority test and tuning risk methodology.
result Leaving some hyperparameters at default can outperform tuning them.
SCoreBO improves Bayesian optimization by learning hyperparameters and self-correcting.
problem Efficient hyperparameter tuning for Gaussian process models in Bayesian optimization.
method Introduces SAL and SCoreBO, which prioritize hyperparameter learning and perform simultaneous optimization and learning.
result SCoreBO outperforms state-of-the-art methods on traditional benchmarks and atypical tasks.
Improved Random Search for hyperparameter optimization.
problem Optimizing machine learning hyperparameters efficiently.
method Weighted Random Search with probabilistic hyperparameter updates.
result Our method outperforms standard Random Search within the same budget.
Mango automates hyperparameter tuning for large-scale ML training.
problem Manual hyperparameter tuning is tedious and inefficient for large-scale machine learning.
method Parallel hyperparameter tuning with intelligent search strategies and flexible abstractions.
result Mango achieves comparable performance to Hyperopt while supporting distributed computing.
WRS improves CNN hyperparameter optimization.
problem Finding optimal hyperparameters for CNNs efficiently.
method Combination of Random Search and probabilistic greedy heuristic.
result WRS outperforms other methods in CNN hyperparameter optimization.
OTSL improves structure learning accuracy with out-of-sample and resampling strategies.
problem Determining optimal hyperparameters for structure learning algorithms.
method Out-of-sample Tuning for Structure Learning (OTSL) using resampling strategies.
result Improves graphical accuracy of structure learning algorithms.
Paper proposes an efficient bandit-based algorithm for hyperparameter optimization.
problem Efficiently evaluating hyperparameters in deep learning models with large search spaces.
method Sub-Sampling (SS) algorithm combined with Bayesian Optimization (BOSS).
result Theoretical proof of optimality and empirical validation of superior performance.
Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyper…
L2O-CFGD meta-learns hyperparameters for FGD, improving performance.
problem Challenges in convergence and hyperparameter selection for FGD.
method Learning to Optimize Caputo Fractional Gradient Descent (L2O-CFGD).
result Meta-learned schedule outperforms static hyperparameters and achieves comparable performance to black-box meta-learners.
New method selects best offline RL policies from logged data.
problem Hyperparameter selection challenges offline RL.
method Offline hyperparameter selection for RL algorithms.
result Reliable ranking and selection of policies across hyperparameters.
The success of machine learning on a given task dependson, among other things, which learning algorithm is selected and its associated hyperparameters. Selecting an appropriate learning algorithm and setting its hyperparameters for a given data set can be a challenging task, especially for users who are not experts in …
Differentially private hyperparameter tuning improves privacy in machine learning.
problem Hyperparameter tuning leaks private information through selected configurations.
method Local Bayesian optimization using Gaussian Process surrogate for private gradient approximation.
result DP-GIBO converges to locally optimal hyperparameters with polynomial dimensional dependence.
SMAC3 optimizes machine learning hyperparameters efficiently.
problem Optimizing hyperparameters for machine learning algorithms.
method Bayesian Optimization framework with facades for various use cases.
result Improves performance with minimal evaluations.
Bayesian optimization outperforms other methods in hyperparameter tuning for reinforcement learning.
problem Finding optimal hyperparameters that generalize across random seeds in reinforcement learning.
method Benchmarked Successive Halving, Random Search, and Bayesian Optimization with and without repetitions on PPO2 algorithms for Cartpole and Inverted Pendulum tasks.
result Bayesian optimization with noise robust acquisition function is the best choice.
The performance of deep (reinforcement) learning systems crucially depends on the choice of hyperparameters. Their tuning is notoriously expensive, typically requiring an iterative training process to run for numerous steps to convergence. Traditional tuning algorithms only consider the final performance of hyperparame…
We introduce the hyperparameter search problem in the field of machine learning and discuss its main challenges from an optimization perspective. Machine learning methods attempt to build models that capture some element of interest based on given data. Most common learning algorithms feature a set of hyperparameters t…
It is typical for a machine learning system to have numerous hyperparameters that affect its learning rate and prediction quality. Finding a good combination of the hyperparameters is, however, a challenging job. This is mainly because evaluation of each combination is extremely expensive computationally; indeed, train…
Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogatescs.AI
Automatically searching for optimal hyperparameter configurations is of crucial importance for applying deep learning algorithms in practice. Recently, Bayesian optimization has been proposed for optimizing hyperparameters of various machine learning algorithms. Those methods adopt probabilistic surrogate models like G…
Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed …
Modern supervised machine learning algorithms involve hyperparameters that have to be set before running them. Options for setting hyperparameters are default values from the software package, manual configuration by the user or configuring them for optimal predictive performance by a tuning procedure. The goal of this…
New method FedEx accelerates federated hyperparameter tuning.
problem Federated hyperparameter tuning challenges in distributed learning.
method FedEx method connecting to weight-sharing, adapted for federated optimization.
result FedEx outperforms natural baselines on various benchmarks.
VisEvol uses evolutionary optimization to find optimal hyperparameters for machine learning models.
problem Finding the best hyperparameters for complex machine learning models is computationally intensive and challenging.
method VisEvol employs evolutionary optimization, storing performant models and improving others through crossover and mutation processes.
result VisEvol generates a voting ensemble of models with improved predictive performance.
This paper quantifies hyperparameter transfer and finds embedding layer learning rate is key.
problem Quantifying optimal hyperparameters for large language models across scales.
method Developed three metrics to quantify hyperparameter transfer and investigated the importance of embedding layer learning rate.
result Maximal Update (μP) parameterization offers high-quality learning rate transfer compared to standard parameterization (SP).
LLMs learn to recommend models and hyperparameters from dataset metadata.
problem Model and hyperparameter selection in machine learning is challenging and resource-intensive.
method Converted datasets into metadata and prompted LLMs to recommend models and hyperparameters.
result LLMs can recommend competitive models and hyperparameters without search.
This study analyzes and optimizes hyperparameters for machine learning models.
problem Optimizing hyperparameters for machine learning models to improve performance.
method Experimental analysis of 30 hyperparameters from six machine learning algorithms using R and SPOT.
result A new consensus ranking method for analyzing results from multiple algorithms.
TaskSet dataset speeds up hyperparameter optimization.
problem Optimizing hyperparameters for various machine learning tasks.
method Meta-learning an ordered list of hyperparameters from a large dataset.
result Meta-learning achieves large speedups in sample efficiency.
HyperSTAR optimizes deep network hyperparameters for specific tasks.
problem Finding optimal hyperparameters for deep neural networks is computationally expensive and task-agnostic approaches are inefficient.
method HyperSTAR learns a dataset representation and a performance predictor from raw images, ranking and recommending hyperparameters conditioned on a joint dataset-hyperparameter space.
result HyperSTAR reduces HPO time by 50% on large-scale image classification datasets.
This paper introduces multi-objective hyperparameter optimization in machine learning.
problem Optimizing machine learning pipelines for multiple objectives, not just accuracy.
method Survey of optimization strategies and applications in multi-objective hyperparameter optimization.
result The importance and utility of multi-objective hyperparameter optimization in applied machine learning.
With the advent of automated machine learning, automated hyperparameter optimization methods are by now routinely used in data mining. However, this progress is not yet matched by equal progress on automatic analyses that yield information beyond performance-optimizing hyperparameter settings. In this work, we aim to a…
Proposes a gradient-based bilevel optimization method for efficient hyperparameter tuning.
problem Efficiently tuning hyperparameters in machine learning models.
method Gradient-based bilevel optimization approach.
result The proposed method is multiple times faster than existing techniques.
The performance of policy gradient methods is sensitive to hyperparameter settings that must be tuned for any new application. Widely used grid search methods for tuning hyperparameters are sample inefficient and computationally expensive. More advanced methods like Population Based Training that learn optimal schedule…
EvoGrad improves efficiency in meta-learning and hyperparameter optimization.
problem Efficiently compute hypergradients for larger network architectures.
method Uses evolutionary techniques to estimate hypergradients without second-order derivatives or longer computational graphs.
result Significant improvements in efficiency, enabling scaling to bigger architectures.
This paper re-evaluates hyperparameters for fine-tuning pre-trained models.
problem Current hyperparameter settings for fine-tuning are often ad-hoc and fixed.
method Empirical evaluation of learning rate, batch size, and momentum for fine-tuning.
result Optimal hyperparameters are not only dataset-dependent but also sensitive to domain similarity.
In this paper we develop a Bayesian optimization based hyperparameter tuning framework inspired by statistical learning theory for classifiers. We utilize two key facts from PAC learning theory; the generalization bound will be higher for a small subset of data compared to the whole, and the highest accuracy for a smal…
LGD algorithm learns optimal hyperparameters for regression tasks.
problem Learning to learn hyperparameters for regression problems.
method Langevin Gradient Descent (LGD) approximates posterior distribution.
result Meta-learning optimal hyperparameters achieves Bayes' optimal solution.
Recently, several optimization methods have been successfully applied to the hyperparameter optimization of deep neural networks (DNNs). The methods work by modeling the joint distribution of hyperparameter values and corresponding error. Those methods become less practical when applied to modern DNNs whose training ma…