New algorithms optimize without tuning, matching tuned SGD performance.
problem Optimizing machine learning models without manual hyperparameter tuning.
method Formalizes tuning-free algorithms for matching SGD performance with loose hints.
result Tuning-free algorithms can match SGD performance, but not optimal convergence rates.
Paper proposes a reinforcement learning framework for efficient hyper-parameter tuning of stochastic optimization algorithms.
problem Efficient tuning of hyper-parameters for stochastic optimization algorithms.
method Modeling hyper-parameter tuning as a Markov decision process and using policy gradient algorithms.
result The proposed framework significantly reduces the time required for hyper-parameter tuning compared to Bayesian optimization.
New algorithms for sampling and optimization without tuning.
problem Efficient sampling and optimization over probability measures.
method Optimization on the space of probability measures, using gradient flows.
result Strong theoretical guarantees and similar performance to optimally tuned algorithms.
Tuning SVM and boosting models using optimization algorithms.
problem Tuning parameters for SVM and boosting models across various datasets.
method Used grid search to identify parameter ranges and optimization algorithms to select models.
result Optimization algorithms outperformed grid search in selecting well-performing models.
In this paper, we address the challenging problem of selecting tuning parameters for high-dimensional sparse regression. We propose a simple and computationally efficient method, called path thresholding (PaTh), that transforms any tuning parameter-dependent sparse regression algorithm into an asymptotically tuning-fre…
New algorithm tunes SGMCMC hyperparameters for scalable Bayesian inference.
problem Tuning hyperparameters for SGMCMC is challenging due to lack of principled methods.
method Proposes a bandit-based algorithm using Stein discrepancies to tune hyperparameters.
result The method effectively tunes SGMCMC hyperparameters for various applications.
Modern machine learning algorithms are increasingly computationally demanding, requiring specialized hardware and distributed computation to achieve high performance in a reasonable time frame. Many hyperparameter search algorithms have been proposed for improving the efficiency of model selection, however their adapta…
HASSO improves SO algorithms by dynamically tuning hyperparameters.
problem Inefficiency of hyperparameter tuning for SO algorithms.
method HASSO is a self-adjusting SO algorithm that dynamically tunes its own hyperparameters.
result HASSO enhances the performance of various SO algorithms across different test problems.
A framework for auto-tuning hyper-parameters in contextual bandit algorithms.
problem Auto-tuning hyper-parameters in real-time for contextual bandit algorithms.
method Proposes a Syndicated Bandits framework to learn multiple hyper-parameters dynamically.
result Achieves optimal regret bounds under certain scenarios and handles multiple contextual bandit algorithms.
We analyze SGAs for statistical inference via asymptotics, improving tuning methods.
problem Improper tuning of SGAs for optimization and sampling.
method Characterize large-sample asymptotics of SGAs via step-size and sample-size scaling limits.
result Iterate averaging with large step size is robust and asymptotically has covariance proportional to MLE's.
Single-head transformers with a single self-attention layer can approximate any sequence-to-sequence function and are efficient under certain conditions.
problem Statistical and computational limits of prompt tuning for transformer-based models.
method Investigation of single-head transformers with a single self-attention layer, proving universality and efficiency under SETH.
result Existence of almost-linear time prompt tuning inference algorithms under certain conditions.
The paper introduces a Hessian-based method to improve generalization in fine-tuned deep neural networks.
problem Improving generalization in fine-tuned deep neural networks, especially in noisy conditions.
method PAC-Bayesian analysis to identify a Hessian-based distance measure, proving generalization bounds, and developing an algorithm with a generalization error guarantee.
result Hessian-based distance measure correlates well with observed generalization gaps and can match the scale of these gaps in practice.
Paper proposes EEIPU, a memoization-aware BO algorithm to reduce hyperparameter tuning costs.
problem High costs in GPU-days for training and fine-tuning language models.
method Memoization-aware Bayesian Optimization (EEIPU) algorithm in tandem with pipeline caching.
result EEIPU produces 103% more hyperparameter candidates and 108% more validation metric improvement.
New algorithms eliminate stepsize tuning for bilevel optimization problems.
problem Bilevel optimization problems with unknown parameters and stepsizes.
method D-TFBO and S-TFBO algorithms with adaptive stepsizes.
result Achieve performance comparable to well-tuned approaches with theoretical guarantees.
BBVI relies on adaptive stochastic optimization algorithms for posterior approximation, but these require extensive tuning.
problem BBVI posterior approximation
method Adaptive stochastic optimization algorithms
result No single method dominates, but a selection of 5 suffices.
Study shows leaving some hyperparameters at default can outperform tuning them.
problem Importance of tuning hyperparameters in machine learning performance.
method Non-inferiority test and tuning risk methodology.
result Leaving some hyperparameters at default can outperform tuning them.
We introduce a means of automating machine learning (ML) for big data tasks, by performing scalable stochastic Bayesian optimisation of ML algorithm parameters and hyper-parameters. More often than not, the critical tuning of ML algorithm parameters has relied on domain expertise from experts, along with laborious hand…
Machine learning algorithms often contain many hyperparameters (HPs) whose values affect the predictive performance of the induced models in intricate ways. Due to the high number of possibilities for these HP configurations and their complex interactions, it is common to use optimization techniques to find settings th…
Fine-tuning improves meta-learning by leveraging shared representations.
problem Meta-learning's challenge in rapidly learning new tasks.
method Theoretical framework and risk bounds on gradient descent fine-tuning.
result Fine-tuning-based methods can provably leverage shared structure.
We study a budgeted hyper-parameter tuning problem, where we optimize the tuning result under a hard resource constraint. We propose to solve it as a sequential decision making problem, such that we can use the partial training progress of configurations to dynamically allocate the remaining budget. Our algorithm combi…
OTSL improves structure learning accuracy with out-of-sample and resampling strategies.
problem Determining optimal hyperparameters for structure learning algorithms.
method Out-of-sample Tuning for Structure Learning (OTSL) using resampling strategies.
result Improves graphical accuracy of structure learning algorithms.
We consider online learning with linear models, where the algorithm predicts on sequentially revealed instances (feature vectors), and is compared against the best linear function (comparator) in hindsight. Popular algorithms in this framework, such as Online Gradient Descent (OGD), have parameters (learning rates), wh…
New method for tuning Graphical Lasso hyperparameters.
problem Tuning hyperparameters of Graphical Lasso.
method Bilevel optimization with first-order method.
result Derivation of Graphical Lasso Jacobian.
Adjoint Matching improves flow and diffusion models with reward fine-tuning.
problem Improving generative models with reward fine-tuning.
method Casting reward fine-tuning as stochastic optimal control (SOC) and enforcing a specific noise schedule.
result Adjoint Matching outperforms existing SOC algorithms.
Amazon SageMaker AMT automates machine learning model tuning.
problem Challenging hyperparameter tuning for complex machine learning systems.
method Gradient-free optimization using random search or Bayesian optimization.
result AMT finds the best hyperparameter configurations for models.
Modern supervised machine learning algorithms involve hyperparameters that have to be set before running them. Options for setting hyperparameters are default values from the software package, manual configuration by the user or configuring them for optimal predictive performance by a tuning procedure. The goal of this…
New algorithms learn latent variable models without tuning, outperforming existing methods.
problem Learning latent variable models without manual tuning.
method Two particle-based algorithms using free energy minimization and coin betting.
result Learning algorithms are entirely tuning-free and competitive with existing methods.
This paper proposes automatic tuning of Bayesian Optimization's acquisition function.
problem Optimizing black-box functions with noisy, expensive evaluations and hyperparameter tuning.
method Exploring heuristics to automatically tune acquisition functions in Bayesian Optimization.
result Demonstrates effectiveness of heuristics in automatic Bayesian Optimization.
Automated hyperparameter tuning aspires to facilitate the application of machine learning for non-experts. In the literature, different optimization approaches are applied for that purpose. This paper investigates the performance of Differential Evolution for tuning hyperparameters of supervised learning algorithms for…
Paper defines hyperparameter importance for efficient tuning.
problem Computational inefficiency in tuning all hyperparameters.
method Defines hyperparameter importance via subsampling procedures.
result Proposed importance consistent with full data under weak conditions.
A new method constrains deep networks during fine-tuning to improve generalization.
problem Improving generalization of fine-tuned deep networks.
method A neural network generalisation bound based on distance from initial weights constrains the hypothesis class to a small sphere.
result Empirical evaluation shows superior generalization performance compared to existing methods.
For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them. However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default val…
Two retraining techniques outperform fine-tuning in neural network pruning.
problem Improving accuracy and compression in neural network pruning.
method Weight rewinding and learning rate rewinding compared to fine-tuning.
result Rewinding techniques outperform fine-tuning in accuracy and compression.
Black box discrete optimization (BBDO) appears in wide range of engineering tasks. Evolutionary or other BBDO approaches have been applied, aiming at automating necessary tuning of system parameters, such as hyper parameter tuning of machine learning based systems when being installed for a specific task. However, auto…
In this short paper we investigate whether meta-learning techniques can be used to more effectively tune the hyperparameters of machine learning models using successive halving (SH). We propose a novel variant of the SH algorithm (MeSH), that uses meta-regressors to determine which candidate configurations should be el…
In sparse regression modeling via regularization such as the lasso, it is important to select appropriate values of tuning parameters including regularization parameters. The choice of tuning parameters can be viewed as a model selection and evaluation problem. Mallows' Cp type criteria may be used as a tuning param…
Paper proposes an algorithm to learn DAGs with indirect dependencies.
problem Learning DAGs misses indirect dependencies in local variables.
method Two-phase algorithm using high-order HSIC for local optimization.
result OT algorithm outperforms existing methods in structure estimation.
New algorithm closes empirical gap in PFSGD performance.
problem Empirical performance gap between tuned SGD and PFSGD.
method Parameter-free algorithm based on Coin-Betting ODE updates.
result New algorithm outperforms tuned baselines and matches optimal performance.
The paper introduces V(I) to guide algorithm choice and parameter tuning in financial forecasting.
problem Selecting optimal algorithms and tuning parameters for financial time-series forecasting.
method Estimating Shannon's mutual information and using it to define performance bounds.
result Illustrates the value of information for mean-square error minimization in cryptocurrency forecasts.
Self-Tuning Actor-Critic improves reinforcement learning performance.
problem Manual hyperparameter tuning is time-consuming and domain-specific.
method Uses metagradients for online hyperparameter adaptation.
result Improves performance across various domains and tasks.
Efficiently tunes hyperparameters with dynamic accuracy method.
problem Optimizing machine learning hyperparameters with inexact evaluations.
method Dynamic accuracy derivative-free optimization for hyperparameter tuning.
result Demonstrates robust and efficient hyperparameter tuning compared to fixed accuracy methods.
This study analyzes and optimizes hyperparameters for machine learning models.
problem Optimizing hyperparameters for machine learning models to improve performance.
method Experimental analysis of 30 hyperparameters from six machine learning algorithms using R and SPOT.
result A new consensus ranking method for analyzing results from multiple algorithms.
Machine learning applications often require hyperparameter tuning. The hyperparameters usually drive both the efficiency of the model training process and the resulting model quality. For hyperparameter tuning, machine learning algorithms are complex black-boxes. This creates a class of challenging optimization problem…
Multi-label classification is an approach which allows a datapoint to be labelled with more than one class at the same time. A common but trivial approach is to train individual binary classifiers per label, but the performance can be improved by considering associations within the labels. Like with any machine learnin…
Mango automates hyperparameter tuning for large-scale ML training.
problem Manual hyperparameter tuning is tedious and inefficient for large-scale machine learning.
method Parallel hyperparameter tuning with intelligent search strategies and flexible abstractions.
result Mango achieves comparable performance to Hyperopt while supporting distributed computing.
Meta-strategy learns tuning parameters for online learning methods.
problem Difficulty in setting tuning parameters for online learning methods.
method Meta-learning approach to learn parameters from past tasks.
result Meta-strategy improves on learning each task in isolation.
Deep learning algorithms have achieved excellent performance lately in a wide range of fields (e.g., computer version). However, a severe challenge faced by deep learning is the high dependency on hyper-parameters. The algorithm results may fluctuate dramatically under the different configuration of hyper-parameters. A…
New method tunes SMC samplers efficiently without high costs.
problem Tuning SMC samplers with unadjusted kernels is challenging.
method Greedy Incremental Divergence Minimization (GIDM) for step size tuning.
result GIDM reduces KL divergence and tunes SMC samplers efficiently.